Enable adaptive stripping and eliminate dependency of weight sharing feature on OVEP qdq stripping #629

saurabhkale17 · 2025-03-27T10:59:14Z

Description

The weight sharing feature in OVEP currently depends on QDQ stripping, requiring users to enable OVEP QDQ stripping for weight sharing to function correctly.

This PR removes that dependency, allowing weight sharing to work independently.

The broader goal is to enable compiler stripping through OVEP using a query, and eliminating this dependency is a crucial first step toward that objective.

Enable Default Compiler Stripping & Rename QDQ Optimizer Flag

Query OV for NPU_QDQ_OPTIMIZATION and set it to true if available, ensuring compiler-based stripping is the default.
Disable OVEP QDQ stripping by setting ovep_qdq_optimizer = false.
Rename enable_qdq_optimizer to enable_ovep_qdq_optimizer for clarity.

Motivation and Context

Ensures compiler-based stripping is the default mechanism for NPU by explicitly setting NPU_DQD_OPTIMIZATION = true if available.

Prevents redundant optimizations by disabling OVEP QDQ stripping (ovep_qdq_optimizer = false).

Improves code clarity by renaming enable_qdq_optimizer to enable_ovep_qdq_optimizer, reducing confusion for future developers.

MayureshV1 · 2025-03-28T00:34:30Z

onnxruntime/core/providers/openvino/backend_manager.cc

@@ -359,14 +359,35 @@ BackendManager::GetModelProtoFromFusedNode(const onnxruntime::Node& fused_node,
    }
  };

+#if (((OPENVINO_VERSION_MAJOR == 2025) && (OPENVINO_VERSION_MINOR > 0)) || (OPENVINO_VERSION_MAJOR > 2025))


@saurabhkale17 .. Is this OV version check required? I think it is redundant and can be optimized out.

Below line, would only enable the property if OV 2025.1 and driver supports QDQ stripping:
if (std::find(supported_properties.begin(), supported_properties.end(), "NPU_QDQ_OPTIMIZATION") != supported_properties.end()) {

This OpenVINO version check is necessary to avoid compilation errors in older versions.
For older OV versions I get this error: C2039: 'qdq_optimization': is not a member of 'ov::intel_npu'

The reason for this behavior:
This check happens at runtime, because it queries the device's supported properties dynamically.
if (std::find(supported_properties.begin(), supported_properties.end(), "NPU_QDQ_OPTIMIZATION") != supported_properties.end())

However, the line
OVCore::Get()->core.set_property("NPU", {ov::intel_npu::qdq_optimization(true)});
happens at compile-time, since the compiler needs to resolve qdq_optimization(true) during compilation.

If ov::intel_npu::qdq_optimization(true) does not exist in an older OpenVINO version, the compiler treats it as an undefined symbol and throws an error before execution—long before the std::find(...) check can even run.

#if preprocessor checks ensures that qdq_optimization(true) is only compiled when the OpenVINO version supports it, preventing compilation failures in older versions.

Interesting. I was looking at the OV C/C++ API documentation to find out what the behavior is when an unregonized property is used but it's not described or undefined. I was trying to see if instead of doing a get_supported_properties ->set_property we could directly do something like if (!set_property) then fallback.

sfatimar · 2025-04-01T13:45:56Z

onnxruntime/core/providers/openvino/qdq_transformations/qdq_stripping.cc

@@ -529,7 +541,7 @@ static void AddQDQNodeUnit(onnxruntime::Graph& dst_graph,
    SkipReason reason = SkipReason::Other;
    bool keep_dq = CheckDQRuleSet(node_unit, dq_node, src_graph, reason);

-    if (keep_dq) {
+    if (keep_dq || (enable_ovep_weight_sharing && !enable_ovep_qdq_optimizer)) {


Saurabh Can we have a flag for this in qdq_stripping.h instead of checking this again and again.
I think this will be a good code design
enable_ovep_weight_sharing && !enable_ovep_qdq_optimizer

Saurabh Can we have a flag for this in qdq_stripping.h instead of checking this again and again. I think this will be a good code design enable_ovep_weight_sharing && !enable_ovep_qdq_optimizer

I'd prefer we separate the passes. Sure, there's replication but it simplifies the design. Alternatively the routine can be redesigned to be more modular.

I’ve noted your comment and completely agree with the improvement. However, given the limited time we have to enable compiler stripping, I’ll defer this change for now and revisit it later once the immediate priorities are addressed.

Sounds good. I'll approve but let's work on that after the merge.

onnxruntime/core/providers/openvino/backend_manager.cc

javier-intel · 2025-04-02T07:31:01Z

onnxruntime/core/providers/openvino/backend_manager.cc

      IsQDQGraph(subgraph)) {
-    LOGS_DEFAULT(INFO) << "[OpenVINO-EP] QDQ optimization pass status: 1";


The message about QDQ being disabled is still there. Let's keep both in their respective implementation blocks.

We will get this addressed in the next PR we will initiate for refactoring.

javier-intel · 2025-04-02T07:31:17Z

onnxruntime/core/providers/openvino/backend_manager.cc

+    } else {
+      LOGS_DEFAULT(INFO) << "[OpenVINO-EP]: OVEP QDQ optimization pass is enabled";
+    }


Move back to implementation block

javier-intel · 2025-04-02T07:38:27Z

onnxruntime/core/providers/openvino/backend_manager.cc

  const auto& onnx_model_path_name = subgraph.ModelPath();
  // QDQ stripping enabled only for the NPU
  if (session_context_.device_type.find("NPU") != std::string::npos &&
-      session_context_.enable_qdq_optimizer &&
+      (enable_ovep_qdq_optimizer || session_context_.so_share_ep_contexts) &&


This check will fail to do WAI if model is not QDQ. That looks like a bug. Let's move that check up to the declaration of
bool enable_ovep_qdq_optimizer = session_context_.enable_qdq_optimizer && IsQDQGraph(subgraph);

Addressed in the latest commit

javier-intel · 2025-04-02T07:41:24Z

onnxruntime/core/providers/openvino/qdq_transformations/qdq_stripping.cc

@@ -529,7 +541,7 @@ static void AddQDQNodeUnit(onnxruntime::Graph& dst_graph,
    SkipReason reason = SkipReason::Other;
    bool keep_dq = CheckDQRuleSet(node_unit, dq_node, src_graph, reason);

-    if (keep_dq) {
+    if (keep_dq || (enable_ovep_weight_sharing && !enable_ovep_qdq_optimizer)) {


Saurabh Can we have a flag for this in qdq_stripping.h instead of checking this again and again. I think this will be a good code design enable_ovep_weight_sharing && !enable_ovep_qdq_optimizer

I'd prefer we separate the passes. Sure, there's replication but it simplifies the design. Alternatively the routine can be redesigned to be more modular.

MayureshV1 · 2025-04-04T21:08:27Z

@saurabhkale17 . Can you please rebase and fix the issue reported by CI

MayureshV1

Have verified the OV property is set accurately while passing to compiler and the model dump is as expected.

Looks good to merge!

…feature on OVEP qdq stripping (#629) * eliminate dependency of weight sharing on ovep qdq stripping pass * fix qdqnodeunit issue * enable compiler stripping * enable adaptive stripping: cleanup code * fix backward compatibility issue * add logs to identify which stripping is enabled * address PR review comments * fix unused variable error * resolve unused var issue * fix CI issues

saurabhkale17 requested review from MayureshV1, sfatimar, ankitm3k, preetha-intel and jatinwadhwa921 March 27, 2025 10:59

MayureshV1 reviewed Mar 28, 2025

View reviewed changes

MayureshV1 requested a review from javier-intel March 28, 2025 04:45

sfatimar reviewed Apr 1, 2025

View reviewed changes

javier-intel requested changes Apr 2, 2025

View reviewed changes

saurabhkale17 added 7 commits April 7, 2025 17:19

eliminate dependency of weight sharing on ovep qdq stripping pass

35a8b37

fix qdqnodeunit issue

0c600c3

enable compiler stripping

b7b468b

enable adaptive stripping: cleanup code

810eba0

fix backward compatibility issue

e2f814f

add logs to identify which stripping is enabled

37ecdd5

address PR review comments

2969048

vthaniel force-pushed the saurabh/enable_adaptive_stripping branch from 2d3846b to 2969048 Compare April 7, 2025 11:53

saurabhkale17 added 3 commits April 7, 2025 18:05

fix unused variable error

89284a8

resolve unused var issue

a8b7fb1

fix CI issues

5f69392

MayureshV1 approved these changes Apr 7, 2025

View reviewed changes

javier-intel approved these changes Apr 7, 2025

View reviewed changes

MayureshV1 merged commit 2e4d541 into ovep-develop Apr 7, 2025
6 of 8 checks passed

		IsQDQGraph(subgraph)) {
		LOGS_DEFAULT(INFO) << "[OpenVINO-EP] QDQ optimization pass status: 1";

Enable adaptive stripping and eliminate dependency of weight sharing feature on OVEP qdq stripping #629

Enable adaptive stripping and eliminate dependency of weight sharing feature on OVEP qdq stripping #629

Uh oh!

Conversation

saurabhkale17 commented Mar 27, 2025 • edited by MayureshV1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Enable Default Compiler Stripping & Rename QDQ Optimizer Flag

Motivation and Context

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MayureshV1 commented Apr 4, 2025

Uh oh!

MayureshV1 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

saurabhkale17 commented Mar 27, 2025 •

edited by MayureshV1

Loading

MayureshV1 left a comment •

edited

Loading