[Rewriter]: fuse successive Relu/Clip nodes #2410

AyoubMDL · 2025-06-22T12:46:43Z

This PR adds the following transformation:

Relu(Relu(X)) -> Relu
Relu(Clip(X)) -> Clip
Clip(Relu(X)) -> Clip
Clip(Clip(X)) -> Clip

codecov · 2025-06-22T15:15:29Z

❌ 13 Tests Failed:

Tests completed	Failed	Passed	Skipped
1450	13	1437	453

View the top 2 failed test(s) by shortest run time

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_0245_test_clip_example

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_clip_example'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_clip_example' (e=No module named 'tests.onnx_backend_test_code.test_clip_example') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_clip_example.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_clip_example.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_clip_example(x: FLOAT[3], min: FLOAT, max: FLOAT) -> (FLOAT[3]):
E       y = opset13.Clip(x, min, max)
E       return y

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1126_test_slice_end_out_of_bounds

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds' (e=No module named 'tests.onnx_backend_test_code.test_slice_end_out_of_bounds') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_slice_end_out_of_bounds.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_slice_end_out_of_bounds.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT, INT64
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_slice_end_out_of_bounds(x: FLOAT[20,10,5], starts: INT64[1], ends: INT64[1], axes: INT64[1], steps: INT64[1]) -> (FLOAT[20,9,5]):
E       y = opset13.Slice(x, starts, ends, axes, steps)
E       return y

View the full list of 1 ❄️ flaky tests

onnxscript.backend.onnx_export_test.TestOnnxBackEnd::test_export2python_produces_correct_onnx_script_model_1134_test_softmax_axis_1

Flake rate in main: 8.70% (Passed 21 times, Failed 2 times)

Stack Traces | 0.003s run time

onnxscript\backend\onnx_export_test.py:137: in extract_functions
    mod = importlib.import_module(import_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
C:\hostedtoolcache\windows\Python\3.11.9\x64\Lib\importlib\__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   ModuleNotFoundError: No module named 'tests.onnx_backend_test_code.test_softmax_axis_1'

The above exception was the direct cause of the following exception:
.nox\test\Lib\site-packages\parameterized\parameterized.py:620: in standalone_func
    return func(*(a + p.args), **p.kwargs, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:271: in test_export2python_produces_correct_onnx_script_model
    functions = extract_functions(backend_test.name, code, self.test_folder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
onnxscript\backend\onnx_export_test.py:139: in extract_functions
    raise AssertionError(
E   AssertionError: Unable to import 'tests.onnx_backend_test_code.test_softmax_axis_1' (e=No module named 'tests.onnx_backend_test_code.test_softmax_axis_1') (file: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_1.py', absolute path: 'D:\\a\\onnxscript\\onnxscript\\tests\\onnx_backend_test_code\\test_softmax_axis_1.py', current folder: D:\a\onnxscript\onnxscript
E   ---- CONTENT --
E   import numpy
E   from onnx import TensorProto
E   from onnx.helper import make_tensor
E   from onnxscript import script, external_tensor
E   from onnxscript.values import Opset
E   from onnxscript.onnx_types import FLOAT
E   from onnxscript.onnx_opset import opset13
E   
E   @script()
E   def bck_test_softmax_axis_1(x: FLOAT[3,4,5]) -> (FLOAT[3,4,5]):
E       y = opset13.Softmax(x, axis=1)
E       return y

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

Copilot

Pull Request Overview

This PR adds graph rewrite rules to fuse consecutive Relu and Clip operations, updates the test harness to control ONNX Runtime’s optimization level, and provides unit tests to validate the new transformations.

Introduce four fusion rules (Relu(Relu), Relu(Clip), Clip(Relu), Clip(Clip)) in fuse_relus_clips.py
Extend assert_numerically_equal in testing.py to accept an ort_optimization_level argument
Add comprehensive tests in fuse_relus_clips_test.py to cover valid and invalid fusion scenarios

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
onnxscript/rewriter/fuse_relus_clips.py	Implement new `RewriteRule` classes and assemble them into a set.
onnxscript/rewriter/testing.py	Update test helper to pass through ONNX Runtime optimization level.
onnxscript/rewriter/fuse_relus_clips_test.py	Add unit tests for each fusion pattern and edge‐case validations.

Comments suppressed due to low confidence (2)

onnxscript/rewriter/fuse_relus_clips.py:161

The variable name fuse_sucessive_relu_clip_rule has a typo (sucessive vs. successive). Rename it to fuse_successive_relu_clip_rule for consistency with the other rules, and update any references.

fuse_sucessive_relu_clip_rule = FuseSuccessiveReluClip().rule()

onnxscript/rewriter/testing.py:27

[nitpick] The Args: section in the docstring does not match the parameter order of the function signature. Consider reordering the entries so they follow (original_model_proto, rewritten_model_proto, args, ort_optimization_level, rtol, atol).

        ort_optimization_level: Onnxruntime optimization level.

justinchuby · 2025-06-23T16:09:24Z

onnxscript/rewriter/fuse_relus_clips_test.py

+        expected_op_type: str,
+        dtype: str = "float",
+    ):
+        base_model = ir.serde.deserialize_model(base_model)


Suggested change

base_model = ir.serde.deserialize_model(base_model)

justinchuby · 2025-06-23T16:09:31Z

onnxscript/rewriter/fuse_relus_clips_test.py

+
+    def run_test(
+        self,
+        base_model: onnx.ModelProto | ir.Model,


Suggested change

base_model: onnx.ModelProto | ir.Model,

base_model: ir.Model,

justinchuby · 2025-06-23T16:09:57Z

onnxscript/rewriter/fuse_relus_clips_test.py

+        model_proto = onnx.parser.parse_model("""
+            < ir_version: 10, opset_import: ["" : 20] >
+            test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)
+            {
+                x1 = Relu(X)
+                x2 = Relu(x1)
+                Y = Relu(x2)
+            }
+        """)
+        self.run_test(model_proto, expected_op_type="Relu")


Suggested change

model_proto = onnx.parser.parse_model("""

< ir_version: 10, opset_import: ["" : 20] >

test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)

{

x1 = Relu(X)

x2 = Relu(x1)

Y = Relu(x2)

}

""")

self.run_test(model_proto, expected_op_type="Relu")

model = ir.from_onnx_text("""

< ir_version: 10, opset_import: ["" : 20] >

test_model (float[N, 32, 14] X) => (float [N, ?, ?] Y)

{

x1 = Relu(X)

x2 = Relu(x1)

Y = Relu(x2)

}

""")

self.run_test(model, expected_op_type="Relu")

Here and below, thanks!

justinchuby

Thank you - I think this can be part of the default rewrite rules

cc @gramalingam

- Relu(Relu(X)) -> Relu - Relu(Clip(X)) -> Clip - Clip(Relu(X)) -> Clip - Clip(Clip(X)) -> Clip

titaiwangms

Please fix optional lint (it's mainly spelling).

titaiwangms · 2025-06-26T23:03:53Z

onnxscript/rewriter/fuse_relus_clips.py

+        del context  # Unused
+        check_result = orp.MatchResult()
+
+        # check clip min/max are initializers


Suggested change

# check clip min/max are initializers

# check if clip min/max are initializers

onnxscript/rewriter/fuse_relus_clips_test.py

titaiwangms · 2025-06-26T23:14:19Z

onnxscript/rewriter/fuse_relus_clips.py

+        check_result = orp.MatchResult()
+
+        # check clip min/max are initializers
+        initializers = []


nit: Using initializers as variable name seems a bit confusing here. Something like clip_min_max would be more understandable. Especially I think is_graph_input() and is_initializer() can express what you are doing here.

titaiwangms · 2025-06-26T23:21:36Z

onnxscript/rewriter/fuse_relus_clips_test.py

+            ),
+        ]
+    )
+    def test_fuse_successive_relu_clip_non_initializers(self, _, nodes, rewrite_rule):


nit: Could you add "successful" or "fail" on the test title so people can know what to expect?

titaiwangms · 2025-06-26T23:23:12Z

onnxscript/rewriter/fuse_relus_clips.py

+            if initializer.is_graph_input():
+                return check_result.fail(f"{initializer.name} is a graph input.")
+
+            if not initializer.is_initializer() or initializer.const_value is None:


Why constants won't work here?

Could you summarize and update the "check logic" in check() description?

Why constants won't work here?

The condition is mainly to check clip min/max are not initializers (inputs). Are you asking if I should also support cases where min and max are Constant nodes ? (is the check I did exclude this case ?)

Yes, I am curious why we don't include constant nodes here?

Do you mean I support or reject Constant nodes ? Also in the rewriter, I supposed that all constant nodes were folded (as ConstantFolding is the first pass applied in the optimizer).

I think the question is this: one would expect that the pattern should include both initialized values and constant nodes. Here the check will reject when the value is produced by a constant node.

Maybe I misunderstood const_value attribute, when it is None and the value is initializer, it means that the value is produced by a constant node right ?

That would be inaccurate. A value is an initializer only when it is part of the graph.initializers mapping. You may use .is_graph_input() to check if the value is an input. You may use ir.convenience.get_const_tensor() to get the tensor value from either an initialized value or a value that comes from a Constant node.

Ok I see, thanks. While doing the change, the constant node gets disconnected from the graph but never removed. Should I also handle the removal of Constant nodes ? I thought the rewriter does it by default.

Yes. You can leave it to the rewriter or the final DCE pass, since it's not always safe to remove them during rewrites (it needs to have a single consumer which is disconnected)

github-project-automation bot added this to ONNX Script Review Board Jun 22, 2025

github-project-automation bot moved this to Todo in ONNX Script Review Board Jun 22, 2025

justinchuby requested a review from Copilot June 22, 2025 15:17

Copilot AI reviewed Jun 22, 2025

View reviewed changes

justinchuby reviewed Jun 23, 2025

View reviewed changes

AyoubMDL added 3 commits June 24, 2025 01:41

[Rewriter]: fuse successive Relu/Clip nodes

d827d63

- Relu(Relu(X)) -> Relu - Relu(Clip(X)) -> Clip - Clip(Relu(X)) -> Clip - Clip(Clip(X)) -> Clip

review: use ir APIs

be812d5

review: add rewriter to default rules

e0f6332

AyoubMDL force-pushed the fuse-sucessive-relus-clips branch from ab90aaf to e0f6332 Compare June 23, 2025 23:41

AyoubMDL mentioned this pull request Jun 24, 2025

Optimizer Suggestions for Common Graph Fusions #2301

Open

titaiwangms requested review from gramalingam and titaiwangms June 26, 2025 22:26

titaiwangms reviewed Jun 26, 2025

View reviewed changes

justinchuby added the module: rewriter label Jun 27, 2025

	base_model: onnx.ModelProto \| ir.Model,
	base_model: ir.Model,

	# check clip min/max are initializers
	# check if clip min/max are initializers

[Rewriter]: fuse successive Relu/Clip nodes #2410

Are you sure you want to change the base?

[Rewriter]: fuse successive Relu/Clip nodes #2410

Conversation

AyoubMDL commented Jun 22, 2025

Uh oh!

codecov bot commented Jun 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 13 Tests Failed:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinchuby left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

titaiwangms left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

titaiwangms Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

justinchuby Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov bot commented Jun 22, 2025 •

edited

Loading

justinchuby left a comment •

edited

Loading

titaiwangms Jun 26, 2025 •

edited

Loading

justinchuby Jun 28, 2025 •

edited

Loading