issue/787 - Split run ops test logic and fix kwargs name in report #788

baominghelly · 2025-12-16T02:32:01Z

Revision 2

Description

func behavior fix and class rename

Test evidence

We comment torch_operator method to create a skipped test case, in this way we have a success case, a partial case and a skipped case.
Command:

python test/infinicore/run.py --ops add acosh acos --nvidia --bench --save

Output:

......
================================================================================
CUMULATIVE TEST SUMMARY
================================================================================
Total tests run: 3
Passed: 1
Failed: 0
Skipped: 1
Partial: 1
----------------------------------------
  PyTorch Host Total Time:          884.667 ms
  InfiniCore Host Total Time:      2419.153 ms
  PyTorch Device Total Time:      10948.652 ms
  InfiniCore Device Total Time:     2410.264 ms
----------------------------------------

✅ PASSED OPERATORS (1):
  add

⏭️  SKIPPED OPERATORS (1):
  acos

⚠️  PARTIAL IMPLEMENTATIONS (1):
  acosh

Success rate: 50.0%

⚠️  Tests completed with some operators not fully implemented

⚠️  Note: Some operators are not fully implemented
   Run individual tests for details on missing implementations

Help info

python test/infinicore/run.py --help
usage: run.py [-h] [--ops-dir OPS_DIR] [--ops OPS [OPS ...]] [--list] [--bench [{host,device,both}]] [--debug]
              [--eq_nan] [--verbose] [--save [SAVE]] [--cpu] [--nvidia] [--cambricon] [--ascend] [--iluvatar]
              [--metax] [--moore] [--kunlun] [--hygon] [--qy]

Run InfiniCore operator tests across multiple hardware platforms

options:
  -h, --help            show this help message and exit
  --ops-dir OPS_DIR     Path to the ops directory (default: auto-detect)
  --ops OPS [OPS ...]   Run specific operators only (e.g., --ops add matmul)
  --list                List all available test files without running them

Common Execution Options:
  --bench [{host,device,both}]
                        Enable performance benchmarking mode. Options: host (CPU time only), device (GPU time only),
                        both (default)
  --debug               Enable debug mode for detailed tensor comparison
  --eq_nan              Enable equal_nan for tensor comparison
  --verbose             Enable verbose mode to stop on first error with full traceback
  --save [SAVE]         Save test results to a JSON file. Defaults to 'test_report.json' if no filename provided.

Hardware Platform Options:
  --cpu                 Standard CPU execution
  --nvidia              NVIDIA GPUs with CUDA support
  --cambricon           Cambricon MLU accelerators (requires torch_mlu)
  --ascend              Huawei Ascend NPUs (requires torch_npu)
  --iluvatar            Iluvatar GPUs
  --metax               Metax GPUs
  --moore               Moore Threads GPUs (requires torch_musa)
  --kunlun              Kunlun XPUs (requires torch_xmlir)
  --hygon               Hygon DCUs
  --qy                  QY GPUs

Examples:
  # Run all operator tests on CPU
  python run.py --cpu

  # Run specific operators
  python run.py --ops add matmul --nvidia

  # Run with debug mode on multiple devices
  python run.py --cpu --nvidia --debug

  # Run with verbose mode to stop on first error with full traceback
  python run.py --cpu --nvidia --verbose

  # Run with benchmarking (both host and device timing)
  python run.py --cpu --bench

  # Run with host timing only
  python run.py --nvidia --bench host

  # Run with device timing only
  python run.py --nvidia --bench device

  # List available tests without running
  python run.py --list

Available Operators:
  abs, acos, acosh, adaptive_avg_pool1d
 ...
  var_mean, vdot, where

Note:
  - Use '--' to pass additional arguments to individual test scripts
  - Operators are automatically discovered from the ops directory
  - --bench mode now shows cumulative timing across all operators
  - --bench host/device/both controls host/device timing measurement
  - --verbose mode stops execution on first error and shows full traceback

Description

Split run ops test logic and fix kwargs name in report

Test evidence

python test/infinicore/run.py --ops add matmul --nvidia --bench --save
InfiniCore Operator Test Runner
Directory: /home/baoming/workplace/InfiniCore/test/infinicore/ops
Tests found: 2

✅  matmul: PASSED (code: 0)

============================================================
Testing Matmul on NVIDIA
============================================================
TestCase(Matmul - OUT_OF_PLACE - inputs=[in_0: tensor(2, 3), float16; in_1: tensor(3, 4), float16])
    PyTorch    time - Host: 0.009846 ms, Device: 0.086326 ms
    InfiniCore time - Host: 0.009691 ms, Device: 0.012473 ms
✓ Passed
...
TestCase(Matmul - OUT_OF_PLACE - inputs=[in_0: tensor(8, 16, 16), float32; in_1: tensor(8, 16, 32), float32])
    PyTorch    time - Host: 0.012982 ms, Device: 0.079762 ms
    InfiniCore time - Host: 0.009568 ms, Device: 0.013310 ms
✓ Passed
TestCase(Matmul - INPLACE(out) - inputs=[in_0: tensor(8, 16, 16), float32; in_1: tensor(8, 16, 32), float32], kwargs={out=tensor(8, 16, 32), float32})
    PyTorch    time - Host: 0.011029 ms, Device: 0.083062 ms
    InfiniCore time - Host: 0.007850 ms, Device: 0.012125 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 42
Passed: 42
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 627.367 ms
PyTorch Device Total Time: 3995.345 ms
InfiniCore Host Total Time: 604.686 ms
InfiniCore Device Total Time: 803.978 ms
Host Speedup (PyTorch/InfiniCore): 1.04x
Device Speedup (PyTorch/InfiniCore): 4.97x
============================================================
💾 Saving to: test_report_20251216_102050_237.json
   ✅ Saved (Structure Matched).
----------------------------------------
✅  add: PASSED (code: 0)

============================================================
Testing Add on NVIDIA
============================================================
TestCase(Add - OUT_OF_PLACE - inputs=[a: tensor(13, 4), float16; b: tensor(13, 4), float16])
    PyTorch    time - Host: 0.009359 ms, Device: 0.080803 ms
    InfiniCore time - Host: 0.013330 ms, Device: 0.016580 ms
✓ Passed
...
TestCase(Add - INPLACE(out) - inputs=[a: tensor(16, 5632), strides=(13312, 1), float32; b: tensor(16, 5632), strides=(13312, 1), float32], kwargs={out=c: tensor(16, 5632), float32})
    PyTorch    time - Host: 0.008407 ms, Device: 0.113156 ms
    InfiniCore time - Host: 0.022931 ms, Device: 0.020717 ms
✓ Passed
TestCase(Add - INPLACE(a) - inputs=[a: tensor(16, 5632), strides=(13312, 1), float32; b: tensor(16, 5632), strides=(13312, 1), float32], kwargs={out=a})
    PyTorch    time - Host: 0.008720 ms, Device: 0.101268 ms
    InfiniCore time - Host: 0.020859 ms, Device: 0.024816 ms
✓ Passed
TestCase(Add - INPLACE(b) - inputs=[a: tensor(16, 5632), strides=(13312, 1), float32; b: tensor(16, 5632), strides=(13312, 1), float32], kwargs={out=b})
    PyTorch    time - Host: 0.008292 ms, Device: 0.111994 ms
    InfiniCore time - Host: 0.023171 ms, Device: 0.023727 ms
✓ Passed

============================================================
TEST SUMMARY
Total tests: 102
Passed: 102
Success rate: 100.0%

All tests passed!
------------------------------------------------------------
BENCHMARK SUMMARY
PyTorch Host Total Time: 845.552 ms
PyTorch Device Total Time: 11067.439 ms
InfiniCore Host Total Time: 2176.486 ms
InfiniCore Device Total Time: 2476.742 ms
Host Speedup (PyTorch/InfiniCore): 0.39x
Device Speedup (PyTorch/InfiniCore): 4.47x
============================================================
💾 Saving to: test_report_20251216_102112_161.json
   ✅ Saved (Structure Matched).
----------------------------------------

================================================================================
CUMULATIVE TEST SUMMARY
================================================================================
Total tests run: 2
Passed: 2
Failed: 0
----------------------------------------
BENCHMARK SUMMARY:
  Operators Tested: 0
  PyTorch Host Total Time:         1472.919 ms
  InfiniCore Host Total Time:      2781.171 ms
  PyTorch Device Total Time:      15062.784 ms
  InfiniCore Device Total Time:     3280.720 ms
----------------------------------------

✅ PASSED OPERATORS (2):
  matmul, add

Success rate: 100.0%

🎉 All tests passed!

json string in test report

[
    {
        "operator": "Matmul",
        "device": "NVIDIA",
        "torch_op": "torch.matmul",
        "infinicore_op": "infinicore.matmul",
        "args": {"bench": "both", "num_prerun": 10, "num_iterations": 1000, "verbose": false, "debug": false},
        "testcases": [
            {
                "description": "Matmul - OUT_OF_PLACE",
                "inputs": [{"name": "in_0", "shape": [2, 3], "dtype": "float16", "strides": null}, {"name": "in_1", "shape": [3, 4], "dtype": "float16", "strides": null}],
                "kwargs": {},
                "comparison_target": null, "tolerance": {"atol": 0, "rtol": 0.01},
                "result": {"status": {"success": true, "error": ""}, "perf_ms": {"torch": {"host": 9.8462, "device": 86.3263}, "infinicore": {"host": 9.6915, "device": 12.4733}}}
            },
            {
                "description": "Matmul - INPLACE(out)",
                "inputs": [{"name": "in_0", "shape": [2, 3], "dtype": "float16", "strides": null}, {"name": "in_1", "shape": [3, 4], "dtype": "float16", "strides": null}],
                "kwargs": {"out": {"name": null, "shape": [2, 4], "dtype": "float16", "strides": null}},
                "comparison_target": "out", "tolerance": {"atol": 0, "rtol": 0.01},
                "result": {"status": {"success": true, "error": ""}, "perf_ms": {"torch": {"host": 9.2163, "device": 82.0994}, "infinicore": {"host": 10.1438, "device": 18.0714}}}
            },
            {
                "description": "Matmul - OUT_OF_PLACE",
                "inputs": [{"name": "in_0", "shape": [2, 3], "dtype": "bfloat16", "strides": null}, {"name": "in_1", "shape": [3, 4], "dtype": "bfloat16", "strides": null}],
                "kwargs": {},
                "comparison_target": null, "tolerance": {"atol": 0, "rtol": 0.05},
                "result": {"status": {"success": true, "error": ""}, "perf_ms": {"torch": {"host": 9.5227, "device": 84.3846}, "infinicore": {"host": 9.2583, "device": 14.0242}}}
            }
        ]
    }
]

wooway777

run.py的行为跟之前不一致了，需要确保主要行为完全一致

test/infinicore/framework/datatypes.py

wooway777 · 2025-12-16T10:46:38Z

test/infinicore/framework/datatypes.py

+    operators_tested: int = 0
+
+@dataclass
+class SingleTestResult:


命名是不是没必要加这个Single

test/infinicore/framework/executor.py

wooway777 · 2025-12-16T10:58:15Z

另外需要看一下，这些结果和信息打印的函数，能否跟现有的整合一下。

baominghelly · 2025-12-18T07:07:32Z

run.py的行为跟之前不一致了，需要确保主要行为完全一致

Sure, will compare the result with old run.py.

baominghelly · 2025-12-18T07:08:31Z

另外需要看一下，这些结果和信息打印的函数，能否跟现有的整合一下。

This pr includes so much changes, I prefer to push this change and TestResult name change to next pr.

wooway777

先改一轮吧。

以及我觉得，print、report、以及很多analyze和extract之类的方法，都是用来处理结果的。可以整合到一个类里面。

原则上讲，如果之前的定义不能满足现在的要求了，就去修改以前的东西，使得同样的信息可以从头用到尾。而不是写一个新的类似的东西把老的定义包起来。不然框架越写越大就不对劲了。

wooway777 · 2025-12-19T01:20:41Z

test/infinicore/framework/datatypes.py

 import torch
 import infinicore
-
+from dataclasses import dataclass, field


wooway777 · 2025-12-19T01:28:24Z

test/infinicore/framework/printer.py

@@ -0,0 +1,180 @@
+# lib/printer.py


此行可删

wooway777 · 2025-12-19T01:34:45Z

test/infinicore/framework/types.py

@@ -0,0 +1,52 @@
+from dataclasses import dataclass, field


这东西肯定不叫types，大概structs好一点？

wooway777 · 2025-12-19T01:38:15Z

test/infinicore/framework/types.py

+from dataclasses import dataclass, field
+from typing import Any
+
+# TODO: Rename it, current class name is abstract.


这个todo是啥？

wooway777 · 2025-12-19T01:42:43Z

test/infinicore/framework/types.py

+    operators_tested: int = 0 
+
+@dataclass
+class OperatorTestResult:


目前这个设计还是很冗余。

有一个TestResult，然后还有一个OperatorTestResult，他俩信息重合了80%，多了两个方法；
然后这两个方法做的事情基本没区别。符号信息可以print的时候再加。
而且现在是以前的东西先反回一个result，然后再parse成新的result···
这两个result肯定可以整合一下。

wooway777 · 2025-12-19T01:43:03Z

test/infinicore/run.py

+        sys.exit(0)
+
+    # 2. Preparation
+    dirver = TestDriver()


wooway777 · 2025-12-19T01:45:23Z

test/infinicore/framework/printer.py

@@ -0,0 +1,180 @@
+# lib/printer.py


文件可以就叫summary.py

loader printer reporter多了就不专业了

test/infinicore/framework/driver.py

wooway777 · 2025-12-19T02:01:40Z

另外需要看一下，这些结果和信息打印的函数，能否跟现有的整合一下。

This pr includes so much changes, I prefer to push this change and TestResult name change to next pr.

现在的主要问题是，设计不合理合不了

Copilot

Pull request overview

This PR refactors the InfiniCore operator test runner by splitting monolithic test logic into modular framework components and fixes kwargs name handling in the test reporter.

Key Changes:

Extracted test discovery, execution, and reporting logic into separate framework modules (loader.py, driver.py, printer.py)
Moved TestResult class from entities.py to new types.py module for better organization
Enhanced kwargs name handling in reporter.py to properly display input/output tensor names in test reports

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 17 comments.

Show a summary per file

File	Description
test/infinicore/run.py	Refactored from 567 lines to 194 lines by delegating logic to framework components; now acts as a thin orchestration layer
test/infinicore/framework/types.py	New file defining core data structures: `TestResult`, `TestTiming`, and `OperatorTestResult`
test/infinicore/framework/loader.py	New `TestDiscoverer` class handling operator test file discovery and validation
test/infinicore/framework/driver.py	New `TestDriver` class managing test execution, output capture, and result aggregation
test/infinicore/framework/printer.py	New `ConsolePrinter` class handling all console output formatting and summaries
test/infinicore/framework/reporter.py	Enhanced kwargs handling to properly resolve and display tensor names instead of null/index values
test/infinicore/framework/entities.py	Removed `TestResult` class (moved to types.py)
test/infinicore/framework/base.py	Updated import to use `TestResult` from `types` module
test/infinicore/framework/init.py	Added exports for new types and classes
test/infinicore/framework/datatypes.py	Minor formatting change (added import statement)

Comments suppressed due to low confidence (1)

test/infinicore/run.py:145

This assignment to 'target_ops' is unnecessary as it is redefined before this value is used.

            target_ops = valid_ops

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-12-19T02:02:56Z

test/infinicore/framework/printer.py

+             self._print_op_list("⏭️  SKIPPED OPERATORS", skipped)
+
+        # PARTIAL
+        if partial:
+             self._print_op_list("⚠️  PARTIAL IMPLEMENTATIONS", partial)


Inconsistent indentation: This line has an extra space before 'self._print_op_list', making it have 13 spaces instead of 12. This should match the indentation of the surrounding lines for consistency.

Suggested change

self._print_op_list("⏭️ SKIPPED OPERATORS", skipped)

# PARTIAL

if partial:

self._print_op_list("⚠️ PARTIAL IMPLEMENTATIONS", partial)

self._print_op_list("⏭️ SKIPPED OPERATORS", skipped)

# PARTIAL

if partial:

self._print_op_list("⚠️ PARTIAL IMPLEMENTATIONS", partial)

Copilot · 2025-12-19T02:02:56Z

test/infinicore/framework/types.py

+from dataclasses import dataclass, field
+from typing import Any
+
+# TODO: Rename it, current class name is abstract.


TODO comment should not be left in production code. Either rename the class to something more specific (e.g., 'OperatorTestResultDetail', 'TestCaseResult', or 'SingleTestResult') or remove the TODO if the current name is acceptable.

Suggested change

# TODO: Rename it, current class name is abstract.

Copilot · 2025-12-19T02:02:57Z

test/infinicore/run.py

        bench_mode = args.bench if args.bench != "both" else "both"
        print(f"Benchmark mode: {bench_mode.upper()} timing")
+
+    target_ops = None


Variable target_ops is initialized here but not used before being potentially reassigned in the conditional blocks below. Consider initializing it only once before the conditional logic or removing this initialization since it's assigned in all branches.

test/infinicore/run.py

Copilot · 2025-12-19T02:02:57Z

test/infinicore/run.py

+        target_ops = valid_ops
+


Duplicate assignment to target_ops. This line is redundant as target_ops is already assigned on line 145 within the else block. The variable is being reassigned to the same value immediately after.

Suggested change

target_ops = valid_ops

Copilot · 2025-12-19T02:03:02Z

test/infinicore/framework/datatypes.py

+from dataclasses import dataclass, field



Import of 'dataclass' is not used.
Import of 'field' is not used.

Suggested change

from dataclasses import dataclass, field

Copilot · 2025-12-19T02:03:02Z

test/infinicore/framework/driver.py

+import importlib.util
+from io import StringIO
+from contextlib import contextmanager
+from .types import OperatorTestResult, TestTiming


Import of 'TestTiming' is not used.

Suggested change

from .types import OperatorTestResult, TestTiming

from .types import OperatorTestResult

Copilot · 2025-12-19T02:03:02Z

test/infinicore/framework/printer.py

@@ -0,0 +1,180 @@
+# lib/printer.py
+import sys


Import of 'sys' is not used.

Suggested change

import sys

Copilot · 2025-12-19T02:03:03Z

test/infinicore/framework/printer.py

@@ -0,0 +1,180 @@
+# lib/printer.py
+import sys
+from .types import OperatorTestResult, TestTiming


Import of 'TestTiming' is not used.
Import of 'OperatorTestResult' is not used.

Copilot · 2025-12-19T02:03:03Z

test/infinicore/framework/loader.py

+                return "infinicore" in content and (
+                    "BaseOperatorTest" in content or "GenericTestRunner" in content
+                )
+        except:


Except block directly handles BaseException.

Suggested change

except:

except Exception:

issue/787 - Split run ops test logic and fix kwargs name in report

0d58c82

baominghelly requested a review from wooway777 December 16, 2025 02:32

baominghelly linked an issue Dec 16, 2025 that may be closed by this pull request

[DEV] 分拆run.py逻辑，化简代码 #787

Open

wooway777 requested changes Dec 16, 2025

View reviewed changes

func behavior fix && rename class name

5aa850a

wooway777 requested changes Dec 19, 2025

View reviewed changes

wooway777 requested a review from Copilot December 19, 2025 01:59

Copilot started reviewing on behalf of wooway777 December 19, 2025 01:59 View session

Copilot AI reviewed Dec 19, 2025

View reviewed changes

	from .types import OperatorTestResult, TestTiming
	from .types import OperatorTestResult

issue/787 - Split run ops test logic and fix kwargs name in report #788

Are you sure you want to change the base?

issue/787 - Split run ops test logic and fix kwargs name in report #788

Uh oh!

Conversation

baominghelly commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Revision 2

Description

Test evidence

Description

Test evidence

Uh oh!

wooway777 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wooway777 Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wooway777 commented Dec 16, 2025

Uh oh!

baominghelly commented Dec 18, 2025

Uh oh!

baominghelly commented Dec 18, 2025

Uh oh!

wooway777 left a comment

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

wooway777 Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wooway777 commented Dec 19, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 19, 2025

baominghelly commented Dec 16, 2025 •

edited

Loading

wooway777 Dec 16, 2025 •

edited

Loading