test-backend-ops: add support for specifying output format #14368

yeahdongcn · 2025-06-25T03:36:31Z

Make sure to read the contributing guidelines before submitting a PR

Testing Done

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops --help
Usage: ./build/bin/test-backend-ops [mode] [-o <op>] [-b <backend>] [-p <params regex>] [--output <console|sql>]
    valid modes:
      - test (default, compare with CPU backend for correctness)
      - grad (compare gradients from backpropagation with method of finite differences)
      - perf (performance evaluation)
    op names for -o are as given by ggml_op_desc() (e.g. ADD, MUL_MAT, etc)
    --output specifies output format (default: console)

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
Testing 2 devices

Backend 1/2: MUSA0
  Skipping
Backend 2/2: CPU
  Device description: 12th Gen Intel(R) Core(TM) i5-12400
  Device memory: 31859 MB (31859 MB free)

  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]):                1572480 runs -     0.64 us/run -       48 kB/run -   71.97 GB/s
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):   ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]):                13338 runs -    76.12 us/run -    24576 kB/run -  308.22 GB/s
  Backend CPU: OK

2/2 backends passed
OK

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes
CREATE TABLE IF NOT EXISTS test_backend_ops (
  test_time TEXT,
  build_commit TEXT,
  build_number INTEGER,
  backend_name TEXT,
  op_name TEXT,
  op_params TEXT,
  test_mode TEXT,
  supported INTEGER,
  passed INTEGER,
  error_message TEXT,
  time_us REAL,
  flops REAL,
  bandwidth_gb_s REAL,
  memory_kb INTEGER,
  n_runs INTEGER
);

INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:54Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,1,1,1]', 'perf', '1', '1', '', '1.003772', '0.000000', '45.608051', '48', '1007370');
INSERT INTO test_backend_ops (test_time, build_commit, build_number, backend_name, op_name, op_params, test_mode, supported, passed, error_message, time_us, flops, bandwidth_gb_s, memory_kb, n_runs) VALUES ('2025-06-27T05:54:55Z', '1d5f25c53', '5756', 'CPU', 'ADD', 'type=f32,ne=[4096,1,1,1],nr=[1,512,1,1]', 'perf', '1', '1', '', '134.092504', '0.000000', '174.956746', '24576', '7524');

root@xiaodongye-s80:/ws# ./build/bin/test-backend-ops perf -b CPU -o ADD --output sql | sqlite3 add.sqlite
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 MUSA devices:
  Device 0: MTT S80, compute capability 2.1, VMM: yes

Edit (added build_commit and build_number for future use):

Copilot

Pull Request Overview

This PR adds support for specifying the output format (console or SQL) for the test-backend-ops tool and unifies logging via a new printer interface.

Introduces a printer interface with concrete implementations for console and SQL output.
Updates test evaluation functions and the CLI to use the printer for logging.
Adds new command-line option "--output" along with helper functions to parse the output format.

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

JohannesGaessler

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

tests/test-backend-ops.cpp

JohannesGaessler · 2025-06-25T12:47:47Z

tests/test-backend-ops.cpp

+        passed = false;
+
+        // Set test time
+        time_t t = time(NULL);


This was added to record the timestamp (the same as llama-bench), but I haven’t decided how to use it.

tests/test-backend-ops.cpp

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn · 2025-06-26T04:42:19Z

From my end I think these changes to test-backend-ops would be fine but keep in mind that it's an important piece of the project with many stakeholders.

Thanks for pointing that out! I’ll add the recent contributors as reviewers.

slaren · 2025-06-30T10:33:43Z

tests/test-backend-ops.cpp

+// External declarations for build info
+extern int LLAMA_BUILD_NUMBER;
+extern const char *LLAMA_COMMIT;


test-backend-ops is part of ggml, and cannot depend on llama.cpp.

The main goal of this work is to generate a comparison table for test-backend-ops, which I believe is especially useful when updating or optimizing op implementations. For more context, please refer to #14354.

The commit hash is used to compare performance changes between two revisions, as shown in #14392.

Do you have any suggestions on how to update this? Thanks.

There are already GGML_BUILD_NUMBER and GGML_BUILD_COMMIT variables in the ggml build. They are currently not passed to the code, but you pass them through a compile_definition.

I noticed that the generated build-info.cpp contains the same commit hash and build number for both LLAMA and GGML:

int LLAMA_BUILD_NUMBER = 5757; char const *LLAMA_COMMIT = "38d4930ec"; char const *LLAMA_COMPILER = "Apple clang version 17.0.0 (clang-1700.0.13.5)"; char const *LLAMA_BUILD_TARGET = "arm64-apple-darwin24.5.0"; int GGML_BUILD_NUMBER = 5757; char const *GGML_BUILD_COMMIT = "38d4930ec";

Please see the new commit: 76ca4f6 Thanks.

No, this way still depends on the llama.cpp common lib.

slaren · 2025-06-30T13:21:44Z

tests/test-backend-ops.cpp

+    // General purpose output methods
+    virtual void print_info(const char * format, ...) = 0;
+    virtual void print_error(const char * format, ...) = 0;
+    virtual void print_device_info(const char * format, ...) = 0;
+    virtual void print_test_summary(const char * format, ...) = 0;
+    virtual void print_status_ok() = 0;
+    virtual void print_status_fail() = 0;


I don't think this a good design, all of these functions do the same and could be replaced with a single print_message. If you want transfer the responsibility of formatting the output to a class, then the class needs to have the information to decide what to print, not an unstructured message. Ideally all of the functions would receive an object to print, and the class would determine how to format that object.

Sounds good. I’ll update the code accordingly.

Please check the latest commit. Thanks.

This doesn't really change anything. The ideal solution would be to remove all the messages entirely, and pass enough information to the printers so that they format the output in any way they want. That would require a significant refactor, and not simply replacing calls to printf with printer->print_message.

Please check if I understand this correctly: f4f5512
Thanks.

Signed-off-by: Xiaodong Ye <[email protected]>

github-actions bot added the testing Everything test related label Jun 25, 2025

yeahdongcn mentioned this pull request Jun 25, 2025

Add script to test op perf and compare #14354

Open

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from bfa7a43 to 359d792 Compare June 25, 2025 08:21

yeahdongcn requested a review from Copilot June 25, 2025 08:44

This comment was marked as outdated.

Sign in to view

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 359d792 to 34500f9 Compare June 25, 2025 09:16

yeahdongcn requested a review from Copilot June 25, 2025 09:19

Copilot AI reviewed Jun 25, 2025

View reviewed changes

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

tests/test-backend-ops.cpp Outdated Show resolved Hide resolved

test-backend-ops: add support for specifying output format

679a141

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 34500f9 to 679a141 Compare June 25, 2025 09:46

yeahdongcn marked this pull request as ready for review June 25, 2025 09:47

JohannesGaessler reviewed Jun 25, 2025

View reviewed changes

yeahdongcn added 2 commits June 26, 2025 12:39

Address review comments

bea01ea

Signed-off-by: Xiaodong Ye <[email protected]>

Add build_commit and build_number in test_result

1d5f25c

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn requested review from am17an, ggerganov and ngxson June 26, 2025 04:42

ggerganov requested a review from slaren June 26, 2025 08:06

yeahdongcn mentioned this pull request Jun 26, 2025

compare-commits.sh: support both llama-bench and test-backend-ops #14392

Open

slaren reviewed Jun 30, 2025

View reviewed changes

Address review comments

38d4930

Signed-off-by: Xiaodong Ye <[email protected]>

slaren mentioned this pull request Jul 1, 2025

ggml : add version function to get lib version ggml-org/ggml#1286

Open

refactor

f4f5512

Signed-off-by: Xiaodong Ye <[email protected]>

yeahdongcn force-pushed the xd/test-backend-ops_sql branch from 76ca4f6 to f4f5512 Compare July 2, 2025 02:10

test-backend-ops: add support for specifying output format #14368

Are you sure you want to change the base?

test-backend-ops: add support for specifying output format #14368

Uh oh!

Conversation

yeahdongcn commented Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing Done

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeahdongcn Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeahdongcn commented Jun 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slaren Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeahdongcn commented Jun 25, 2025 •

edited

Loading

yeahdongcn Jun 26, 2025 •

edited

Loading

slaren Jun 30, 2025 •

edited

Loading