[system test] [perf] Performance perf report into PR #12124

see-quick · 2025-11-11T12:14:21Z

Type of change

Enhancement / new feature

Description

This PR adds a perf report, which might look like this [1] (ignore those numbers; important is the format). This report would always be appended to the PR as a message when performance tests are triggered.

Currently, I am using only one agent (i.e., ubuntu-latest) for testing purposes on my fork, but the plan is to use both x64 and arm-based agents.

[1] - see-quick#15 (comment)

Checklist

Write tests
Make sure all tests pass
Update documentation

codecov · 2025-11-11T13:22:14Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.78%. Comparing base (4503119) to head (42a8dd1).
⚠️ Report is 19 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12124      +/-   ##
============================================
- Coverage     74.81%   74.78%   -0.04%     
- Complexity     6619     6624       +5     
============================================
  Files           377      377              
  Lines         25329    25349      +20     
  Branches       3394     3398       +4     
============================================
+ Hits          18951    18957       +6     
- Misses         4991     5007      +16     
+ Partials       1387     1385       -2

see 12 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

.github/workflows/system-tests.yml

...test/src/test/java/io/strimzi/systemtest/performance/UserOperatorScalabilityPerformance.java

...est/src/test/java/io/strimzi/systemtest/performance/TopicOperatorScalabilityPerformance.java

...test/src/test/java/io/strimzi/systemtest/performance/UserOperatorScalabilityPerformance.java

.github/actions/systemtests/perf-report/action.yml

.github/actions/utils/add-to-summary/action.yml

.github/actions/utils/create-warning/action.yml

.github/actions/utils/list-artifacts/action.yml

Frawless · 2025-11-13T18:12:49Z

.github/actions/systemtests/run-perf-report/action.yml

+
+          ${{ steps.generate_report.outputs.summary }}
+
+    - name: Add performance report to job summary


This is cool, maybe there could be some usage for common sts execution as well?

see-quick · 2025-11-18T15:08:45Z

/gha run pipeline=performance

github-actions · 2025-11-18T15:09:26Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-18T16:52:49Z

❌ System test verification failed: link

see-quick · 2025-11-19T09:14:26Z

/gha run pipeline=performance

github-actions · 2025-11-19T09:15:05Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

Signed-off-by: see-quick <[email protected]>

github-actions · 2025-11-20T17:20:52Z

❌ System test verification failed: link

see-quick · 2025-11-20T20:34:38Z

/gha run pipeline=performance

github-actions · 2025-11-20T20:35:13Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-20T21:45:34Z

🎉 System test verification passed: link

Signed-off-by: see-quick <[email protected]>

see-quick · 2025-11-21T08:47:33Z

/gha run pipeline=performance

github-actions · 2025-11-21T08:48:06Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-21T10:16:05Z

🎉 System test verification passed: link

Signed-off-by: see-quick <[email protected]>

see-quick · 2025-11-24T11:47:50Z

/gha run pipeline=performance

github-actions · 2025-11-24T11:48:21Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-11-24T13:24:46Z

🎉 System test verification passed: link

Signed-off-by: see-quick <[email protected]>

.github/tests/workflows/perf-report-template.yaml

.github/actions/systemtests/run-perf-report/action.yml

Frawless · 2025-11-25T07:46:07Z

.github/actions/systemtests/run-perf-report/generate-report.js

+/**
+ * Find the latest timestamped results directory
+ */
+function findLatestResultsDir(baseDir) {


there are multiple results from one run that you need to find the latest?

Also, this was mainly for local testing where I had multiple directories and runs:

├── 2025-11-20-15-57-41 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-16-10-15 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-16-35-19 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-17-02-58 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-20-17-15-26 │ └── user-operator │ └── latencyUseCase │ ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- │ └── users-3000-tp--cache--bq--bb-100-bt-100-utp- ├── 2025-11-21-09-50-34 │ ├── topic-operator │ │ └── scalabilityUseCase │ │ ├── max-batch-size-100-max-linger-time-100-with-clients-false-number-of-topics-2 │ │ └── max-batch-size-100-max-linger-time-100-with-clients-false-number-of-topics-3 │ └── user-operator │ ├── latencyUseCase │ │ ├── users-10-tp--cache--bq--bb-100-bt-100-utp- │ │ ├── users-20-tp--cache--bq--bb-100-bt-100-utp- │ │ └── users-30-tp--cache--bq--bb-100-bt-100-utp- │ └── scalabilityUseCase │ ├── users-10-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-12-tp--cache--bq--bb-100-bt-100-utp- │ ├── users-14-tp--cache--bq--bb-100-bt-100-utp- │ └── users-16-tp--cache--bq--bb-100-bt-100-utp- └── 2025-11-24-12-16-22 └── user-operator └── latencyUseCase ├── users-1000-tp--cache--bq--bb-100-bt-100-utp- ├── users-2000-tp--cache--bq--bb-100-bt-100-utp- └── users-3000-tp--cache--bq--bb-100-bt-100-utp-

so it will always pick latest. I think I can re-name that method (to something like that findTimestampedResultsDir``)? But in general per one architecture there should be just only ONE timestamp`. If more architectures are run then we handle it differently... (we don't care about that here).

.github/actions/systemtests/run-perf-report/generate-report.js

.github/tests/workflows/perf-report-template.yaml

Signed-off-by: see-quick <[email protected]>

see-quick · 2025-12-01T10:14:35Z

/gha run pipeline=performance

github-actions · 2025-12-01T10:15:15Z

⏳ System test verification started: link

The following 2 job(s) will be executed:

performance-amd64 (oracle-vm-8cpu-32gb-x86-64)
performance-arm64 (oracle-vm-8cpu-32gb-arm64)

Tests will start after successful build completion.

github-actions · 2025-12-01T11:31:41Z

🎉 System test verification passed: link

Frawless

The changes are fine form my POV. I am not sure with 450 lines of js to generate the report especially when we have something similar already in the tests. AFAIU it is not trivial to re-use it. I wonder what others think about it. Otherwise I am fine with it as long as it will be used :)

see-quick · 2025-12-02T14:00:05Z

Basically, there are two approaches... the first one is (where we would have +500 LOC to merge two or more results from architectures):

Performance Test Results

Test Run: 2025-11-12 20:47

Topic Operator

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms) [AMD64]	Reconciliation interval (ms) [ARM64]
1	2	8	10229	10167
2	32	98	11505	10504
3	125	375	42367	41202
4	250	750	74596	72361

User Operator

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms) [AMD64]	Reconciliation interval (ms) [ARM64]
1	10	10472	10797
2	100	33036	33851
3	200	54940	55822
4	500	133782	135474

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms) [AMD64]	Min Latency (ms) [ARM64]	Max Latency (ms) [AMD64]	Max Latency (ms) [ARM64]	Average Latency (ms) [AMD64]	Average Latency (ms) [ARM64]	P50 Latency (ms) [AMD64]	P50 Latency (ms) [ARM64]	P95 Latency (ms) [AMD64]	P95 Latency (ms) [ARM64]	P99 Latency (ms) [AMD64]	P99 Latency (ms) [ARM64]
1	110	12	14	69	103	27.78	25.03	26	22	39	45	54	79
2	200	11	15	75	66	29.93	27.13	28	25	48	45	75	59
3	300	10	12	61	98	26.0	25.53	26	23	41	41	50	89

see-quick · 2025-12-02T14:01:06Z

or second one have (but with no need + 450LOC of javascript) => but with price with a lot of redudancy from my POV:

Performance Test Results

AMD64

Test Run: 2025-11-18 14:43

Topic Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	300	14	51	24.24	22	38	49
3	200	13	72	25.56	23	44	57

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms)
1	2	8	10130
2	32	98	10441
3	125	375	41369
4	250	750	71977

User Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	200	13	72	25.56	23	44	57
3	300	14	51	24.24	22	38	49

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms)
1	10	10641
2	100	33107
3	200	54157
4	500	134269

ARCH (another arch)

Test Run: 2025-11-18 14:49

Topic Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	300	14	51	24.24	22	38	49
3	200	13	72	25.56	23	44	57

Use Case: scalabilityUseCase

Configuration:

MAX QUEUE SIZE: 2147483647
MAX BATCH SIZE (ms): 100
MAX BATCH LINGER (ms): 100
PROCESS TYPE: TOPIC-CONCURRENT

Results:

#	NUMBER OF TOPICS	NUMBER OF EVENTS	Reconciliation interval (ms)
1	2	8	10130
2	32	98	10441
3	125	375	41369
4	250	750	71977

User Operator

Use Case: latencyUseCase

Configuration:

WORK_QUEUE_SIZE: 2048
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Min Latency (ms)	Max Latency (ms)	Average Latency (ms)	P50 Latency (ms)	P95 Latency (ms)	P99 Latency (ms)
1	110	14	75	27.51	24	51	73
2	200	13	72	25.56	23	44	57
3	300	14	51	24.24	22	38	49

Use Case: scalabilityUseCase

Configuration:

WORK_QUEUE_SIZE: 1024
BATCH_MAXIMUM_BLOCK_SIZE: 100
BATCH_MAXIMUM_BLOCK_TIME_MS: 100

Results:

#	NUMBER OF KAFKA USERS	Reconciliation interval (ms)
1	10	10641
2	100	33107
3	200	54157
4	500	134269

see-quick added this to the 0.50.0 milestone Nov 11, 2025

see-quick self-assigned this Nov 11, 2025

see-quick added System tests performance labels Nov 11, 2025

see-quick requested review from a team and Frawless November 11, 2025 12:15

see-quick force-pushed the perf-report-st branch from 6ef694a to aaf5e72 Compare November 13, 2025 09:23

Frawless reviewed Nov 13, 2025

View reviewed changes

see-quick force-pushed the perf-report-st branch from 7fa2d0c to cea51a6 Compare November 18, 2025 15:02

see-quick added 15 commits November 19, 2025 10:41

[system test] [perf] Performance perf report into PR

54ea727

Signed-off-by: see-quick <[email protected]>

correct doc after rebase

e24a841

Signed-off-by: see-quick <[email protected]>

also add oracle agent instead ubuntu latest

af0ee3a

Signed-off-by: see-quick <[email protected]>

check 2 agents

83de883

Signed-off-by: see-quick <[email protected]>

correct javadoc

975dc3d

Signed-off-by: see-quick <[email protected]>

move logic stuff to actions and also lower the creation

cdee595

Signed-off-by: see-quick <[email protected]>

do not merge perf tests within workflow

9425e2f

Signed-off-by: see-quick <[email protected]>

some fix

bfd5afa

Signed-off-by: see-quick <[email protected]>

have epthemeral storage

60af3fc

Signed-off-by: see-quick <[email protected]>

also imports

2288b8a

Signed-off-by: see-quick <[email protected]>

try both arches to display

1df141f

Signed-off-by: see-quick <[email protected]>

[system test] [perf] Performance perf report into PR

76c1da9

Signed-off-by: see-quick <[email protected]>

refactor a bit

6beb9a6

Signed-off-by: see-quick <[email protected]>

[system test] [perf] Performance perf report into PR 4

eee3dd3

Signed-off-by: see-quick <[email protected]>

fix actions

44c0c95

Signed-off-by: see-quick <[email protected]>

do not limit cpu

b1b05a5

Signed-off-by: see-quick <[email protected]>

see-quick added 2 commits November 24, 2025 09:51

keep IN: and OUT: parameters within results-table.txt

84ee02b

Signed-off-by: see-quick <[email protected]>

adjust cpus specific to GHA runners

ab130c5

Signed-off-by: see-quick <[email protected]>

see-quick added 2 commits November 24, 2025 14:29

modify input tests in perf report

4423660

Signed-off-by: see-quick <[email protected]>

update perf-report expected results

c5a05b1

Signed-off-by: see-quick <[email protected]>

see-quick requested a review from Frawless November 24, 2025 13:41

Frawless reviewed Nov 25, 2025

View reviewed changes

see-quick added 5 commits December 1, 2025 10:22

Jakub review + fix tests + re-name .txt to .md + refactor generation.js

a347815

Signed-off-by: see-quick <[email protected]>

remove ref from run-perf-report step

a4f356e

Signed-off-by: see-quick <[email protected]>

udpate workflow of tests

c5c416c

Signed-off-by: see-quick <[email protected]>

try

55fcd5f

Signed-off-by: see-quick <[email protected]>

a few leftover and ready to review again

42a8dd1

Signed-off-by: see-quick <[email protected]>

Frawless approved these changes Dec 2, 2025

View reviewed changes

see-quick requested review from a team and im-konge December 2, 2025 13:55


		${{ steps.generate_report.outputs.summary }}

		- name: Add performance report to job summary

[system test] [perf] Performance perf report into PR #12124

Are you sure you want to change the base?

[system test] [perf] Performance perf report into PR #12124

Uh oh!

Conversation

see-quick commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Type of change

Description

Checklist

Uh oh!

codecov bot commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frawless Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

see-quick commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

see-quick commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 20, 2025

Uh oh!

see-quick commented Nov 20, 2025

Uh oh!

github-actions bot commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 20, 2025

Uh oh!

see-quick commented Nov 21, 2025

Uh oh!

github-actions bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 21, 2025

Uh oh!

see-quick commented Nov 24, 2025

Uh oh!

github-actions bot commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Frawless Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

see-quick Nov 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

see-quick commented Dec 1, 2025

Uh oh!

github-actions bot commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 1, 2025

Uh oh!

see-quick commented Nov 11, 2025 •

edited

Loading

codecov bot commented Nov 11, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 19, 2025 •

edited

Loading

github-actions bot commented Nov 20, 2025 •

edited

Loading

github-actions bot commented Nov 21, 2025 •

edited

Loading

github-actions bot commented Nov 24, 2025 •

edited

Loading

github-actions bot commented Dec 1, 2025 •

edited

Loading

see-quick commented Dec 2, 2025 •

edited

Loading