-
Notifications
You must be signed in to change notification settings - Fork 253
feat(benchmarking): enable tests to run in dedicated environment or in docker #3157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
59 commits
Select commit
Hold shift + click to select a range
81dc810
refactor: move spamoor benchmark into testify suite in test/e2e/bench…
chatton cc56590
merge main into cian/bench-refactor
chatton 18fc15a
fix: correct BENCH_JSON_OUTPUT path for spamoor benchmark
chatton fccd9db
fix: place package pattern before test binary flags in benchmark CI
chatton ae525ca
fix: adjust evm-binary path for benchmark subpackage working directory
chatton 039eaf7
wip: erc20 benchmark test
chatton 85c9d2d
fix: exclude benchmark subpackage from make test-e2e
chatton fe8d166
fix: replace FilterLogs with header iteration and optimize spamoor co…
chatton 99e3e42
fix: improve benchmark measurement window and reliability
chatton e4e06c5
fix: address PR review feedback for benchmark suite
chatton 1c3b560
Merge branch 'main' into cian/bench-refactor
chatton 03b9239
chore: specify http
chatton fe3ca23
chore: filter out benchmark tests from test-e2e
chatton 8752fee
Merge branch 'main' into cian/bench-refactor
chatton 06f532b
Merge branch 'cian/bench-refactor' into cian/erc20-benchmark
chatton 560974a
Merge remote-tracking branch 'origin/main' into cian/erc20-benchmark
chatton 054f2c6
refactor: centralize reth config and lower ERC20 spammer count
chatton 26bb117
chore: collect all traces at once
chatton b88cae3
chore: self review
chatton 272ab71
refactor: extract benchmark helpers to slim down ERC20 test body
chatton 676e0d1
docs: add detailed documentation to benchmark helper methods
chatton f4949a1
ci: add ERC20 throughput benchmark job
chatton 4c7b7e1
chore: remove span assertions
chatton 6f56c80
chore: adding gas burner test
chatton 40ff689
feat: allowing for env vars to be set for external resources
chatton 4a7af54
chore: adding private key as env var
chatton d2aeebc
chore: use host networking for spamoor in external mode
chatton 7a85f3b
fix: pin tastora to correct commit with host network support
chatton f42ffe0
chore: change service name
chatton 53d12e1
chore: scope victoria traces queries to current test run
chatton bdd7132
chore: paginate victoria traces queries
chatton 0fe81a1
chore: reduce tryCollectSpans timeout to 15s for victoria provider
chatton 2756ce3
chore: increase gas burner spammers from 4 to 8
chatton 8802838
chore: revert gas burner spammers back to 4
chatton 2f27ab9
chore: switch victoria traces to LogsQL API
chatton 67c00d3
fix: use _stream filter syntax for LogsQL query
chatton f7a8c5f
chore: tune gasburner config for 100M gas limit blocks
chatton 1308af4
chore: match spamoor slot-duration to deployed 250ms block time
chatton 3b4f5d1
chore: increase gasburner tx count for sustained load
chatton c0de7a3
chore: use 8 spammers for sustained gasburner load
chatton f5d0eaa
chore: revert to 4 spammers to avoid nonce collisions
chatton 6c4ca40
chore: bump gasburner fees to 1000/500 gwei to clear stale txs
chatton 2d4ab9a
chore: adding support for rich flowchart display
chatton 9bd2381
adding host name to span
chatton dffc429
chore: add re-broadcast
chatton 7fac0a3
chore: adding flowchart
chatton 4445cde
chore: bump fee
chatton 64633d8
chore: bump wallet amounts
chatton f6100a2
chore: increase throughput
chatton 51754b8
chore: 4x tx volume
chatton f1e68d2
chore: fewer tx, higher gas
chatton a3325dd
chore: remove amount
chatton aea69a2
Bumped from 50/20 ETH to 500/200 ETH per wallet
chatton 1468615
Merge remote-tracking branch 'origin/main' into cian/enable-tests-to-…
chatton ae26530
feat: parameterize benchmark config via env vars and add engine span …
chatton 8866358
refactor: extract benchConfig and benchmarkResult types
chatton 87348de
chore: cleaning up flowchart dissplay and fixing UI link error
chatton 1990e85
refactor: address PR review feedback on benchmark harness
chatton 0a4c8a4
Merge branch 'main' into cian/enable-tests-to-run-without-infra
chatton File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| //go:build evm | ||
|
|
||
| package benchmark | ||
|
|
||
| import ( | ||
| "os" | ||
| "strconv" | ||
| "testing" | ||
| "time" | ||
| ) | ||
|
|
||
| // benchConfig holds all tunable parameters for a benchmark run. | ||
| // fields are populated from BENCH_* env vars with sensible defaults. | ||
| type benchConfig struct { | ||
| ServiceName string | ||
|
|
||
| // infrastructure (used by setupLocalEnv) | ||
| BlockTime string | ||
| SlotDuration string | ||
| GasLimit string | ||
| ScrapeInterval string | ||
|
|
||
| // load generation (used by test functions) | ||
| NumSpammers int | ||
| CountPerSpammer int | ||
| Throughput int | ||
| WarmupTxs int | ||
| GasUnitsToBurn int | ||
| MaxWallets int | ||
| WaitTimeout time.Duration | ||
| } | ||
|
|
||
| func newBenchConfig(serviceName string) benchConfig { | ||
| return benchConfig{ | ||
| ServiceName: serviceName, | ||
| BlockTime: envOrDefault("BENCH_BLOCK_TIME", "100ms"), | ||
| SlotDuration: envOrDefault("BENCH_SLOT_DURATION", "250ms"), | ||
| GasLimit: envOrDefault("BENCH_GAS_LIMIT", ""), | ||
| ScrapeInterval: envOrDefault("BENCH_SCRAPE_INTERVAL", "1s"), | ||
| NumSpammers: envInt("BENCH_NUM_SPAMMERS", 2), | ||
| CountPerSpammer: envInt("BENCH_COUNT_PER_SPAMMER", 2000), | ||
| Throughput: envInt("BENCH_THROUGHPUT", 200), | ||
| WarmupTxs: envInt("BENCH_WARMUP_TXS", 200), | ||
| GasUnitsToBurn: envInt("BENCH_GAS_UNITS_TO_BURN", 1_000_000), | ||
| MaxWallets: envInt("BENCH_MAX_WALLETS", 500), | ||
| WaitTimeout: envDuration("BENCH_WAIT_TIMEOUT", 10*time.Minute), | ||
| } | ||
| } | ||
|
|
||
| func (c benchConfig) totalCount() int { | ||
| return c.NumSpammers * c.CountPerSpammer | ||
| } | ||
|
|
||
| func (c benchConfig) log(t testing.TB) { | ||
| t.Logf("load: spammers=%d, count_per=%d, throughput=%d, warmup=%d, gas_units=%d, max_wallets=%d", | ||
| c.NumSpammers, c.CountPerSpammer, c.Throughput, c.WarmupTxs, c.GasUnitsToBurn, c.MaxWallets) | ||
| t.Logf("infra: block_time=%s, slot_duration=%s, gas_limit=%s, scrape_interval=%s", | ||
| c.BlockTime, c.SlotDuration, c.GasLimit, c.ScrapeInterval) | ||
| } | ||
|
|
||
| func envOrDefault(key, fallback string) string { | ||
| if v := os.Getenv(key); v != "" { | ||
| return v | ||
| } | ||
| return fallback | ||
| } | ||
|
|
||
| // envInt returns the integer value of the given env var, or fallback if unset | ||
| // or unparseable. Invalid values silently fall back to the default. | ||
| func envInt(key string, fallback int) int { | ||
| v := os.Getenv(key) | ||
| if v == "" { | ||
| return fallback | ||
| } | ||
| n, err := strconv.Atoi(v) | ||
| if err != nil { | ||
| return fallback | ||
| } | ||
| return n | ||
| } | ||
|
|
||
| // envDuration returns the duration value of the given env var (e.g. "5m", "30s"), | ||
| // or fallback if unset or unparseable. | ||
| func envDuration(key string, fallback time.Duration) time.Duration { | ||
| v := os.Getenv(key) | ||
| if v == "" { | ||
| return fallback | ||
| } | ||
| d, err := time.ParseDuration(v) | ||
| if err != nil { | ||
| return fallback | ||
| } | ||
| return d | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fail fast on invalid
BENCH_*overrides.envInt/envDurationcurrently mask typos, and parseable but nonsensical values like negative counts or timeouts still flow through. That means a mis-setBENCH_NUM_SPAMMERS,BENCH_WARMUP_TXS, orBENCH_WAIT_TIMEOUTcan still produce benchmark output for a different workload than the job requested. Please validate bounds when buildingbenchConfigand return an error instead of silently defaulting.As per coding guidelines, "Validate all inputs from external sources".
Also applies to: 68-94
🤖 Prompt for AI Agents