Skip to content

chore: disable SQLite disk sync in benchmarks to reduce walltime noise - #1841

Merged
collerek merged 2 commits into
masterfrom
chore/benchmarks-sqlite-no-sync
Oct 12, 2026
Merged

collerek merged 2 commits into
masterfrom
chore/benchmarks-sqlite-no-sync

Conversation

@collerek

Copy link
Copy Markdown
Collaborator

Benchmarks that write rows one by one were dominated by SQLite fsync and journal-file I/O, so CodSpeed's walltime runs flagged random 15–30% "regressions" on PRs that never touch those code paths (e.g. #1840, #1824). The benchmark fixture now sets PRAGMA synchronous=OFF and journal_mode=MEMORY on SQLite connections, so these benchmarks measure ormar overhead instead of disk latency.

Locally, over 3 runs of the 21 write-per-row benchmarks, the mean run-to-run spread dropped from 17% to 6% (max 39% → 12%). Expect the first CodSpeed report here to show large improvements on those benchmarks, as the baseline resets once this is merged.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@codspeed

codspeed Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by ×2.5

⚡ 39 improved benchmarks
✅ 84 untouched benchmarks

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_updating_models_individually[40] 323.3 ms 106.9 ms ×3
⚡ test_saving_models_individually[20] 157.5 ms 52.1 ms ×3
⚡ test_saving_models_individually[40] 314.9 ms 104.7 ms ×3
⚡ test_updating_models_individually[10] 80.6 ms 26.9 ms ×3
⚡ test_updating_models_individually[20] 161.1 ms 53.8 ms ×3
⚡ test_creating_models_individually[40] 318.4 ms 107 ms ×3
⚡ test_creating_models_individually[20] 158.8 ms 53.5 ms ×3
⚡ test_saving_models_individually[10] 78.2 ms 26.4 ms ×3
⚡ test_update_or_create_when_create[40] 318.3 ms 107.8 ms ×3
⚡ test_creating_models_individually[10] 79.3 ms 26.9 ms ×2.9
⚡ test_update_or_create_when_create[20] 158.9 ms 54 ms ×2.9
⚡ test_saving_related_new_children[10] 92.2 ms 31.7 ms ×2.9
⚡ test_update_or_create_when_create[10] 77.9 ms 26.9 ms ×2.9
⚡ test_saving_related_new_children[20] 177.4 ms 61.3 ms ×2.9
⚡ test_saving_models_individually_with_related_models[40] 339.5 ms 118.8 ms ×2.9
⚡ test_saving_models_individually_with_related_models[20] 169.5 ms 59.4 ms ×2.9
⚡ test_saving_related_new_children[40] 341.9 ms 120.3 ms ×2.8
⚡ test_adding_new_children_to_reverse_relation[40] 340 ms 119.7 ms ×2.8
⚡ test_saving_models_individually_with_related_models[10] 84.7 ms 29.8 ms ×2.8
⚡ test_adding_new_children_to_reverse_relation[20] 168.4 ms 60.1 ms ×2.8
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing chore/benchmarks-sqlite-no-sync (30bbbb8) with master (8993d4e)

Open in CodSpeed

@collerek
collerek merged commit fcbc486 into master Oct 12, 2026
11 checks passed
@collerek
collerek deleted the chore/benchmarks-sqlite-no-sync branch October 12, 2026 16:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants