Repository navigation
Bound SPG frame concurrency by CPU count and available memory - #112
Merged
Merged
Conversation
Every frame previously got its own thread, all running at once, each holding full resolution image buffers. A single SPG test peaked at ~14.5GB RSS and starved 16GB CI runners. ThreadedFunction now shares a semaphore sized to min(cpu_count, 8), overridable with SPG_MAX_WORKERS. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replace the fixed cap with a gate that limits concurrent frames by os.cpu_count() and by a memory budget (60% of available RAM, container-aware) using a per-frame estimate from image resolution. Large machines run fully parallel; small or low-memory machines are throttled. SPG_MAX_WORKERS and SPG_MEMORY_BUDGET_MB override. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Measured on a 16 core Mac: 16 workers 90s/8.2GB, 32 workers 76s/13.9GB, unbounded 67s/21GB. Frame work includes file I/O so >1 worker per CPU helps. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
SPG started one thread per frame, all running at once, each holding full-resolution image buffers.
test_generate_spg_patterns_for_led_wallspeaked at ~14.5 GB RSS on its own and starved 16 GB GitHub runners, so the Ubuntu job (and occasionally Windows) died withThe runner has received a shutdown signal/lost communication. Disk was ruled out (77 GB free).Fix
ThreadedFunctionnow acquires a process-wideResourceGatebefore doing work:2 x os.cpu_count()frames in flight (frame work includes file I/O, so >1 per CPU helps);GlobalMemoryStatusExon Windows; total RAM on macOS), using a per-frame estimate from image resolution;SPG_MAX_WORKERSandSPG_MEMORY_BUDGET_MBoverride.Measurements
SPG test alone, 4-CPU / 16 GB GitHub Ubuntu runner: unbounded 419 s / ~15 GB; this change 264 s / 3.5 GB.
Local 16-CPU / 128 GB Mac: unbounded 67 s / 21.1 GB; 16 workers 90 s / 8.2 GB; 32 workers (new default) 76 s / 13.9 GB; 48 workers 71 s / 18.6 GB. The default is ~13% slower than unbounded locally at about two thirds of the memory.
Independent of #111 (macOS nightly OCIO fix).
🤖 Generated with Claude Code