Skip to content

Bound SPG frame concurrency by CPU count and available memory - #112

Merged
davisadam10 merged 3 commits into
devfrom
fix/spg-bounded-concurrency
Nov 20, 2026
Merged

davisadam10 merged 3 commits into
devfrom
fix/spg-bounded-concurrency

Conversation

@davisadam10

Copy link
Copy Markdown
Collaborator

Problem

SPG started one thread per frame, all running at once, each holding full-resolution image buffers. test_generate_spg_patterns_for_led_walls peaked at ~14.5 GB RSS on its own and starved 16 GB GitHub runners, so the Ubuntu job (and occasionally Windows) died with The runner has received a shutdown signal / lost communication. Disk was ruled out (77 GB free).

Fix

ThreadedFunction now acquires a process-wide ResourceGate before doing work:

  • at most 2 x os.cpu_count() frames in flight (frame work includes file I/O, so >1 per CPU helps);
  • estimated memory of in-flight frames stays under 60% of available RAM (container aware on Linux, GlobalMemoryStatusEx on Windows; total RAM on macOS), using a per-frame estimate from image resolution;
  • an oversize frame still runs, alone, so progress is guaranteed;
  • stdlib only, no new dependency; SPG_MAX_WORKERS and SPG_MEMORY_BUDGET_MB override.

Measurements

SPG test alone, 4-CPU / 16 GB GitHub Ubuntu runner: unbounded 419 s / ~15 GB; this change 264 s / 3.5 GB.

Local 16-CPU / 128 GB Mac: unbounded 67 s / 21.1 GB; 16 workers 90 s / 8.2 GB; 32 workers (new default) 76 s / 13.9 GB; 48 workers 71 s / 18.6 GB. The default is ~13% slower than unbounded locally at about two thirds of the memory.

Independent of #111 (macOS nightly OCIO fix).

🤖 Generated with Claude Code

davisadam10 and others added 3 commits November 20, 2026 12:55
Every frame previously got its own thread, all running at once, each holding
full resolution image buffers. A single SPG test peaked at ~14.5GB RSS and
starved 16GB CI runners. ThreadedFunction now shares a semaphore sized to
min(cpu_count, 8), overridable with SPG_MAX_WORKERS.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Replace the fixed cap with a gate that limits concurrent frames by os.cpu_count()
and by a memory budget (60% of available RAM, container-aware) using a per-frame
estimate from image resolution. Large machines run fully parallel; small or
low-memory machines are throttled. SPG_MAX_WORKERS and SPG_MEMORY_BUDGET_MB override.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Measured on a 16 core Mac: 16 workers 90s/8.2GB, 32 workers 76s/13.9GB,
unbounded 67s/21GB. Frame work includes file I/O so >1 worker per CPU helps.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@davisadam10
davisadam10 merged commit a15068f into dev Nov 20, 2026
10 of 12 checks passed
@davisadam10
davisadam10 deleted the fix/spg-bounded-concurrency branch November 20, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant