fix: batch pr_issue_links backfill to avoid event-loop stall

[enjoyandlove]

Summary

runPrIssueLinksBackfill() was executing a single monolithic better-sqlite3 transaction over every PR in a repo, holding the DB writer connection for the entire duration. On large repos (1 000+ PRs) this blocked the Node.js event loop for 30–60+ seconds, causing Cloudflare 524 timeouts on concurrent requests.

The function is now async and processes PRs in batches of 200. Each batch commits its own transaction and then yields the event loop via await yieldEventLoop() before the next batch, keeping the writer lock held for only a bounded window at a time. The setImmediate wrapper in backfillPrIssueLinksIfNeeded() is updated to async so it correctly awaits the batched loop before writing the completion marker and cleaning up inFlightLinksBackfill.

Related Issues

Fixes #209

Type of Change

• Bug fix

Testing

• pnpm build passes
• Manual browser smoke test (for UI changes)
• N/A — docs / config only

Checklist

• Self-reviewed the diff
• Follows existing code patterns and naming (yieldEventLoop, persistInChunks pattern already used elsewhere)
• No unrelated changes included
• Documentation updated if behavior changed

Summary by CodeRabbit

• Chores

  • Enabled workspace build settings to allow specific native-package builds.

• Refactor

  • Improved PR-to-issue link backfill: now runs as a deferred background task, processes data in safe paginated batches, and marks completion only after successful work to avoid incorrect state or blocking.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 540f8ad5-38c1-483b-8c06-cfe200e1450b

📥 Commits

Reviewing files that changed from the base of the PR and between 5cf9214 and 2d5ad62.

📒 Files selected for processing (1)

• src/lib/refresh.ts

🚧 Files skipped from review as they are similar to previous changes (1)

• src/lib/refresh.ts

---

📝 Walkthrough

Walkthrough

Adds pnpm workspace allowBuilds flags for three native packages and refactors PR→issue link backfill to an async, paginated loop that processes pulls in batches, yields between batches, and marks completion only after the awaited backfill finishes.

Changes

Workspace and async paginated backfill

Layer / File(s)	Summary
Workspace build configuration pnpm-workspace.yaml	allowBuilds configuration section is added with boolean flags for better-sqlite3, sharp, and unrs-resolver.
Deferred scheduler awaits async backfill src/lib/refresh.ts	backfillPrIssueLinksIfNeeded changes its deferred callback to an async setImmediate callback that awaits runPrIssueLinksBackfill and writes repo_meta.pr_issue_links_backfilled_at only after completion; errors still prevent writing the marker and the in-flight dedupe entry is cleared in a finally block.
Async paginated PR-issue link backfill src/lib/refresh.ts	runPrIssueLinksBackfill is rewritten to fetch pulls in bounded batches using LIMIT/OFFSET, extract linked issues, insert links with INSERT OR IGNORE per batch, increment inserted counts, advance the offset, and call yieldEventLoop() between batches.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• MkDev11/gittensor-hub#172: Modifies src/lib/refresh.ts backfill scheduling and insertion logic.
• MkDev11/gittensor-hub#168: Implements similar deferred/paginated backfill changes in src/lib/refresh.ts.
• MkDev11/gittensor-hub#195: Related pagination fixes to GitHub linked-PR/closing-issue fetchers used by backfill.

Poem

🐰 I hop in ordered batches, light,
Yielding ticks to keep things bright.
OFFSET hums, inserts take root,
Completion waits — no janky loot.
Fields backfilled, the DB sleeps right.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Out of Scope Changes check	⚠️ Warning	The pnpm-workspace.yaml change (adding allowBuilds config) appears out of scope relative to the linked issue #209, which focuses solely on fixing the backfill blocking issue.	Clarify whether the pnpm-workspace.yaml changes are necessary for the backfill fix, or separate them into a distinct PR to maintain focus on the primary objective.
Docstring Coverage	⚠️ Warning	Docstring coverage is 66.67% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically summarizes the main change: batching the PR issue links backfill to avoid event-loop stalling, which directly addresses the primary objective.
Linked Issues check	✅ Passed	The code changes successfully implement all requirements from issue #209: paginated batching (200 PRs per batch), async processing with event-loop yields, DB writer lock management, and proper async/await handling with cleanup.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

src/lib/refresh.ts

ESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox.

Warning

Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pnpm-workspace.yaml`:
- Around line 1-4: The allowBuilds map currently contains invalid string
placeholders; replace the placeholder values for the package matchers
better-sqlite3, sharp, and unrs-resolver with boolean literals (true or false)
so they become valid pnpm allowBuilds entries (e.g., set better-sqlite3: true,
sharp: false, unrs-resolver: true or whatever policy you intend) — update the
values in the allowBuilds block to use true/false for those three keys.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3400aa43-9c8a-4980-aafb-c6b3dd4be2d5

📥 Commits

Reviewing files that changed from the base of the PR and between e223bc2 and ac7079b.

📒 Files selected for processing (2)

• pnpm-workspace.yaml
• src/lib/refresh.ts

[enjoyandlove]

Hi,@MkDev11
Could you please review this PR?

[MkDev11]

please remove pnpm-workspace.yaml from this PR. It changes pnpm native build-script policy for better-sqlite3, sharp, and unrs-resolver, which is unrelated to the backfill/event-loop fix. The src/lib/refresh.ts change is the relevant part here.