fix(fetchers): honor Retry-After header on 429/503 responses

[Sanjays2402]

Summary

The LLM and embedding fetchers in apps/memos-local-plugin classified HTTP 429 as transient and retried with exponential backoff, but never read the Retry-After header. Under provider rate limiting, retries fired before the upstream-requested cooldown, extending throttling and wasting paid API calls.

This PR parses Retry-After per RFC 7231 §7.1.3 (delta-seconds or HTTP-date), caps the value to a sane maximum, and prefers it over exponential backoff on 429/503. Falls back to the existing backoff when the header is absent or unparseable.

Changes

• New shared helper apps/memos-local-plugin/core/util/retry-after.ts exporting parseRetryAfterMs(resp, capMs).
• apps/memos-local-plugin/core/llm/fetcher.ts: parse Retry-After on 429/503 and use it as the sleep duration; small refactor split backoff() into backoffMs() + sleep() so both code paths share one timer.
• apps/memos-local-plugin/core/embedding/fetcher.ts: same treatment, mirroring the LLM fetcher.
• Tests added/extended:
  • tests/unit/util/retry-after.test.ts — covers integer seconds, HTTP-date, capping, malformed, missing, past-date, and negative inputs.
  • tests/unit/llm/fetcher.test.ts and tests/unit/embedding/fetcher.test.ts — three new cases each: integer Retry-After is honored, HTTP-date is honored, malformed value falls back to backoff.

Behavior

• Retry-After: 5 → sleep ~5s instead of next backoff step.
• Retry-After: Wed, 21 Oct 2025 07:28:00 GMT → sleep ~(date - now), capped.
• Header absent or malformed → falls back to existing exponential backoff (no behavior change).
• Capped at 60s (MAX_RETRY_AFTER_MS) to prevent hostile or buggy servers from pinning the client indefinitely.
• Only 429 and 503 honor Retry-After. Other 5xx still use exponential backoff (the header is rarely set on those, and we want to keep retrying quickly on non-rate-limit failures).

Notes

• Could not run tsc --noEmit locally (no node_modules in this worktree), so types were verified by inspection. The new helper has explicit return type number | null and matches the import-style (.js extension, Bundler resolution) used elsewhere in the codebase.
• Tests use the existing real-timer pattern in tests/unit/embedding/fetcher.test.ts; sleep upper bounds are generous (≥900 ms for Retry-After: 1) to avoid flakes on slow CI.

Fixes #1620

[Copilot]

Pull request overview

This PR updates the LLM and embedding HTTP fetchers in apps/memos-local-plugin to honor the upstream Retry-After header on 429/503 responses, preferring the server-provided cooldown over the existing exponential backoff (with a 60s cap), and adds unit coverage for the new behavior.

Changes:

• Added a shared parseRetryAfterMs(resp, capMs) helper that supports delta-seconds and HTTP-date formats.
• Updated both LLM and embedding fetchers to use Retry-After (when present/parseable) for 429/503 retry sleeps, otherwise falling back to exponential backoff.
• Added/extended unit tests to cover parsing behavior and fetcher retry timing behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
apps/memos-local-plugin/core/util/retry-after.ts	New helper to parse and cap Retry-After into a millisecond delay.
apps/memos-local-plugin/core/llm/fetcher.ts	Uses Retry-After delay for 429/503 retries; refactors backoff into backoffMs() + sleep().
apps/memos-local-plugin/core/embedding/fetcher.ts	Mirrors LLM fetcher behavior to honor Retry-After for 429/503 retries.
apps/memos-local-plugin/tests/unit/util/retry-after.test.ts	Adds unit tests for delta-seconds, HTTP-date, malformed values, and capping behavior.
apps/memos-local-plugin/tests/unit/llm/fetcher.test.ts	Adds timing-based tests asserting Retry-After is honored or falls back to backoff.
apps/memos-local-plugin/tests/unit/embedding/fetcher.test.ts	Adds timing-based tests asserting Retry-After is honored or falls back to backoff.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Sanjays2402]

Thanks for the review — addressed in the latest commit:

• retry-after.ts:11: Updated the JSDoc to clarify that 0 (retry immediately) is a valid value; only negative or unparseable values return null. Tests already covered the 0 case and continue to pass.
• llm/fetcher.ts:17 / embedding/fetcher.ts:15: Applied Math.min(sleepMs, MAX_RETRY_AFTER_MS) to the final sleep duration in both fetchers, so the 60s cap applies whether the value came from Retry-After or backoffMs(). Previously a maxRetries: 10 config could let exponential backoff exceed 60s.

Diff is small and behavior-only-on-the-margin (only changes anything when attempt >= ~6 in a long retry chain, or when Retry-After: 0 arrives).

[github-actions]

This PR has been automatically marked as stale due to inactivity.

[Memtensor-AI]

Automated Test Results: FAILED

Cloud test-engine rerun against dev-v2.0.22 failed.

• Run: tr-f2a7758b-879 on cloud test-engine 10010

• Verdict: fail_code_bug

• Summary: Tests failed. Failed cases: test -- tests/unit/embedding/fetcher.test.ts tests/unit/llm/fetcher.test.ts tests/unit/util/retry-after.test.ts

• memos_local_plugin/unit: 29 passed, 2 failed, 0 skipped

Failed cases:

• test -- tests/unit/embedding/fetcher.test.ts tests/unit/llm/fetcher.test.ts tests/unit/util/retry-after.test.ts: t.ts > llm/fetcher > 429 with Retry-After: HTTP-date in the near future is honored
  AssertionError: expected 650 to be greater than or equal to 800
  ❯ tests/unit/llm/fetcher.test.ts:123:21
  121| expect(f).toHaveBeenCalledTimes(2);
  122| // Roughly the diff (allow generous slack for clock + jitter).
  123| expect(elapsed).toBeGreaterThanOrEqual(800);
  | ^
  124| expect(elapsed).toBeLessThan(5_000);
  125| });

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯

Do not merge until this is fixed and cloud tests pass.