Skip to content

feat(core): store session media as media:<hash> refs in a content-addressed blob table - #58656

Merged
kitlangton merged 8 commits into
devfrom
media-storage
Oct 25, 2026
Merged

kitlangton merged 8 commits into
devfrom
media-storage

Conversation

@kitlangton

@kitlangton kitlangton commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Part 1 of 5 in the media blob stack (#58656 → #58741 → #58742 → #58657 → #58743). No API URL is ever persisted, and the endpoint is experimental.

What it does

Adds a general-purpose, content-addressed blob table so large inline attachments can live outside session_message rows, and makes every read hand back the same inline base64 as before. Nothing writes refs yet; that's #58741.

The tables are named blob / blob_ref rather than anything media-specific so large tool outputs (and later instruction values, today in instruction_blob) can share the same store and cleanup.

How it works

  • A stored attachment points at a blob as media:<hash> in its existing uri field. <hash> is the SHA-256 hex of the decoded bytes. Only media:<64 hex> counts as a ref.
  • session/stored.ts (SessionStored) is the schema for what a stored file or tool attachment looks like on disk: either inline data or a ref. Readers decode rows with it instead of casting, and the sql.ts columns are typed with it. No as any left in the read path.
  • blob_ref records which message uses which blob, so unreferenced blobs can be garbage-collected later.
  • On read, refs are hydrated back to inline data + source, so clients and the model see the same shape as today. blob.ts is split into small single-purpose functions (find refs, load blobs, hydrate one file, hydrate one tool result).
  • Readers that never use file bytes (next promotable inbox item, title generation, compaction recent text) read no blobs. firstUserMessage reads only the text via json_extract. The output-limit streak check selects only type and finish.
  • Delivering a queued prompt copies the raw stored files into session_message, so its media: refs survive promotion.
  • If a ref's blob is gone, the read returns a [media unavailable: name (mime)] text placeholder instead of a bare media: ref (user files become a text/plain file, tool results a text item).

Risk

Core storage and read path only. Rows without refs are untouched, and no ref rows exist yet in production, so the small behaviour changes below can't affect existing data:

  • a stored ref must include source;
  • a ref's own mime wins over the blob's;
  • picking the next promotable inbox item no longer decodes user and synthetic payloads (delivery still decodes them and catches bad ones).

The migration (20261025123711_blob) only creates two empty tables and two indexes; it rewrites no data.

Proof

  • bun test (packages/core): 334 tests across session-projector, session-runner, session-title, session-compaction, all pass. The projector tests cover "missing blob → placeholder" and check the raw session_message row still holds media:<hash> after an inbox prompt is delivered.
  • bun run typecheck (core): clean
  • bun run migration --check: clean
  • bunx oxlint on changed files: 0 errors
  • Independent review of the cleanup commit: no blockers; its one nit is folded in.

Added by the opencode bot (AI-generated).

Related: #58436, #36444

@kitlangton kitlangton changed the title feat(core): externalize session media into deduplicated media_blob table feat(core): add media_blob schema and hydrate externalized media on read Oct 24, 2026
@kitlangton kitlangton changed the title feat(core): add media_blob schema and hydrate externalized media on read feat(core): store session media as media:<hash> refs in a content-addressed media_blob table Oct 24, 2026
Rows reference blobs by a URL-neutral media:<hash> in the existing uri field. The hash (lowercase hex SHA-256 of the raw bytes) is defined once in SessionMediaBlob. Reads still accept the legacy /api/media/<hash> form.
@kitlangton kitlangton changed the title feat(core): store session media as media:<hash> refs in a content-addressed media_blob table feat(core): store session media as media:<hash> refs in a content-addressed blob table Oct 25, 2026
@kitlangton
kitlangton marked this pull request as ready for review October 25, 2026 12:44

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The read-path hydration looks sound. I checked it with the new projector tests (19/19 pass), tsc -b on core (clean) and bun run migration --check (no drift). Nothing writes refs yet, so there was nothing to drive in the TUI. I went through the callers that still decode rows without hydrating (projector getAssistant/getShell/getCompaction, revert.ts, compaction recovery in llm.ts). They read and write back assistant, shell or compaction metadata, so keeping the raw refs there seems right.

Public API: unchanged. The only addition is two new internal SQLite tables (blob, blob_ref), created by a migration that rewrites no data.

One small point inline: the legacy /api/media/ ref handling looks unnecessary, since no released build could have written those refs.

The stored user-file shape (uri, bytes, origin) isn't part of Prompt.FileAttachment. So it is only described by the local StoredUserFile type and as any in the tests (AGENTS.md says to avoid any). It may be worth giving it a real stored schema when #58741 defines the write side.

Related: #58436, #36444

Comment thread packages/core/src/session/blob.ts Outdated
*/
export const REF_SCHEME = "media:"
// Written only by early builds of this feature; still read, never written.
const LEGACY_REF_PREFIX = "/api/media/"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blob table first appears in this PR, so I don't think any released build can have written /api/media/<hash> refs; they only existed earlier in this stack. Could this legacy branch, and the legacy cases in the tests, be dropped? As it stands, any file uri that happens to be /api/media/<64 hex> would be rewritten from the blob table or turned into a placeholder.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The legacy /api/media/ point is addressed: only media:<hash> is treated as a ref now, and the test checks that a URI shaped like /api/media/<hash> is left as is. I re-ran the projector tests (19/19 pass) and tsc -b on core (clean). Public API unchanged. Nothing new to raise. The PR description still mentions legacy /api/media/<hash> refs, which this commit removed.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new commit looks sound. Delivery now keeps media: refs in the stored row, and the readers that never use bytes (title generation, compaction recent text, nextPromotable, the output-limit check) skip blob reads. I checked their callers: they use only text, type, delivery or id, so the empty data from elide is safe there. Projector, title, runner and related session tests pass (313/313), and tsc -b on core is clean. Public API unchanged. The earlier /api/media/ point was addressed in the previous commit. Nothing new to raise.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema refactor looks sound. SessionStored addresses the earlier stored-shape point: stored user files and tool refs now have a real schema, and the tests no longer use as any. I traced the callers of the slimmer nextPromotable / firstUserMessage results (runner, execution, title); they only read fields that are still returned. Delivery round-trips refs through the stored schema, and the projector test confirms the raw row still holds media:<hash>. The projector, runner, title, history, store and compaction tests pass (368/368), and tsc -b on core is clean. Public API unchanged. Nothing new to raise.

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit is a tidy-up with no behaviour change: an explicit sequential forEach, checks for a missing value instead of ! assertions, and a Ref type for refHash. tsc -b on core is clean and the session projector, runner, title, history, store and compaction tests pass (304/304). Public API unchanged. Nothing new to raise.

@kitlangton
kitlangton merged commit 70c69f0 into dev Oct 25, 2026
19 of 20 checks passed
@kitlangton
kitlangton deleted the media-storage branch October 25, 2026 15:28
@amrkmn amrkmn mentioned this pull request Oct 27, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant