fix(model-artifacts): materialize longcat-video on the controller, and support companion repos by localai-bot · Pull Request #10949 · mudler/LocalAI

localai-bot · 2026-07-19T08:31:51Z

Loading longcat-video-avatar-1.5 in distributed mode failed with DeadlineExceeded after exactly 302s. The apparent cause was a short timeout. The actual cause was that the controller never acquired the weights at all, so the worker's Python backend downloaded ~57 GB from HuggingFace inside the LoadModel RPC.

Observed on a live cluster:

INFO Staging model files for remote node node="nvidia-thor" modelFile="/models/meituan-longcat/LongCat-Video-Avatar-1.5"
INFO Loading model on remote node addr="192.168.68.23:30232"
ERROR Failed to load model ... DeadlineExceeded

/models/meituan-longcat existed on neither the controller nor the worker. Meanwhile the worker's HF cache grew to 57 GB:

35G  models--meituan-longcat--LongCat-Video-Avatar-1.5
22G  models--meituan-longcat--LongCat-Video

Root cause — a regression from #10910

core/config/model_artifact_inference.go gates artifact inference on a backend allow-list. longcat-video was absent, so PrimaryArtifactSpec returned hasArtifact=false, bindPrimaryArtifact was never reached, and Ensure was never called.

That gate arrived in 2dade4a9f (#10910), which post-dates the longcat backend (#10792). #10910 was itself a correctness fix — single-file backends like llama.cpp were being handed a snapshot directory — and it enumerated the directory-consuming backends known at the time. longcat-video was simply never added.

Confirmed by three independent checks on the live controller: /models/.artifacts/huggingface/ exists and holds snapshots for other models; the longcat config carries no artifacts: key; and no "falling back to legacy" log line was ever emitted for it.

Bare owner/repo references were never the problem — PrimaryArtifactSpec synthesizes a spec via CanonicalRepo for 2-part references. The allow-list was the only thing blocking it.

The backend already implements the intended contract (backend/python/longcat-video/backend.py:117-119): use ModelFile when it is a directory, otherwise fall back to snapshot_download. The self-download is the fallback, not the design.

Changes

1. Add longcat-video to managedArtifactBackends. Same criterion as transformers/vllm/diffusers — it consumes a snapshot directory.

2. Warn when staging skips a missing source. stageModelFiles skipped a non-existent path at xlog.Debug and blanked the field, while a transport failure on the same field 40 lines later is xlog.Error + hard failure. That asymmetry is why this looked like a timeout bug for hours: the reassuring "Staging model files for remote node" line was emitted for a path that existed nowhere. Behavior is unchanged (a missing ModelFile is legitimate for backends that fetch their own weights) — only visibility.

3. Companion artifacts. Avatar 1.5 needs two repos: its own (35 GB) and the base LongCat-Video (22 GB) for tokenizer, text encoder and VAE. Spec.Normalize previously rejected any artifact not named model, and only Artifacts[0] was ever materialized, so the second repo could not be expressed and kept being fetched in-band.

target ∈ {model, companion} is now accepted, with exactly one primary at index 0. Both preloadOne and bindPrimaryArtifact loop. Failure policy: the inferred primary keeps warn-and-fall-back; declared companions are all-or-nothing.

Cache-key stability is pinned two ways, since a key change would silently re-download every installed managed model: one spec proves a companion and a primary with identical sources yield the same key (name/target are not in the identity struct), another pins a known primary's literal digest.

4. Hand the resolved snapshot to the backend. withCompanionArtifactOptions synthesizes <name>:<relative snapshot path> into pb.ModelOptions.Options at load time, never persisted — a re-resolve to a new revision would otherwise leave a stale path in the on-disk YAML. Author-set keys win. backend.py resolves base_model through request.ModelPath, the convention qwen-tts/voxcpm/outetts/ace-step already use, and the sibling heuristic at :443-450 is deleted — it looked for <snapshot>/../LongCat-Video, which cannot exist under a cache-key layout.

5. Stage from the models root. ModelsRootForModelFile anchors on the directory containing the .artifacts tree; DeriveRemoteModelPath takes the models-root-relative path instead of the Model field. Previously frontendModelsDir = TrimSuffix(ModelFile, Model) never matched for a managed primary (ModelFile ends in .artifacts/huggingface/<key>/snapshot, Model is a bare repo id), so it collapsed to the snapshot dir itself — self-consistent for the primary, fatal for a companion, which stageGenericOptions then skipped silently. That is the same silent-skip class as (2), in a second location.

router_dirstage_test.go is updated deliberately: it pinned ModelFile == ModelPath == snapshot, and now ModelPath is the models root with keys keeping their full relative path. Reasoning is recorded inline and in the commit message. A legacy-layout regression spec was added that passes before and after, as evidence the old path is untouched.

Tradeoff worth reviewing

The gallery allow_patterns pin base_model/**, matching this entry's use_distill:true and use_int8 defaulting false. Setting use_int8:true on this entry would find the int8 weights absent from the snapshot. Documented inline in the gallery entry rather than fetching both subfolders and forfeiting roughly half the download.

Testing

Red-first throughout: 7 functional failures across pkg/modelartifacts and core/config for step 1, 2 for step 2, and step 3's sole failure was exactly the router_dirstage_test.go invariant.

One test was corrected rather than papered over: a "rejects two primaries" spec asserted an unreachable path (two primaries must both be named model, so duplicate-name fires first). It was split into the genuinely reachable zero-primary and second-artifact-claiming-primary cases.

ok  core/config, core/gallery, core/gallery/importers, pkg/modelartifacts, core/services/nodes

make lint → 0 issues. Coverage 53.0% (baseline 48.5%).

Merge notes

core/services/nodes/router.go is touched in 4 small hunks (net +15/−10), all inside stageModelFiles at lines ~1033-1173. #10948 touches the same file at lines 18-336. git merge-tree between the two branches reports zero conflicts.

Future cleanup (not in this PR)

diffusers/backend.py:259-263 hardcodes a bfl_repo to pull text_encoder_2 from a FLUX base repo — the same companion pattern, now expressible declaratively.

🤖 Generated with Claude Code

…ntroller longcat-video loads a checkpoint directory: its backend.py takes request.ModelFile when os.path.isdir(request.ModelFile) and otherwise falls back to snapshot_download. That places it in the same class as transformers/vllm/diffusers/sglang, but the allow-list added in #10910 did not enumerate it, so PrimaryArtifactSpec returned no managed artifact for a bare HuggingFace repo id. The consequence in distributed mode: nothing was acquired on the controller, ModelFileName fell through to the raw repo id, and staging skipped the resulting phantom /models/<owner>/<repo> path. The worker received a blank ModelFile, fell back to request.Model, and downloaded ~83GB from HuggingFace inside the remote LoadModel deadline - so the load could only ever fail with DeadlineExceeded while an abandoned backend process kept downloading. Note this materializes the full repository. The backend restricts its own snapshot_download with allow_patterns, and the avatar repo ships both base_model/ and base_model_int8/ where only one is ever loaded; inferred specs have no way to carry patterns today. Tracked separately. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

stageModelFiles logs "Staging model files for remote node" up front, then silently drops any path field that does not exist on the controller. The skip itself is legitimate and must stay: a backend outside managedArtifactBackends that takes a bare HuggingFace repo id gets an optimistically constructed path (ModelFileName falls through to the raw model reference) that was never materialized, and sources its own weights on the worker. Erroring would break those configs. But at debug level the operator is left with a reassuring staging line and no trace of the skip, so a genuine controller-side acquisition gap is indistinguishable from a healthy pass-through - it surfaces much later as a remote LoadModel timeout, on a worker that is quietly downloading tens of gigabytes. Raise the skip to warn and name the field, path, node and tracking key. Behavior is unchanged. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

A composed pipeline needs more than one HuggingFace snapshot. LongCat-Video-Avatar-1.5 loads its own transformer but takes the tokenizer, text encoder and VAE from the separate LongCat-Video base repo, so a single-artifact config cannot express it and the backend is left to fetch the second repo itself at load time. Widen the artifact model to target: model plus any number of named target: companion entries. Normalize accepts the new target and constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that name is the option key the backend later receives; a companion may not claim primary_file, which only means anything for a load target. ModelConfig.Validate requires exactly one primary and requires it first, since Artifacts[0] is what ModelFileName, size estimation and staging all resolve from. Both acquisition paths now loop instead of touching index 0 alone: preloadOne for an already-installed config, bindPrimaryArtifact for a gallery install. Failure policy differs by provenance. An inferred primary keeps its warn-and-fall-back, because the legacy download path still exists for it. Companions are explicit by construction, so they are all-or-nothing: a config naming one is asserting the backend needs it, and failing at the acquisition boundary is far more legible than a missing-weights error surfacing later inside the backend. The cache key is deliberately unchanged. It hashes source identity only, never name or target, so every already-installed managed model still hits its existing snapshot instead of silently re-downloading. Two specs pin that: one proving a companion and a primary with identical sources agree on the key, and one pinning the digest of a known primary outright. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

A materialized companion is useless until the backend can find it, and its location is a content-addressed cache key that does not exist until the artifact resolves. A static gallery override cannot carry that, and persisting it into the config YAML would rot the moment a re-resolve produced a new key. Synthesize it instead at load time: each resolved companion becomes "<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the key:value convention backends already parse for options like attention_backend. The value stays relative to the models directory so a remote worker can resolve it under its own ModelPath once staging has rewritten the model root. An option the author set explicitly always wins, so pinning a companion to a local checkout still beats the managed snapshot. longcat-video resolves base_model through ModelPath, the same convention qwen-tts, voxcpm, outetts and ace-step already use for companion assets. Its sibling-directory heuristic is deleted: it looked for a LongCat-Video directory next to the model, which cannot exist under the content addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead code the moment the model became managed. The gallery entry declares both repositories and restricts each with allow_patterns. The avatar repo ships base_model/ and base_model_int8/ and only ever loads one, so fetching the whole repo would roughly double the download. The patterns match the entry's own options (use_distill true, use_int8 default false); enabling use_int8 here also requires adding base_model_int8/**, which is called out in the entry. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Staging anchored the worker's models directory on the primary snapshot whenever a model was managed, so a companion snapshot could not reach the worker at all. frontendModelsDir was derived by stripping the Model relative path off the end of ModelFile. For a managed artifact nothing matches: ModelFile is .artifacts/huggingface/<key>/snapshot while Model stays a bare HuggingFace repo id, so the strip was a no-op and the "models directory" came out as the snapshot itself. Two consequences, both silent. Staging keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two snapshots of one model were indistinguishable on the worker. And a companion, which lives in a sibling snapshot directory outside the primary, fell outside that directory entirely: StagingKeyMapper.Key collapsed its files to bare basenames and resolveOptionPath could not resolve the relative option at all, so it was skipped without a word. Derive the models root from the artifact tree instead when the path runs through it, and compute the worker's ModelPath from the file's path relative to that root rather than from the Model field. The legacy layout is unaffected: where Model really is the relative path, the new derivation reduces to the old one, which a regression spec pins. This deliberately changes an invariant that router_dirstage_test.go pinned: for a managed primary, ModelFile and ModelPath were both the snapshot directory, and staging keys were relative to it. Now ModelFile is the snapshot, ModelPath is the models root above it, and keys keep the full relative path. That spec is updated rather than accommodated, with the reasoning recorded inline, because the old invariant is exactly what made a sibling companion unreachable. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

… coverage-ratchet fixes (#10989) * fix(http): make /readyz reflect startup readiness instead of always 200 /readyz was registered as a static handler returning 200 unconditionally, so it carried no information: it was green whenever it could be reached at all. Readiness could not distinguish "serving" from "still starting", and any future change that started the HTTP listener earlier would silently turn the probe into a lie. Track startup completion on the Application (atomic flag, flipped at the very end of New() on the success path only) and have the readiness handler consult it per request, returning 503 with a small JSON body while startup is in progress. A nil readiness source fails open so embedders keep the historical behaviour. /healthz is deliberately left readiness-independent. Liveness and readiness answer different questions, and failing liveness during a long preload makes an orchestrator restart the pod mid-download so the preload never finishes. This matters because since #10949 the startup preload materializes HuggingFace artifacts for managed backends: tens of GB for a large model (31 GB observed on a live cluster). Both probes stay in quietPaths and stay exempt from auth. Note the listener is still started only after New() returns, so today the not-ready state is not observable over HTTP. Moving the listener earlier is a separate, deliberate decision and is not made here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gitignore): anchor the mock-backend pattern so its source dir is traversable The bare `mock-backend` pattern matched the *directory* tests/e2e/mock-backend/, not just the binary built into it. Git will not descend into an ignored directory even for tracked files, so `git add tests/e2e/mock-backend/main.go` required -f. This was hit while working on #10970. Anchor it to the artifact's full path. The built binary stays ignored (it is also covered by tests/e2e/mock-backend/.gitignore) while the source directory becomes traversable again. Verified with `git check-ignore -v`: a new source file under tests/e2e/mock-backend/ is no longer ignored, and the binary produced by `make build-mock-backend` still is. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(coverage): raise the coverage ratchet from 48.5% to 54.2% The committed baseline had drifted well below reality: it still read 48.5% while a full instrumented run measures 54.2%. A stale-low baseline makes the gate meaningless — coverage could regress by more than 5 percentage points and still pass. Raising a ratchet is a deliberate act, not something to fold into an unrelated fix, so it gets its own commit. The headroom was earned by tests landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and #10975. Measured with `make test-coverage` on this branch (the same instrumented run `make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and pkg/..., generated protobuf excluded). The run completed with exit 0 and zero spec failures; the total was then written with the exact command the test-coverage-baseline target uses: go tool cover -func=coverage/coverage.out \ | awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt Verified afterwards with scripts/coverage-check.sh, which reports OK. Note the measured figure includes the readiness specs added earlier on this branch, so it is a demonstrated floor rather than an estimate. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…HEALTHCHECK (#10999) fix(worker): give the worker a real health endpoint (#10987) The image bakes in a single HEALTHCHECK that curls http://localhost:8080/readyz, but the same image also runs `local-ai worker`, which serves HTTP on the gRPC base port minus one and never binds 8080. Every worker container was therefore permanently `unhealthy` (43 consecutive failures observed on a production node), which is worse than having no healthcheck: a genuinely broken worker and a perfectly good one both report `unhealthy`, so the signal carries no information and orchestration that keys on it misbehaves. The worker already served /readyz on that port via the file-transfer server, but as a constant 200 — it only proved the listener was bound, which is precisely the failure mode at issue. Readiness now tracks the live NATS connection: all of a worker's actual work (backend lifecycle events, inference dispatch, file staging) arrives over NATS, so a worker whose link is dead is up and useless. Registration is already implied, since the server only starts after registration succeeds. This reports something the controller cannot already see. The node registry's status/last_heartbeat is fed by an HTTP heartbeat to the frontend, a different network path from NATS — a worker can keep heartbeating while its NATS connection is dead and still look healthy in the registry. /healthz stays a constant 200: liveness must not follow readiness, or a NATS blip becomes a cluster-wide restart storm. The HEALTHCHECK is now a script that derives its endpoint from the mode the container is actually running plus the env vars that configure the bind address, so a frontend moved off 8080 with LOCALAI_ADDRESS (broken the same way) and a worker on a non-default base port are both probed correctly. Modes with no HTTP surface (agent-worker, one-shot commands) report healthy rather than false-unhealthy. HEALTHCHECK_ENDPOINT remains as an explicit override, so the workaround shipped in docker-compose.distributed.yaml keeps working; both overrides in that file are now unnecessary and have been removed. Also fixes the latent --start-period gap. Since #10949 a frontend's startup preload materializes HuggingFace artifacts before the HTTP server binds (31 GB observed on a live cluster), so a healthy replica can legitimately fail probes for a long time. --start-period is Docker's knob for exactly this: failures inside it leave the container `starting` instead of burning retries, and it ends early on the first success, so a generous 60m costs a fast-starting container nothing. --timeout drops from 10m to 10s — it is a per-probe deadline, and a localhost curl that has not answered in 10s is itself the fault being detected. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

…et the base_model option (#11075) fix(model-artifacts): persist companion artifacts, not just the primary A managed model can declare companion artifacts (LongCat-Video-Avatar-1.5 pulls its tokenizer, text encoder and VAE from the separate LongCat-Video base repo via a target: companion artifact). preloadOne resolves the whole set in memory, but the binding written back to disk carried only the primary: persistArtifactBinding marshalled []Spec{result.Spec} and replaced the entire artifacts: list with it, silently dropping every companion. In a single process the loss is invisible because the in-memory config keeps the companion. It bites on the next controller restart: the config reloads from the mangled file with the primary alone, so withCompanionArtifactOptions finds no resolved companion and synthesizes no base_model option. The remote longcat-video backend then never receives base_model, falls back to BASE_MODEL_ID and downloads the repo itself ("Downloading required files for meituan-longcat/LongCat-Video"), failing the load with "base_model must point to a LongCat-Video checkpoint". This is why an explicit base_model:<path> added to the config options works where the managed companion does not: an explicit option lives in options:, which is never rewritten, while the managed companion lives in artifacts:, which the binding overwrote. Persist the full resolved set (primary + every companion), and widen bindingNeedsPersistence to compare the whole artifact list so a companion resolving for the first time still triggers a write. The single-node path is unaffected: there the in-memory config already carried the companion, and the staging/ModelPath resolution for a remote worker (nested per-model staged root, #10949) is unchanged and already correct once the option is generated. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

mudler added 5 commits July 18, 2026 23:40

mudler merged commit 626ae4d into master Jul 19, 2026
67 checks passed

mudler deleted the investigate/hf-materializer-staging branch July 19, 2026 10:01

localai-bot mentioned this pull request Jul 20, 2026

fix(http): make /readyz reflect startup readiness, plus gitignore and coverage-ratchet fixes #10989

Merged

localai-bot mentioned this pull request Jul 20, 2026

fix(worker): give the worker a real health endpoint and a mode-aware HEALTHCHECK #10999

Merged

localai-bot mentioned this pull request Jul 23, 2026

fix(model-artifacts): persist companion artifacts so remote workers get the base_model option #11075

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos#10949

fix(model-artifacts): materialize longcat-video on the controller, and support companion repos#10949
mudler merged 5 commits into
masterfrom
investigate/hf-materializer-staging

localai-bot commented Jul 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

localai-bot commented Jul 19, 2026

Root cause — a regression from #10910

Changes

Tradeoff worth reviewing

Testing

Merge notes

Future cleanup (not in this PR)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants