KolorsPipeline does not support from_single_file

[Thekey756]

from diffusers import KolorsPipeline
KolorsPipeline.from_single_file("models/kolrs-8steps.safetensors")

How does KolorsPipeline load a single file model?

[hlky]

Can you link the model you are trying to load, seems it would be this?

[hlky]

That model is already in Diffusers format, it's just the UNet2DConditionModel, we can load it straight to pipe.unet. Alternatively a version with this UNet2DConditionModel could be uploaded to the Hub then it could be used directly with KolorsPipeline.from_pretrained.

import torch
from diffusers import KolorsPipeline, EulerAncestralDiscreteScheduler
import safetensors.torch

weights = safetensors.torch.load_file("kolors8AcceleratedVersionOf8_accelerated.safetensors")

pipe = KolorsPipeline.from_pretrained(
    "Kwai-Kolors/Kolors-diffusers", variant="fp16", torch_dtype=torch.float16
)
pipe.unet.load_state_dict(weights)
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)
pipe = pipe.to("cuda")
prompt = (
    "A photo of a ladybug, macro, zoom, high quality, film, holding a wooden sign with the text 'KOLORS'"
)
image = pipe(prompt, num_inference_steps=10, height=720, width=1280, guidance_scale=1.0, generator=torch.Generator().manual_seed(0)).images[0]
image.save("kolors.png")

Parameters num_inference_steps=10, guidance_scale=1.0, resolution and EulerAncestral seem to be important, I get bad output with the defaults.

Common parameters:

CFG 1 / Steps 10 / Sampler Euler a / VAE sdxl-vae-fp16-fix.safetensors

Size: 1280x720 / 1536x768 / 960x1920 / 1920x960 / 2048x1024 / 1024x2048

cc @a-r-r-o-w @DN6 from_single_file won't work here, looks like we should check for some known Diffusers format key and return if it's already converted.

diffusers/src/diffusers/loaders/single_file_utils.py

Line 960 in cef0e36

def convert_ldm_unet_checkpoint(checkpoint, config, extract_ema=False, **kwargs):

Failed to load UNet2DConditionModel. Weights for this component appear to be missing in the checkpoint.

edit: Just returning early breaks detection (e.g. XL and this Kolors unet are detected as v1), we also need to update CHECKPOINT_KEY_NAMES with Diffusers versions, we already do this for controlnet and flux

diffusers/src/diffusers/loaders/single_file_utils.py

Line 60 in cef0e36

CHECKPOINT_KEY_NAMES = {

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.