-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SD3 IP-Adapter runtime checkpoint conversion #10718
Conversation
Example output: Inference codeimport os
import torch
from pathlib import Path
from diffusers import StableDiffusion3Pipeline
from diffusers.utils import load_image
from transformers import SiglipVisionModel, SiglipImageProcessor
model_id = "stabilityai/stable-diffusion-3.5-large"
image_encoder_id = "google/siglip-so400m-patch14-384"
ip_adapter_id = "InstantX/SD3.5-Large-IP-Adapter"
feature_extractor = SiglipImageProcessor.from_pretrained(
image_encoder_id, torch_dtype=torch.bfloat16
)
image_encoder = SiglipVisionModel.from_pretrained(
image_encoder_id, torch_dtype=torch.bfloat16
)
pipe = StableDiffusion3Pipeline.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
feature_extractor=feature_extractor,
image_encoder=image_encoder,
)
# Load IP Adapter
pipe.load_ip_adapter(ip_adapter_id, "ip-adapter.bin")
pipe.set_ip_adapter_scale(0.5)
pipe._exclude_from_cpu_offload.append("image_encoder")
pipe.enable_sequential_cpu_offload()
# Input
ip_adapter_img = load_image("astronaut.jpg")
# please note that SD3.5 Large is sensitive to highres generation like 1536x1536
image = pipe(
width=1024,
height=1024,
prompt="an astronaut on top of a waffle",
negative_prompt="lowres, low quality, worst quality",
num_images_per_prompt=1,
generator=torch.manual_seed(42),
ip_adapter_image=ip_adapter_img,
guidance_scale=6,
num_inference_steps=24,
).images[0]
image.save("result.jpg") Prompt: "an astronaut on top of a waffle" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @guiyrt. We can also change the checkpoint in docs/examples back to the original. You can also go ahead and add Fixes #9966
to close the issue when this is merged.
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Done and done, thanks for the review @hlky ! |
Updated PR accounting for #10728 |
thanks a lot everyone!! |
What does this PR do?
Fixes #9966.
Now IP-Adapter from InstantX/SD3.5-Large-IP-Adapter can be used directly, with checkpoint runtime conversion.
I also changed the structure to have the familiar
_convert_ip_adapter_attn_to_diffusers
and_convert_ip_adapter_image_proj_to_diffusers
functions.Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@hlky @yiyixuxu