Not getting good realistic results with Hyper-SD + IP-Adapter

[juancopi81]

Hi everyone, (maybe you @asomoza know about this?)

Does hyper-sd works well with IP-Adapter? I am testing hyper-sd in Diffusers as explained in the repo. I thought that I was going to get better results than LCM-LoRA but my results so far are in fact bad when used with IP-Adapter. They seem very cartoonish to me.

Here is the code I am using, taken from the Model Card in HF:

!pip install -U diffusers peft
!pip list | grep diffusers # -> 0.27.2

import torch
from diffusers.utils import load_image
import numpy as np
import cv2
from PIL import Image
# Tests with different schedulers
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline, DDIMScheduler, TCDScheduler, LCMScheduler
from huggingface_hub import hf_hub_download

controlnet_checkpoint = "lllyasviel/control_v11p_sd15_openpose"
# Load original image
cn_image = load_image("assets/neutral_openpose_woman.jpg")

# Initialize pipeline
controlnet = ControlNetModel.from_pretrained(controlnet_checkpoint, torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained("SG161222/Realistic_Vision_V6.0_B1_noVAE", controlnet=controlnet, torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights(hf_hub_download("ByteDance/Hyper-SD", "Hyper-SD15-8steps-lora.safetensors"))
pipe.fuse_lora()

face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-full-face_sd15.bin")
pipe.set_ip_adapter_scale(0.85)

# Ensure ddim scheduler timestep spacing set as trailing !!!
# Tested with different schedulers: DDIM, LCM, TCD...
# pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
pipe.scheduler = TCDScheduler.from_config(pipe.scheduler.config)
prompt="RAW photo, a photo of a woman wearing jeans and a t-shirt. Best quality. hyper-realistic photo, detailed face"
negative_prompt="worst quality, low quality, extra fingers, extra arms, extra ears"
image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=8, image=cn_image, guidance_scale=0, eta=1.0, ip_adapter_image=face_image).images[0]
image.save('image_out.png')

These are the relevante images:
ControlNet Image:

IP-Adaper Image:

Resulting Image:

For comparison, if I keep everything the same and use LCM-LoRA, changing only these lines:

pipe.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)

I get this much more realistic image:

Not sure if I am doing something wrong or if it is just that ip-adapter does not yield realistic results with hyper-sd. Any help will be highly appreciated!

Thanks!

[asomoza]

I haven't really tested Hyper-SD with SD 1.5, when I have time later I can do it. But with SDXL I don't have any problems, just a quick test since I can't download all what you're using right now, but with a depth controlnet and your images.

Using RealVis, TCD and the 8-step LoRA:

without mask	with mask

If I just use the IP Adapter

normal	plus

So for now I can reduce that what's happening with you is either the controlnet (guidance to high?) or just that SD 1.5 can't handle it.

[juancopi81]

Thank you very much for your answer @asomoza, I was playing with different conditioning scales, but still very cartoonish, maybe it is how it behaves for SD 1.5 🤔

[asomoza]

I did some tests with this, I'm not that familiar with SD 1.5 anymore and also I'm too used to the SDXL results, so I can't really tell when something is good or bad with SD 1.5 unless it's really bad.

With your IP Face Image, If I use a Plus Face IP Adapter:

Euler	DPMSolverMultistep

And with Hyper-SD:

TCD	DPMSolverMultistep

All the results don't look real to me, but the Hyper-SD results look more cartoonish. Still they're not as bad as the image you posted.

Bytedance released a new one that can be used with CFG: Hyper-SD15-8steps-CFG-lora.safetensors, with that one and a CFG of 1.8, it looks decent, and with 3.0 it looks like the old one.

I think you're right, it has to do with how SD 1.5 behaves. The normal Hyper-SD with IP Adapters behaves like if it has a high CFG which makes the results over saturated and cartoonish.

CFG 2.0	CFG 3.0

[juancopi81]

Thanks @asomoza! I'll give it a try to Hyper-SD15-8steps-CFG-lora.safetensors! Hopefully works better with the ip-adapter-

[juancopi81]

Yeap! I can confirm this one is working much better

[juancopi81]

Maybe the ip-adapter needs some CFG in SD1.5 so that's why this model works better 🤔