StableDiffusionXLControlNetInpaintPipeline images look weird

[tiimgreen]

I'm using the Real Vis inpainting model (https://civitai.com/models/139562?modelVersionId=297320) which has a baked VAE. But my images are coming out like this (subject is same, BG is inpainted):

See below for my code. Is this a VAE issue? I tried specifying vae=None (since the VAE is baked) but that doesn't work. Any ideas here?

controlnet = ControlNetModel.from_pretrained("diffusers/controlnet-canny-sdxl-1.0")

pipe = StableDiffusionXLControlNetInpaintPipeline.from_single_file(
    "./realvisxlV40_v30InpaintBakedvae.safetensors",
    torch_dtype=torch.float16,
    controlnet=controlnet,
    use_safetensors=True,
    add_watermarker=False,
    low_cpu_mem_usage=True,
)
pipe = pipe.to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config, algorithm_type="sde-dpmsolver++", use_karras_sigmas=True)

[sayakpaul]

1. Have you tried using any other schedulers?
2. Not sure if the underlying base pipeline checkpoint would yield good results with the ControlNet checkpoint because it wasn't trained on it.

[tiimgreen]

Ok I tried exactly those settings and got the same problem. So either my images are weird somehow or its the machine I'm running it on.

Could you send your images please?

Also how are you running this? I'm on RunPod using the runpod/pytorch:2.0.1-py3.10-cuda11.8.0-devel-ubuntu22.04 image

[asomoza]

I did the last one on windows, the firsts ones were on linux, I switch a lot. In windows I use pytorch 2.1.2 and in linux pytorch 2.2.2 both with cuda 12.1

Here are the images, you can click on them to get the full size image.

source	mask	canny

[tiimgreen]

Ok with your images it works. I see that yours are 1024 and mine are 512, could be that? Or maybe some weird encoding stuff?

I'll do some digging though, this was helpful. Thanks!

[asomoza]

yeah, that's it, SDXL works with 1024, don't use 512 with SDXL unless you use the turbo model.

[tiimgreen]

Ah ok, I was migrating from a 1.5 model, didn't think to check that. Thanks for your help