Experimental per control type scale for ControlNet Union

[hlky]

What does this PR do?

This adds experimental support for per control type scale to ControlNet Union.

Code

from diffusers import (
    StableDiffusionXLControlNetUnionPipeline,
    ControlNetUnionModel,
    AutoencoderKL,
)
from diffusers.utils import load_image
import torch

prompt = "A bird"
seg = load_image("house_seg.png")
canny = load_image("bird_canny.png")
controlnet = ControlNetUnionModel.from_pretrained(
    "xinsir/controlnet-union-sdxl-1.0", torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetUnionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe.enable_model_cpu_offload()
image = pipe(
    prompt,
    control_image=[seg, canny],
    control_mode=[5, 3],
    height=1024,
    width=1024,
    controlnet_conditioning_scale=[0.0, 1.0],
    generator=torch.Generator().manual_seed(0),
).images[0]

Control images

Canny	Canny Seg	Canny No Seg

Canny: control_image=[canny]
Canny Seg: control_image=[seg, canny]
Canny No Seg: control_image=[seg, canny] with controlnet_conditioning_scale=[0.0, 1.0]

Fixes #10656

Note: we will also add MultiControlNet support for ControlNet Union in another PR.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu @asomoza @vladmandic

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[john09282922]

Is this one possible to use pro-max version as well?
also if it is possible, can you merge to sdxl_union_inpaint model as well?
@hlky

Thanks

[hlky]

@john09282922 Yes.

[hlky]

Pose+Seg	Pose	Seg

import torch

from diffusers import StableDiffusionXLControlNetUnionPipeline
from diffusers.models import ControlNetUnionModel, AutoencoderKL
from diffusers.utils import load_image

room_seg_img = load_image(
    "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/room_seg.png"
)
pose_img = load_image(
    "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/pose.png"
)

controlnet = ControlNetUnionModel.from_pretrained(
    "brad-twinkl/controlnet-union-sdxl-1.0-promax", torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained(
    "madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetUnionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    torch_dtype=torch.float16,
    variant="fp16",
)
pipe.enable_model_cpu_offload()

image = pipe(
    prompt="an astronaut in space",
    width=1024,
    height=1024,
    negative_prompt="lowres, low quality, worst quality",
    control_image=[pose_img, room_seg_img],
    control_mode=[0, 5],
    #controlnet_conditioning_scale=[0.0, 1.0],
    num_inference_steps=50,
    generator=torch.Generator().manual_seed(42),
).images[0]

image.save("result.jpg")

Also unifies the interface for MultiControlNet cc @vladmandic simply splits to list of list internally

cc @guiyrt However I can't seem to reproduce the results here with MultiControlNet, checked on PR and main. Edit: seems ok now

Pose+Seg	Pose	Seg

Uses standard scaling with MultiControlNet and experimental scaling otherwise.