Support multiple control nets in the `StableDiffusionControlNetXSPipeline`/`StableDiffusionXLControlNetXSPipeline`

[chrismaltais]

Is your feature request related to a problem? Please describe.
The StableDiffusionControlNetPipeline/StableDiffusionXLControlNetPipeline allow you to specify a list of control nets (i.e. depth and canny)

However, the new ControlNetXS pipelines only allow you provide a single control net

Describe the solution you'd like.
I want to be able to do:

vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
canny_controlnet = ControlNetXSAdapter.from_pretrained(
    "UmerHA/Testing-ConrolNetXS-SDXL-canny", torch_dtype=torch.float16
)
depth_controlnet = ControlNetXSAdapter.from_pretrained(
    "UmerHA/Testing-ConrolNetXS-SDXL-depth", torch_dtype=torch.float16
)
pipe = StableDiffusionXLControlNetXSPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=[canny_controlnet, depth_controlnet], torch_dtype=torch.float16
)

[sayakpaul]

Thanks for your interest. Would you want to open a PR to add this support? We'd be more than happy to guide you. Cc: @UmerHA as well.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[KarthikSundar2002]

I would like to work in this issue, and this is my first time contributing to Diffusers, so would really like some guidelines on where to start

[sayakpaul]

Thanks for that. Feel free to start working on it and ask questions along the way.

[KarthikSundar2002]

I looked into the codebase for a while and also read the corresponding ControlNetXS Paper. I have observed the following

1. StableDiffusionControlNetPipeline creates MultiControlNetModel which then adds all the outputs of the list of controllers
2. On the other hand, StableDiffusionControlNetXSPipeline doesn't allow List Inputs and thus doesn't create a MultiControlNetModel

Possible Solutions

1. MultiControlNetModel's functionality is to just run the forward pass for the given controllers and add them together. Thus we could just add support for ControlNetXSAdaptor in the MultiControlNetModel class. This shouldn't be complicated as the forward pass of both ControlNetModel and ControlNetXSAdaptor are similar
2. We could have a separate class (MultiControlNetXS) and then just run the forward passes and add them together there

I personally feel the first way is good and solves the issue. What are your thoughts? Any Suggestions? Is this the right direction?

[sayakpaul]

2. could be better as it keeps things isolated. @DN6 @yiyixuxu WDYT?

[UmerHA]

@KarthikSundar2002 @sayakpaul

I'm not sure this is possible in principle.

For controlnet (not xs), the controlnet part processes information independently of the main model, and its information is injected into the main model at a single point only. Thinks of it like computing an embedding and then adding it to a representation. So if we have 2 controlnets, they can process information independently of each other (ie compute their embeddings respectively), and then inject their info into the main model (ie add both embeddings to the representation).

For controlnet-xs the information flows to and from the main model many times. So if we have 2 controlnet-xs's, their information becomes muddled together. Ie the embeddings can't be computed correctly in the first place.

Happy to be convinced otherwise by empirical data!

[KarthikSundar2002]

Oh that makes sense, like the base model's feedback signal will have information corresponding to both the controlnet's and we won't be able to separate it into it's constituents. I thought of adding an encoding (something similar to that of positional encoding) to bypass this issue, but then we have to modify the base model's architecture, thus neglecting the premise of controlnets