[Control-LoRA] Picking Control-LoRA up again

[sayakpaul]

Stability AI's Control LoRA has been around for a while: https://huggingface.co/stabilityai/control-lora.

We also have had requests for it, e.g., #4679.

I even started working on it: #4899, but I couldn't reproduce the results. I have tried debugging it from the ComfyUI codebase, too, but couldn't actually find any non-trivial differences.

Would be great if someone from the community picked it up. Especially cc: @HighCWu (because of https://github.com/HighCWu/control-lora-v2).

[shauray8]

@sayakpaul I can have a look at this, Are there any specific architectural changes you have in mind apart from what you described in #4899, If not maybe I should start where you left off with finding the point where it starts to diverge.

[sayakpaul]

Sure. Nothing specific in mind regarding architectural changes.

[samedii]

This seems like it could be a useful reference as a very minimal implementation: https://github.com/Mikubill/sd-webui-controlnet/pull/1936/files#diff-02f1f09af10a0ab88a72b1ddf7adb637edb649364a5125b2df6756976f85350a

[amosyou]

hi! is this still a desired feature? interested in making this happen

[sayakpaul]

Sure contributions are welcome. Happy to help.

[lavinal712]

Hi! Half a year ago I use peft to reproduce control-lora-v2, it works well. Maybe it will help you @sayakpaul . https://github.com/[lavinal712/control-lora-v3](https://github.com/lavinal712/control-lora-v3)

[sayakpaul]

What is Control LoRA V2? Were you able to load up the SAI Control LoRA weights in diffusers with your implementation?