Is there any lighter version of the ControlNet model?

[AHHHZ975]

Hi,

I know there are a couple of ControlNet models trained on the base Stable Diffusion v1-5 model (e.g. lllyasviel/sd-controlnet-depth). However, I am looking for a lighter and faster version of this type of ControlNet model which is trained on a lighter version of the base SD model (e.g. ByteDance/SDXL-Lightning or segmind/tiny-sd). But, I could not find any.

Can anyone please suggest the papers/models that propose a version of ControlNet models that are lighter and faster than the original ones? Thanks!

Best,
Amir

[asomoza]

Hi, we currently don't support it because of the lack of bandwidth but there's control loras.

I really don't recommend anything lighter or faster specially for lighting and distilled models because you will get worse results since they will by less accurate and have less steps to condition the generation.

Also, practically every older controlnet is subpar right now if you compare it to the latest ones like controlnet union, also it should be faster if you take into consideration that you can use multiple condition images with just one controlnet.

Also you're kind of mixing the architectures, you started with SD 1.5 but the ones you're mentioning are based in SDXL, so the controlnets are going to be bigger anyways.

[AHHHZ975]

Hi Álvaro,

Thanks for the detailed reply and your suggestions. Sorry for the confusion about the mixing architectures. What I meant by those architectures is that, with the same base model (whether it is a normal SD model or an SDXL or whatever), is there any ControlNet model that performs faster?

Because, in the current inference code I have now, my only concern and the bottleneck is that from the time I feed the text prompt and the condition signal to the ControlNet model until it gives me the output image, it takes around a couple of seconds (e.g. ~3 seconds one of my GPUs). So, I need to reduce this computation time for the inference and one of the ideas that I came up with is to replace the ControlNet model with a lighter one. So, that's why I am asking for that. Although there were other ideas like reducing the number of diffusion steps, they affected the quality of the output image a lot.

But, if you have any other suggestions to reduce the inference time of the ControlNet model (of course without compromising the quality of the output image too much!), it would be more than welcome. Thanks!

Best,
Amir

[asomoza]

in my experience t2i adapters are faster and lighter but less accurate, sadly in my tests and experience, the controlnet union and a couple of the last ones, are the only ones I use because the difference in quality is too much.

You can still do some other techniques for speeding up inference, for example you don't need to use the CFG on all the steps, in my tests you can use it (in the case of lighting) for 2 steps and then do the rest of the generation without it, also depending on your use case, you can also stop the controlnet early.

I did something like that for this demo space and it's fast (with controlnet union) if you use a good GPU, I remember it took less than 2s on a 3090 but I could be wrong and don't have the time to run it locally right now.

[AHHHZ975]

Thank you for the great suggestions and for sharing your experiences. I will try t2i adapters, controlnet union, and control loras in my project to see what the results are and I will let you know here if I have any questions. Also, the technique you mentioned to not use the CFG on all steps is great and I believe it can have a huge impact. Thanks!