Very granular application of LoRAs

[UmerHA]

Hey all 👋🏽, we shipped a feature that allows you to very granularly apply LoRAs. Here's the PR: #7352.

What?

When loading LoRAs, you now have granular control over how much the LoRAs affect each part of the pipeline.

Before the PR, you could use a single scale number per LoRA that would be applied to the entire pipeline (i.e., unet, text_encoder, and optional text_encoder_2). With this PR, you can now (i) set a scale number for text_encoder and text_encoder_2 separately, and (ii) be as detailed as you want for the UNet - up to per transformer!

Let's say we have a single LoRA named "cartoonify" loaded and want to specify different scales for the text_encoders and the down, mid, and up parts of the UNet. Here you go:

adapter_weight_scales = {
    "text_encoder": 0.5,
    "text_encoder_2": 0.8,
    "unet": {
        "down": 1, 
        "mid": 0.9,
        "up": 0
    }
}
pipe.set_adapters("cartoonify", adapter_weight_scales)

Here's an example with two LoRA where the second is applied very granularly:

scales_1 = 0.9
scales_2 = {
    "text_encoder": 0.5,
    "text_encoder_2": 0.5,
    "unet": {
        "down": 0.9,  # all transformers in the down-part will use scale 0.9
        # "mid"  # because "mid" is not given, all transformers in the mid part will use the default scale 1.0
        "up": {
            "block_0": 0.6,  # all 3 transformers in the 0th block in the up-part will use scale 0.6
            "block_1": [0.4, 0.8, 1.0],  # the 3 transformers in the 1st block in the up-part will use scales 0.4, 0.8 and 1.0 respectively
        }
    }
}
pipe.set_adapters(["my_adapter_1", "my_adapter_2"], [scales_1, scales_2])

Why?

Only applying LoRAs to certain pipeline parts changes the image output. So this PR gives users another tool in their quest to get exactly the image they imagined.

The inspiration comes from this extension for AUTOMATIC1111's stable-diffusion-webui. Here's an example (taken from their repo) that illustrates how the same LoRA gives different results depending on where it is applied.

How? - artist's perspective

Are there rules on how applying LoRAs to different parts with different scale strengths changes the image? Currently, no. So far we only know that is changes the output. How it does so is still for us all to figure out. Please share any insights you have in this thread!

---

Let me know what you think!

Cheers,
Umer

@UmerHAdil on Twitter

---

cc @sayakpaul @asomoza

[Scorpinaus]

Nice! Can this be used for the other types such as Lycoris? Locons, lohas?

[UmerHA]

It should. Other LoRA types are normalized, i.e. converted into the diffusers LoRA format. So it should work. Feel free to test it and report back!