Dynamic Lora Selection In Runtime❓

[teneous]

I have a GPU 4090 with 24GB of memory, and I'm using SDXL and SDXL Lora. I want to achieve the following effect:

Within the limited 24GB memory, suppose I have 26 Loras named from A to Z. I want to implement pseudocode such that when a request passes the name of a Lora, if it's Z, for example, I'll load Z, set its weight to 0.3, perform inference, and then unload this Lora, but cuda would not release memory , and also this would waste a lot of loading time.

So, I thought of using the LFU mechanism to load 5 to 10 Loras at once. Then, by checking if the requested Lora name hits the currently loaded list, if it does, set its weight to 0.3 and others to 0. If it doesn't hit, load it and set the weights of others to 0. I referred to the official documentation of diffusers.

Below is my code:

# recover unet
pipe.unfuse_lora()
# lora_code
if ip_code is None:
    pipe.disable_lora()
else:
    lora = global_config.get_lora(ip_code)
    # load lora
    pipe.load_lora_weights(pretrained_model_name_or_path_or_dict=lora.path, adapter_name=lora.name)
    pipe.fuse_lora(lora.name)
    pipe.enable_lora()

My implementation is flawed because load_lora_weights cannot execute the same name twice. Is there a better way to implement this request?

thanks in advance

[asomoza]

Any reason as to why you're using fuse_lora? I only use fuse_lora when I want to save the merged model with the lora.

In my app I can do this:

The only LoRA that gets used in that image is the Chocolate Coffee. I can achieve this with set_adapters, in my code, when you don't press the X in the UI, the LoRAs are kept in the VRAM.

In short, I manage a list of LoRAs, to set which are loaded in the VRAM or used depends if the user adds, enables or adjusts their weights:

        if self.lora:
            if isinstance(self.lora, list):
                unzipped_list = zip(*self.lora)
                reordered_list = [list(item) for item in unzipped_list]
                self.model.set_adapters(*reordered_list)
            else:
                self.model.set_adapters([self.lora[0]], [self.lora[1]])

This is for my code which is a little more complex than just a list and I don't use pipelines, but with the set_adapters, any lora that you don't pass won't get used, you'll need to set the names and remember them in your code. You can also get the list of active adapters if you need that to validate them before activating them.

Maybe I'm missing something in what you're trying to do, but all of this is in the documentation:

https://huggingface.co/docs/diffusers/tutorials/using_peft_for_inference

If you need help on how to maintain a list of LoRAs to use the LFU mechanism and activate or load them in diffusers, that's more a coding question than a diffusers question, I can still help you but you'll need to provide a full reproducible snippet with your problem.

Edit: if you're using the fuse_lora for generation speed, I don't think that's going to help you if you'll have to keep fusing and unloading them, apart from the merge to model, the only other use case is when you know you're going to use the same loras with the same scale for a long time which is not your case for what I can see.

[sayakpaul]

Cc: @BenjaminBossan in case you have anything more to add.

[BenjaminBossan]

Not sure, @asomoza gave a great response. To me, it's not clear if using an LFU cache is super helpful, I guess it depends on the access pattern and the memory constraint. I'd probably try to make room to load all LoRAs at once if possible and then switch between them without the extra complication.