What's the most efficient way of loading controlnet models?

[AbhinavGopal]

Hey guys, I've tried the following two scripts:

from diffusers import ControlNetModel, DDIMScheduler, StableDiffusionXLControlNetPipeline
from diffusers.models import AutoencoderKL
import torch
import time
cur_time = time.time()
controlnet_names = ["thibaud/controlnet-openpose-sdxl-1.0"]
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, token="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").to('cuda')
controlnets = [ControlNetModel.from_pretrained(x, torch_dtype=torch.float16, token="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").to('cuda') for x in controlnet_names]
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnets, torch_dtype=torch.float16, safety_checker=None, vae=vae, token="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").to('cuda')
print("time to load model", time.time() - cur_time)

and

from diffusers import ControlNetModel, DDIMScheduler, StableDiffusionXLControlNetPipeline
from diffusers.models import AutoencoderKL
import torch
import time
cur_time = time.time()
controlnet_names = ["thibaud/controlnet-openpose-sdxl-1.0"]
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, token="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", device_map="balanced")
controlnets = [ControlNetModel.from_pretrained(x, torch_dtype=torch.float16, token="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").to('cuda') for x in controlnet_names]
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnets, torch_dtype=torch.float16, safety_checker=None, vae=vae, token="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"
, device_map="balanced")
print("time to load model", time.time() - cur_time)

In both scripts, the model takes about 35-40s to load.

In the first script, a long amount of time (21s) is spent here https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/modeling_utils.py#L747-L753, for the unet model loading.

In the 2nd script, a long amount of time (23s) is spent here https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pipeline_loading_utils.py#L700, where the load method is ModelMixIn.from_pretrained for a of <class 'diffusers.models.unets.unet_2d_condition.UNet2DConditionModel'> object.

Is there any way to increase this efficiency? Open to caching options too.