Can't change device in pipeline or controlnet

[CaledoniaProject]

I'm running diffusers on mac m3, but this does not allow me to change the device to mps,

import sys
import torch
from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel

from controlnet_aux import OpenposeDetector
from diffusers.utils import load_image

device = 'mps' if sys.platform == 'darwin' else 'cuda'
torchDtype = torch.float32 if sys.platform == 'darwin' else torch.float16

controlnet = ControlNetModel.from_pretrained(
    "thibaud/controlnet-openpose-sdxl-1.0", 
    torch_dtype=torchDtype
).to(device)

pipe = StableDiffusionXLControlNetPipeline.from_single_file(
    "./sdXL_v10VAEFix.safetensors", 
    controlnet=controlnet, 
    torch_dtype=torchDtype
).to(device)

It still complains about the device,

Traceback (most recent call last):
  File "./main.py", line 41, in <module>
    images = pipe(
  File "./venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "./venv/lib/python3.10/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py", line 1288, in __call__
    ) = self.encode_prompt(
  File "./venv/lib/python3.10/site-packages/diffusers/pipelines/controlnet/pipeline_controlnet_sd_xl.py", line 418, in encode_prompt
    prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=True)
  File "./venv/lib/python3.10/site-packages/torch/cuda/__init__.py", line 305, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

What's wrong?

[CaledoniaProject]

Hi @asomoza, can you also take a look at this issue? Thanks!

[asomoza]

Hi, sadly I don't have a mac to test this, I'm guessing this has something to do with your environment.

ccing @pcuenca

[CaledoniaProject]

Hi @pcuenca Any ideas on how to debug this issue? Thanks!

[CaledoniaProject]

Tried to debug it, although self.device is set in constructor, but self.pipe._execution_device remains cuda. I tried to do a dirty hack

1. controlnet.pipe._execution_device = device, it won't allow me to set that attribute.
2. controlnet.pipe.to(device), no effect