How can I connect Krita to the ComfyUI backend that powers SwarmUI?

[Taruvi]

I have a Docker container which allows me to switch between several image diffusion frontends. The container also symlinks just about everything so almost all models can be shared without duplication. Two of these frontends are ComfyUI (standalone) and SwarmUI (which obviously includes ComfyUI as its backend). I then connect to this Docker container via another, separate computer.

I have Krita installed on this separate computer, along with the awesome plugin by Acly.

I use a display tablet which has obvious benefits whereby I can draw something I want, have it rendered by some model, then I can take a 'final' image, switch to SwarmUI and use IMG2IMG with any number of other different models and settings, along with SwarmUI's easier to use interface. At least, this part is easier for me in SwarmUI than Krita. It's a bit fiddly to do this kind of thing in Krita.

However, while I can connect the Krita plugin to the standalone instance of ComfyUI, I cannot figure out a way to connect it to the instance of ComfyUI that powers SwarmUI.

Connecting to the standalone ComfyUI is as simple as inputting the correct IP and port and it just works.

Attempting to connect to the ComfyUI instance behind SwarmUI appears to not be possible. Even if I input the correct IP address and port (for SwarmUI's ComfyUI instance - not SwarmUI itself) it refuses to connect.

SwarmUI log for 'ComfyUI-0':

2025-03-22 12:13:59.280 [ComfyUI-0] [STDERR] To see the GUI go to: http://127.0.0.1:7821

I substitute the IP address for the actual IP on the network, since I'm connecting from a difference computer, but I get this in Krita:

Krita's log simply says 'INFO Connecting to' IP:port. There is no error message and it sits there forever with no further log entries.

Searching in this repo gives no results for the search term 'krita'.

I'm surprised no-one else has encountered this to write something about it. No-one else has this or a similar workflow with Krita involved?

The current way I have to do things is to start my Docker container with just the standalone ComfyUI, do things in Krita, then shutdown the container, changing it to run SwarmUI and then start the container again. It takes a few minutes to run through all of this. But if I realise I would like to make even a small edit to the image in Krita, I have to reverse that whole process, make the edit, then reverse the whole process again to run IMG2IMG in SwarmUI.

I'll be honest: This is perfectly doable this way. It's simply time-consuming. And if I want to make any edits, or even start a new composition, I need to run through this whole process repeatedly. So it goes from being time-consuming to VERY time consuming. For example, in Krita I make heavy use of regions. Krita with the plugin mentioned above is amazingly powerful for regional prompting. SwarmUI is amazingly powerful for all sorts of reasons of its own. But when I switch everything to SwarmUI, sometimes I need access to the actual prompts I typed for different regions which is inaccessible in Krita if it's not connected to a ComfyUI backend. These prompts do not get saved in metadata, either. There's a lot of manual work to copy/paste prompts, should I need them. Not to mention I have to remember to do that in the first place. Because instead of having to constantly switch the Docker container back and forth, I try to do batches in Krita, then run that whole batch through SwarmUI. Things can get easily muddled.

I suppose what I'm asking, really, is how to expose the port for the ComfyUI backend when I'm running SwarmUI?

My dream scenario is to be able to simply ALT+TAB between Krita and SwarmUI and have it all just be working all at once.

[mcmonkey4eva]

It should be something like http://localhost:7801/ComfyBackendDirect/

The ComfyBackendDirect route, as the name implies, forwards requests directly to the first available comfy backend

[Taruvi]

The name of it makes sense but it's not the most obvious unless you know what you're looking for.

If anyone else is looking for help with this in the future, direct the Krita AI plugin to:

ipaddress:swarmuiport/ComfyBackendDirect

Where I went wrong was in trying to connect the plugin directly to ComfyUI, whereas it needs connecting directly to SwarmUI, which will handle redirecting from there.

And after installing/moving around a bunch of requirements:

Thanks for the help!