comfyui --benchmark option

[alexheretic]

Feature Idea

During the discussion of #10302 it would be useful if we have a standard way to benchmark performance of various usages.

Could include:

• vae encode/decode speed (for popular base model(s))
• ksampler speed (for popular base model(s))
• Upscale by model speed

Could start with sdxl & wan, Ultrasharp upscaling.

Perhaps run comfy with --benchmark [scenarios] and it runs the scenarios (if the models are available on the filesystem) outputting a text report.

Would be a useful tool for just trying all the different options available locally to see what performs best in addition to comparing perf across platforms & setups.

Existing Solutions

Adhoc testing by running workflows and checking the speed

Other

No response

[sfinktah]

Not to rain on your parade, but this is something I've been thinking about for a very long time, and it's too ill-defined a request (have you thought about how you're going to automate the workflow entirely from the command line?) and besides which, would be too much work for the developers), and hence: the reason I developed my timing node.
https://github.com/sfinktah/comfy-ovum#timer which can export its results via an API, and logs the startup options for ComfyUI.

Add the Get Environment Var node from the same node-pack and concatenate that info into the notes field of the Timer node, and you're good to go. You can read environment variables, convert them to booleans, and use any of the commonly used logic nodes to switch what it is you want to test.

If you want to switch something with a combo box there's even a widget for manipulating those.

The only I don't have in my kit is a node to auto-start a workflow, but that would be a no-brainer. I believe there are already nodes to exit comfy after a workflow is completed -- though you'd need to do at least 2 runs, as there may be compiling in the first run.

As long as you kept a browser open and didn't auto-launch a new one each time, you could then run comfy through a batch of tests and grab all the results from you web-browser once you were done. (Yes, it would be nicer if I added a JSON output to my timing node, I'll put that on the list).

[alexheretic]

It's nice but I'm not sure it serves the same purpose. I can already time an arbitrary workflow and read the node execution time from stdout. This idea is to have the lowest barrier of entry for others to run a standard bench on their setup to report perf differences in a fair way.

So:

• Everyone who runs the benchmark is comparing the same operation(s).
• Easy to run and paste output, e.g. to justify a PR changing default settings.

have you thought about how you're going to automate the workflow entirely from the command line

When --benchmark arg is selected the nodes or underlying fns, like tiled_scale, would be executed programatically on standard reference images, run n times then output the score.

I guess an alternative would be to have some standard workflows to benchmark and use nodes like yours to help time them. But don't you need to restart comfy a lot anyway, since pytorch & others will cache env var config on first read?

I guess my instinct is to have a CLI for this as it just seems simpler to use to me.

[sfinktah]

@alexheretic You had me at "for other to run a standard bench"... and again at "justify a PR". Those are very worthy goals.

As for CLI -- when I was growing up modems were our bottled milk. Absolutely the preferred option. (Unless you're going to be using runcomfy.com or something).

Restarting a comfy a lot? Absolutely -- if you want to test different things, you need to restart. Though I've found if you want really consistant results, leave something running overnight endlessly repeating. How much of that is removing user activity, and how much is memory/pagefiles/etc I couldn't say. I'm on Windows and I have to share VRAM, which makes that really problematic.

A few questions. How do you get timestamps in your ComfyUI logs? And can you get timestamps for the compilation time (torch.compile)?

Now I have a mess of python files I might be able to coalesce into a usable github repo that makes really nice graphs showing VRAM/RAM/Shared VRAM, and GPU/POWER usage (even temperature). Either as singular graphs, or as n overlaid comparison graphs. It has proven very useful in comparing different version/methodologies/workflows, but I do not know of any way of obtaining the required information on Linux. i.e., how can you obtain VRAM, GPU usage (actually I prefer power usage as a metric).

If I could get that information from the GPU, and also more specific information from Comfy (e.g., node_id, start_time) or just python hooks into the standard events that are send via websockets (which cover all that), then it should be possible to build a benchmarking tool.

[alexheretic]

How do you get timestamps in your ComfyUI logs? And can you get timestamps for the compilation time (torch.compile)?

Right, what I said earlier about this was wrong. I just realised today that during my manual tests I'd been reading stdout logs provided by a custom node ComfyUI-Dev-Utils. It had been so long since I added this that at some point I started thinking this was default behaviour 😅 .

So agree with you that for manual testing your custom timing node (or equivalent as I had installed and forgot about) is indeed necessary.

I still think this is a separate thing to a standard CLI benchmark option proposed here. I can understand if the maintainers don't want to maintain benchmarks functionality though and instead prefer users to figure this out for themselves. This is why I raise this as an issue and not dive into another potentially doomed PR.