If you are using an NVIDIA Card, driver 535.98 is causing massive slowdown. Roll back to 531.79!

[morphinapg]

Many people are reporting issues with Stable Diffusion with the new NVIDIA driver, such as:

https://www.reddit.com/r/StableDiffusion/comments/1458fv4/nvidia_driver_memory_changes/

https://www.reddit.com/r/StableDiffusion/comments/141is23/performance_regression_with_latest_nvidia_drivers/

It turns out, NVIDIA changed the behavior of shared video memory, so that it appears to be swapping to system RAM long before it would ever have an OOM error, causing obvious slow downs. Even on a workflow that only used 9 of my 12 GB of VRAM, I was experiencing significant slowdown. Here's another thread discussing the issue in more detail:

vladmandic/sdnext#1285 (comment)

It turns out even the previous version of the driver had some of this issue, although much less noticeable, but the current driver can cause massive slowdown. For now, I would highly recommend reverting to 531.79. I went from getting about 3 s/it in Dreambooth to about 4.2 it/s, a difference of 12.6x!

[AlexMcCown]

I can confirm this. I had speeds of 4s/it before and 3.2it/s now on a 12gb 3080. I used DDU to uninstall the drivers (did not enter safe mode or restart, only needed to remove the base graphics drivers nothing else) and installed 531.79 from nvidia's website all without restarting and it works fine.
This was driving me nuts, sometimes it would work fine for a few epochs before dropping, sometimes as bad as 22s/it, i played around with different xformer, torch and transformers versions and thought i had some faulty vram.
Unsure if some part of Dreambooth or its requirements need updating or if its a bug on Nvidia's end, however i'll be sticking with 531.79 for now.

[morphinapg]

I would recommend subscribing to the thread I posted above to follow the updates on this:

vladmandic/sdnext#1285

Nvidia is aware of this and plans to fix it in a future update, although the newest driver update does NOT currently fix it