Is anybody successfully training 2.1 768 based models on a 12GB card?

[morphinapg]

I can get training working with Lora although I haven't had great results with the different settings I've tried. Ideally I'd prefer to train a full model properly, but I've tried turning on every memory saving feature I can, including disabling text encoder training and even disabling previews, and it still runs out of memory after about 26 steps no matter what I do.

Perhaps I'm missing something. Anybody have advice for training a 2.1 768 based model on a 12GB card, or perhaps have recommendations for Lora settings? Since I wanted to get as close to training a full model as possible, I increased my Lora ranks to max, and saving it as a checkpoint when training is done, and while there is some resemblance to my samples, the results I'm getting here overall are still nowhere near as good as what I get when I train a 1.5 based model without Lora. I've tried higher learning rates and lower learning rates. The default Lora rates seemed to overtrain very quickly, even with a polynomial schedule with a high power, while the lower rates (closer to those I use without Lora) worked to a point, but didn't really refine smaller details very well. Maybe something in between would work better but it seems like it would require a lot of trial and error to figure that out, so recommendations would be good.

Ideally I'd prefer to be able to train without Lora if it's possible, but maybe 12GB just isn't enough for that.

[Renaldas111]

Yes, but only of full fp16 training in kohya_ss implementation of Dreambooth. But I dont know how much image quality is worse comparing to mixed precision training.
Also, you may try to plug your monitor to secondary GPU, maybe you have internal GPU, and get some VRAM for a discrete GPU. I had 2 GPUs in my system, and I had my monitor plugged into the secondary one, it looks then the main one receives a little bit of additional free VRAM.
Now I have 3 GPUs, and my main training GPU (RTX3060 with 12G VRAM) is not connected to a display. Despite this, I cant train 768 model via this repo even at the least VRAM consuming settings. But, as I wrote, I easily done this in kohya_ss trainer, and the reported VRAM usage was far from 12G.

[morphinapg]

Perhaps I can try doing it in kohya. Do you have suggestions for what settings to use there other than fp16?

Unfortunately I don't have a second GPU, and my PCIe slots are all used by other devices too.

[Renaldas111]

Ok, just tried, and looks the training started on RTX3060-12G. But as I mentioned, this GPU is not connected to a monitor as my system has several GPUs. These are the settings I tried:

Preparing Dataset (With Caching)
Bucket 0 (768, 768, 0) - Instance Images: 40 | Class Images: 0 | Max Examples/batch: 40
Caching latents...: 100%|██████████████████████████████████████████████████████████████| 40/40 [00:09<00:00, 5.03it/s]Saving cache!
Total Buckets 1 - Instance Images: 40 | Class Images: 0 | Max Examples/batch: 40
Caching latents...: 100%|██████████████████████████████████████████████████████████████| 40/40 [00:09<00:00, 5.03it/s]
Total images / batch: 40, total examples: 40
Caching latents...: 100%|██████████████████████████████████████████████████████████████| 40/40 [00:09<00:00, 4.44it/s]
Total dataset length (steps): 40
Initializing bucket counter!
***** Running training *****
Num batches each epoch = 40
Num Epochs = 200
Batch Size Per Device = 1
Gradient Accumulation steps = 1
Total train batch size (w. parallel, distributed & accumulation) = 1
Text Encoder Epochs: 0
Total optimization steps = 8000
Total training steps = 8000
Resuming from checkpoint: False
First resume epoch: 0
First resume step: 0
Lora: False, Optimizer: 8bit AdamW, Prec: fp16
Gradient Checkpointing: True
EMA: False
UNET: True
Freeze CLIP Normalization Layers: False
LR: 2e-06
V2: True
Steps: 9%|▌ | 695/8000 [08:55<1:35:14, 1.28it/s, inst_loss=0.251, loss=0.251, lr=2e-6, prior_loss=0, vram=10.2]

The usage is 11,735Mb, looks I have still a little bit of reserve:

C:\Users\user>nvidia-smi
Tue Apr 25 21:16:05 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 531.18 Driver Version: 531.18 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 3060 WDDM | 00000000:01:00.0 Off | N/A |
| 96% 88C P2 151W / 170W| 11735MiB / 12288MiB | 99% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce RTX 2060 S... WDDM | 00000000:02:00.0 On | N/A |
| 29% 45C P0 40W / 175W| 1511MiB / 8192MiB | 1% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce RTX 2080 Ti WDDM | 00000000:08:00.0 Off | N/A |
| 27% 30C P8 20W / 250W| 3758MiB / 11264MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

[Renaldas111]

BTW, the training is extremely long, not only timewise, but also if we count epochs. I train a cartoon character, and at 200 epochs it is still a total mess, at about 500 it is still quite different from original, but decent. So, it looks it will take up to 1000 epochs to train, what is hours of GPU time.

[morphinapg]

Well hmm, that may not be useful to me then. My datasets are hundreds of images, and I already train for something like 12 hours on SD1.5, typically doing 100 epochs or slightly less, although I use a polynomial learning rate with 5 epochs of warm up and an exponent of 6. The idea is to refine the training over time to smaller and smaller details. I usually get better results with that than a constant learning rate.

It sounds to me like your results are similar to the bad results I was getting when I tried either training 2.1 with Lora or with no text encoder training. I got results that looked a little bit closer to what I was looking for, but were not good quality at all. Perhaps more steps would solve that, but it was already taking about 24 hours for Dreambooth without text encoder training and no previews, for just those same ~100 or less epochs that I use on 1.5. If I'm going to need 10x that, it's just not useful to me.

I did have some limited success training a higher resolution model based on 1.5, for only a slight increase in training time. It works better at rendering higher resolutions than the base model did, but it's still got plenty of issues that 2.1 doesn't have. Of course, this only works if you have a large and varied dataset like I do. Too small of a dataset or not enough variety and it's not going to be able to learn how to handle image composition at that higher resolution from your images alone.