Training Collapses with train_text_to_image.py

[nighting0le01]

If i use a larger (400,000k image text pairs) dataset and using the training script provided. I see weird collapse on all prompts. I haven't even finished 1 epoch and trian with 1e-5 lr . i start with something like this from pretrained SD1.5

but it transitions to

after less than 200 steps at batch size 64
(essentially not even seeing the full data!)
Is there any bug in the train_text_to_image.py?
@sayakpaul have you seen something like this a collapse like this before?
training on a small subset of around 200 images does not lead to collapse but very blurry results

[bghira]

it just looks to me like a very high learning rate for a batch size of 64! are you using lr_scale option? have you tried lower value like 1e-6 or 4e-7 ?