train_text_to_image Poor training generation effect

[wzhiyuan2016]

I use the train_text_to_image.py script to train UNet,Some of my relevant information is as follows:
1.My training script

2.My training data

3.My training loss

4.inference generating codes

5.Generate results after training
prompt_list = ["lakes"]

6.Original["CompVis/stable-diffusion-v1-4"] generated results
prompt_list = ["lakes"]

My reasoning code is fine.

What is the reason why my generation effect has deteriorated? thank you!

[wzhiyuan2016]

@sayakpaul

[wzhiyuan2016]

@sayakpaul
Are you saying that I used these two parameters?
--center_crop --random_flip

I saw it https://github.com/huggingface/diffusers/tree/main/examples/text_to_image
recommend using this parameter below.

So I used these two parameters

I will delete these two parameters and retrain them。

[sayakpaul]

Can you copy-paste the training command you're using?

[wzhiyuan2016]

@sayakpaul
train_text_to_image.sh：
export MODEL_NAME="CompVis/stable-diffusion-v1-4"
export VAE_NAME="madebyollin/sdxl-vae-fp16-fix"
export TRAIN_DIR="./datas17/v1_tmp/"
export OUT_DIR="./train_snapshots/t2i_sd1.4_LHQ_unet_1210_tmp_del_center_crop_random_flip"
log=$1
LD_LIBRARY_PATH=./cudnn_8_7_0/cudnn-linux-x86_64-8.7.0.84_cuda11-archive/lib/
XFORMERS_MORE_DETAILS=1
accelerate launch --main_process_port 29501 --mixed_precision="fp16" --multi_gpu --config_file "default_config_new2.yaml" train_text_to_image.py
--pretrained_model_name_or_path=$MODEL_NAME
--train_data_dir=$TRAIN_DIR
--caption_column="text"
--resolution=1024
--train_batch_size=4
--dataloader_num_workers=8
--max_train_steps=20000
--mixed_precision="fp16"
--validation_epochs=50
--gradient_accumulation_steps=2
--checkpointing_steps=5000
--learning_rate=1e-05
--lr_scheduler=constant
--lr_warmup_steps=10
--seed=42
--use_8bit_adam
--output_dir=$OUT_DIR
--gradient_checkpointing
--resume_from_checkpoint="latest"
--enable_xformers_memory_efficient_attention
--prediction_type="epsilon" 2>&1 | tee $log

default_config_new2.yaml:
compute_environment: LOCAL_MACHINE
debug: false
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: '4,5,6,7'
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 4
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

[sayakpaul]

Have you tried with a lower resolution like 512?

[wzhiyuan2016]

Have you tried with a lower resolution like 512?

no, I'll try it out later. Set the resolution to 512 for training ,What are the points you are considering?
Training stability?-----> Lower resolution may make the training process more stable and reduce the risk of overfitting, as fewer details mean that the model needs to learn less complexity.