Gray Artifacts Spreading During Fine-tuning SD 1.5 Inpainting for Object Removal

[BlueBabyYoda46]

Issue Description

Problem Manifestation

Im trying to train stable diffusion 1.5 inpaint and the model is experiencing a progressive degradation in output quality, specifically:

1. Initial Performance (0-10k iterations):

   • Model showed good learning progress
   • Most inpainting results were acceptable
   • Only 2 validation images showed gray artifacts in inpainted regions

2. Degradation Pattern (after 10k iterations):

   • The gray artifact problem began spreading to more samples
   • As training progressed, more images started showing similar gray patches
   • The issue seems to be contagious across the dataset

Training Setup

• Using Stable Diffusion 1.5 Inpainting as the base model
• Task: Object removal from images using inpainting

Dataset & Training Config

• Dataset Size: 5000 images
• Current Training Progress: 270k/500k iterations
• Learning Rate: 1e-5
• Prompt Strategy: Using "empty scene" as the conditioning prompt
• Loss Function: Standard SD inpainting loss (MSE on noise prediction)

Examples

a) ground truth image // b) mask // c) predicted image // d) training output

1. Successful early results

2. Unsuccessful early results and Initial gray artifacts in validation

3. Current state showing spread of the issue

Questions

1. Has anyone encountered a similar gray artifact during fine-tuning SD 1.5 inpainting?
2. Are there known solutions or preventive measures for this type of training instability?
3. Could this be related to:
   • The learning rate being too high (1e-5)?
   • The simplistic prompt ("empty scene")?
   • The long training schedule (500k iterations)?
   • The loss function not enforcing enough constraints?

Additional Context

• The model starts with good performance and learns effectively
  (inference on test data 75K iterations vs 270k iterations)
  

• The training loss is decreasing as expected
  

• The issue seems to be a form of mode collapse or error propagation

Any insights or suggestions would be greatly appreciated!

Tags

#stable-diffusion #inpainting #fine-tuning #training-stability #mode-collapse

[BlueBabyYoda46]

@everyone please help me :(

[xduzhangjiayu]

Have you solve it？I have the same issue

[a-r-r-o-w]

The scope of this question is too wide for us to be able to address/help in any way. I'll try my best to provide some thoughts, but am afraid that we cannot really help with these kinds of things because it is too broad an ask and not really Diffusers-specific.

1. Is the script you are using present in the Diffusers repo? If not, we will be unable to help. If yes, have you modified it in any way? Did it work before modifications?

2. Are you using the 4-channel latent without any additional model weights for model inpainting? I assume yes because you mention the model performs well in the first few thousand steps. In my experience, 10k steps is a bit too low for seeing any good convergence when adding additional model weights initialized randomly (which would be the case if you're training, say, a 9-channel inpainting model with new projection layers).

3. Assuming standard 4-channel latent without additional random model weights, could the initial "good" performance be because SD1.5 already works decently for inpainting? As training progresses, the model just gets worse due to bugs in training script?

4. What happens if you try to overfit the inpainting model on a single image-mask pair for 1000 steps (validation every 200 steps - does model become worse or better)? If it works correctly, could you try single image-multiple mask pairs with the same training setting? If either doesn't work, it's most likely a bug in training script.