Is it possible to add vocabulary to a clip model packaged inside of a distributed SD checkpoint?

[TeKett]

Is it possible to add vocabulary to a clip model packaged inside of a distributed SD checkpoint? I have a checkpoint of the .safetensor variety that i downloaded on civitai, that i want to train/finetune and make loras for. But i want to add new vocabulary to it in order to improve training, and not have the tokens split into other words, and damage things it already knows. To make it easy for me i wanted to add the booru tags of my training set since i can just grab them when i get the image automatically.

[asomoza]

I don't fully understand what you mean with not have the tokens split into other words, and damage things it already knows, do you mean catastrophic forgetting? That's what LoRAs are for though but common (wrong) practices where people train LoRA's with a rank of 64, 128 or even 256 makes the model change more of what it already knows too.

Booru tags most of the time are common words so it would be impossible that any training doesn't change what the model already knows, for example the tag smile already has 1 token associated to it, so if you train a model with anime images with that tag it would start associating it to the anime style more often, that's why people sometimes train dreambooth models with rarely used words like sks.

Still what you're probably looking for is pivotal tuning where you can also train textual inversion with the LoRA, you can test it here in diffusers:

https://github.com/huggingface/diffusers/tree/main/examples/advanced_diffusion_training

[TeKett]

Well you are, hence why i said it. Since as a person with adhd, i know its possible to forget that ppl dont know things you do.

Unintuitive, so, what i want is called from_single_file. I cant see most of the documentaion or most functions with autocomplete in IDE, and half the time huggingface documentation search doesnt work. I cant tell yet if the code does what i want, i got it mostly from the transformers ppl, but it errors out at line 18. cannot assign 'torch.FloatTensor' as child module 'token_embedding'

from diffusers import StableDiffusionPipeline

array = []
with open("D:/tagstest.txt",encoding="utf8") as file:
        array = [row.rstrip("\n") for row in file.readlines()]

pipeline = StableDiffusionPipeline.from_single_file("C:/Train/checkpoint.safetensors")

num_new_tokens = pipeline.tokenizer.add_tokens(array, special_tokens=False)

# simple resize (is this correct?)
pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer) + num_new_tokens)

# overwrite the content to have better results
input_embeddings = pipeline.text_encoder.get_input_embeddings().weight.data
input_embeddings_avg = input_embeddings[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings[-num_new_tokens:] = input_embeddings_avg
pipeline.text_encoder.set_input_embeddings(input_embeddings) # Error
pipeline.model.save_pretrained("c:/test")

[TeKett]

And how can i save it as a checkpoint? Since i assume that pipeline.model.save_pretrained() saves it as a diffusers model, and not as a single file.

[sayakpaul]

You will have to run your own conversion for getting the checkpoint to a single file-format.

[TeKett]

So, i managed to fix the erroring so i think its working, no thanks to the people here. Now i just have to check if its working, and or if i have to train something.

from diffusers import StableDiffusionPipeline

array = []
with open("D:/tagstest.txt",encoding="utf8") as file:
        array = [row.rstrip("\n") for row in file.readlines()]

pipeline = StableDiffusionPipeline.from_single_file("C:/Train/ricecakeRemix.safetensors")

num_new_tokens = pipeline.tokenizer.add_tokens(array, special_tokens=False)

# simple resize
pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))

# overwrite the content to have better results
input_embeddings = pipeline.text_encoder.get_input_embeddings()
input_embeddings_avg = input_embeddings.weight.data[:-num_new_tokens].mean(dim=0, keepdim=True)
input_embeddings.weight.data[-num_new_tokens:] = input_embeddings_avg
pipeline.text_encoder.set_input_embeddings(input_embeddings)
pipeline.save_pretrained("c:/test")

[TeKett]

Well, when i try to load whatever i got

Diffusers is giving me

RuntimeError: Error(s) in loading state_dict for CLIPTextModel:
	size mismatch for text_model.embeddings.token_embedding.weight: copying a param with shape torch.Size([90323, 768]) from checkpoint, the shape in current model is torch.Size([49408, 768]).

And A1111 is giving me

RuntimeError: Error(s) in loading state_dict for LatentDiffusion:
        size mismatch for cond_stage_model.transformer.text_model.embeddings.token_embedding.weight: copying a param with shape torch.Size([90323, 768]) from checkpoint, the shape in current model is torch.Size([49408, 768]).