How does ControlNet understands the relationship between the layout map and the descriptive text.?

[ShanZard]

I have a question. When training ControlNet, the first image's semantic segmentation mask values (0, 1, 2) represent the background, the aircraft, and the train, respectively, and the corresponding text also describes these goals. In the next image, the semantic segmentation mask value represents other objects. Is it OK to do so? Or I need some other values to represent new objects. It would be great if someone had done similar experiments

Furthermore, this raises a question about how ControlNet understands the relationship between the layout map and the descriptive text. If the above is possible, then the layout map doesn't really need to represent any semantics, just the spatial layout. On the other hand, if this is not possible, it means that the layout map is better to provide semantic information as well.

[asomoza]

HI, I a semantic segmentation controlnet should be trained the same as any other controlnet, I don't really understand what you're saying with the mask values, these values are not relevant for the controlnet, just for the semantic segmentation model, you just need to match the same classes with both.

The controlnet is just trained with a text and an image pair, the original image, the condition image and a text to describe the original image, the model will learn everything else by itself. For example for an open pose controlnet, you don't tell it what part(color) of the skeleton is an arm, a leg or the head, you just describe the image and the model will learn the pose.

Sadly I haven't trained a controlnet yet (I really want to so I have more insights) but you can learn how to train it from the datasets available

When I do a training myself I will have more information, but I think the best way to train it is like I said, use something like the ade20k dataset and describe the image making sure it contains all the classes that are mapped.

Just be advised that to train a controlnet you need a really big dataset, 5k or 10k won't do it, the best controlnet that works for me, was trained with over 500k images.

[ShanZard]

Thank you for your reply! The core issue I want to discuss is the setting of this condition image. Take open pose as an example. If I use yellow to represent the arm in the first image, can I use yellow to represent the leg in the second image? Is this kind of training feasible? If it is a semantic mask, does yellow represent an airplane in the first image, and can yellow represent a ship in the second image? Throughout the above process, the text has always accurately described the original image.

Alternatively, if I want to concatenate a new dataset with ade20k, do I need to find semantic mask values that are the same but represent different objects in two datasets, and replace them with new values?

In fact, this is a question of whether the control image requires some semantic information. What do you think of it？

[asomoza]

If I use yellow to represent the arm in the first image, can I use yellow to represent the leg in the second image?

No, if you do that the model will not learn and won't know what to do with the yellow, they have to be coherent in the whole dataset and training, same as when you train a lora, you can't describe a dog as a dog in some images and a cat as a dog in the other images, the training won't work.

If it is a semantic mask, does yellow represent an airplane in the first image, and can yellow represent a ship in the second image? Throughout the above process, the text has always accurately described the original image.

same as before, you have to always make sure the colors match, if you don't want semantics, you can just train with something like normals, depth maps, or you can even invent your own conditions, but again, you will need a huge dataset.

Alternatively, if I want to concatenate a new dataset with ade20k, do I need to find semantic mask values that are the same but represent different objects in two datasets, and replace them with new values?

if I understand what you're saying, yes, you will need to make them match, the mapped classes (spatial and color) need to match, you can't train a semantic segmentation controlnet without this, that's why you usually use the same model to do the semantic segmentation of the images or use datasets that have this.

If you have additional classes that aren't in the original dataset, your best option is to use a model trained with that dataset and finetune it to your own dataset, adding those classes, so you can use it to segment your images.

In fact, this is a question of whether the control image requires some semantic information. What do you think of it？

that a lot easier to understand, the answer is yes but I also think there's a chance that if you use a really huge dataset, that the model will discard the color information and will start learning the shapes (this is a big maybe).

edit: changed my last answer after thinking about it more

[ShanZard]

Thank you very much for your answer! This has provided me with a clear direction for my subsequent experiments. Of course, I will try to build a really huge dataset in the future to directly learn shapes.