Advice on Flux LoRA training

[Clement-Lelievre]

Hi all,

I'm looking for advice on how to best train a LoRA on the following task: the LoRA should learn to represent black and white wireframe drawings seen from top (orthogonal view ie objects seen from top, no perspective).
I attach one example of image from my dataset depicting a desk in the above mentioned style.

So far I've mostly tried fine-tuning flux dev, varying lr, rank etc. to see what I can get; and the LoRA partially learns but definitely isn't great.

Do you have any recommendations of model/hyperparams? (if this task is feasible at all)

Thanks vm!
Happy to provide more details if needed

(maybe @asomoza ?)

[asomoza]

can you post an image generated with your lora?

It's probably doable with any model but it depends on your definition of 'isn't great'. For example, there's a lora for ikea instructions trained in the base SDXL model which is fine if you don't consider the text.

About your decision to use Flux, I haven't tried to train a lora on it yet but for what I've seen and tested, Flux looses a lot of the quality when you finetune it, the loras I tested work fine but they degrade a lot the other stuff and also the model isn't that great for lines (for example anime) and it's really overtrained in professional photos with bokeh.

For example, take this lora for SDXL and this one for Flux which are for 3D wireframes, if you ask me, based in the demo images, I think the SDXL one is better.

Also how big if your dataset? Depending on the size, you will probably need to finetune a model that's really over trained in lines like the anime models. This probably needs a bigger dataset than a normal character lora.

I really can't help with the hyperparameters for Flux, but one good start would be to train the whole model and then extract loras in different ranks to test which one works better for your use case. Also as a good default start, you should use the same LR than what the base model used.

[asomoza]

that's how training is, it's not an exact science and it involves a lot of trial and error.

I like to train more on epochs than steps, but I've got a lot of cases where an earlier epoch was better than most of the later ones until there's another one in the later epochs that also works, to me, it seems your lora it's still under trained even if you think the 700 step one works better. There also some other strategies you can apply, like lowering the LR after the 700 steps mark.

Also it seems you are underestimating training and overestimating the model capabilities, as I told you before, Flux degrades quite a lot with finetuning, so you'll need to mitigate this with something, in your case, since you want to preserve orthographic views or for it to preserve the angles and camera views, you should also have those in your dataset, specially if you're using dreambooth.

For example, since you're using a token I guess you're using dreambooth, and if most of your dataset has top views, the model will learn that your token means top view, it doesn't matter if it understands other views, just having a couple of images with orthographic views won't mitigate this.

[Clement-Lelievre]

yes I'm using Dreambooth and LoRA

it seems your lora it's still under trained even if you think the 700 step one works better

at least on the intermediate inferences made with the checkpoints every 100 steps (up to 6000 steps!), like I said the 700 steps is by far the best-performing but yeah there may very well be a superior set of params

There also some other strategies you can apply, like lowering the LR after the 700 steps mark

yes will try this thks

since you want to preserve orthographic views or for it to preserve the angles and camera views, you should also have those in your dataset, specially if you're using dreambooth.

All of my dataset images are seen from top with orthographic view, in exactly the same style as the initial image I shared

[asomoza]

I misunderstood what you said, I read it again and you were referring to the base training, don't really know about Flux but SDXL can do top down views without a problem if you fine tune it for it.

I did a quick search for a lora and I could generate these images really easy:

Didn't see anything similar in Flux, just some people that trained some top down map generators.

What you're trying to do shouldn't be that hard, you're just training with grayscale images and with straight lines, a lot easier than some of the other finetuning's like those weird styles like yarn or spaghetti, also I saw someone that trained a lora with dotted lines for drawing which seems a lot more complex to train than yours.

My guess is that since you're fixated in using Flux, you will need to teach it everything, first the top view, then the line drawings and then the conversion of the objects between for example, a front view and a top view.

Probably you won't be able to do this with dreambooth and will need to train a full lora or model for this, also take into account, that people mostly finetune loras with Flux for portrait photos of a person with a very simple pose, that is probably for a reason.

[Clement-Lelievre]

thanks for the reply. haha don't worry I'm not fixated with using flux, if another model is better suited to the task then I'll go for it.
So you're saying you found an existing SDXL LoRA that generated the top view images you shared?
I will look into dreambooth sdxl lora training

[asomoza]

yeah, this one but as I told you, it's for a model that's not that safe to use but that is only trained with lines.

I used to never look at those models but I read and then tested (and corroborted) that they're really good for a lot of other stuff because of the same training. The problem is that even if you don't prompt it, you can sometimes get some nsfw results, so if that bothers you, you will need to train or find a model trained with lines that is sfw but because of the same, they lack the understanding of these newer models.

Same problem as base models with anatomy, SDXL, SD3, an all the newer models are really bad with anatomy because they weren't trained with "good anatomy" to avoid the backslash and to be stigmatized as nsfw.