New model: Anima

[dxqb]

Test in preview branch: https://github.com/Nerogar/OneTrainer/tree/preview

Includes:

• Upgrade torch #1392 — Upgrade torch - needed because of Anima uses the Qwen VAE: [Bug]: Qwen Image sampling VRAM regression #1371
• Upgrade transformers to 5.x and other dependencies #1285 — Upgrade transformers to 5.x: needed by diffusers now

[AmaelG]

Unsure if this is in scope for this PR, but I think it would be useful to expose an optional toggle to train Anima's llm_adapter.
From my testing with multi-concept training, training the llm adapter seems to improve concept adherence, converge faster, and reach lower loss/val.
I know tdrussel recommends to avoid training it, but in my experiments I have not seen obvious degradation of general knowledge, while the trained concepts became more reliable.

[dxqb]

Unsure if this is in scope for this PR, but I think it would be useful to expose an optional toggle to train Anima's llm_adapter. From my testing with multi-concept training, training the llm adapter seems to improve concept adherence, converge faster, and reach lower loss/val. I know tdrussel recommends to avoid training it, but in my experiments I have not seen obvious degradation of general knowledge, while the trained concepts became more reliable.

training text components should be a thing of the past. it's always been a crutch for diffusion models that weren't very capable yet. So I'm hesitant to reintroduce this, with all the problems that come with it (such as having multiple learning rates and many more failure modes).
If there is strong community support that this is needed, maybe, but if even the model's creator advises against it...

[dxqb]

torch._dynamo.exc.FailOnRecompileLimitHit: Hard failure due to fullgraph=True

fixed by #1495, merged into this PR

[FuouM]

Thank you for your great work!

I've been training Anima LoRAs in OneTrainer and using the same dataset/settings in Kohya sd-scripts. Training itself works fine, but LoRAs saved from OneTrainer (preview branch) don't load correctly in ComfyUI when paired with the standard single-checkpoint Anima model (anima-base-v1.0.safetensors). I believe this is due to the LoRA key naming on export.

Example key:

• sd-scripts: lora_unet_blocks_0_self_attn_q_proj.lora_down.weight
• OneTrainer: transformer.transformer_blocks.0.attn1.to_q.lora_down.weight

Other model types in OneTrainer already handle this via convert_*_lora.py key sets (e.g. Flux, HiDream, SD3). Anima's AnimaLoRASaver and AnimaLoRALoader both return None from _get_convert_key_sets(), so no conversion runs on save or load.

Diffusers has the inverse mapping in _convert_non_diffusers_anima_lora_to_diffusers() (lora_conversion_utils.py), which lines up with the rename table already documented in AnimaModel.py (diffusers_to_original()).

I prototyped the conversion script as below. It might be missing things as I haven't tested exhaustively yet:

# convert_anima_lora.py
from modules.util.convert.lora.convert_lora_util import LoraConversionKeySet


def __map_anima_blocks(parent: LoraConversionKeySet) -> list[LoraConversionKeySet]:
    return [LoraConversionKeySet(
        omi_prefix=f"blocks.{i}",
        diffusers_prefix=f"transformer_blocks.{i}",
        legacy_diffusers_prefix=f"blocks_{i}",
        parent=parent,
        next_omi_prefix=f"blocks.{i + 1}",
        next_diffusers_prefix=f"transformer_blocks.{i + 1}",
    ) for i in range(100)]


def __map_transformer_block(key_prefix: LoraConversionKeySet) -> list[LoraConversionKeySet]:
    mappings = [
        ("self_attn.q_proj", "attn1.to_q", "self_attn_q_proj"),
        ("self_attn.k_proj", "attn1.to_k", "self_attn_k_proj"),
        ("self_attn.v_proj", "attn1.to_v", "self_attn_v_proj"),
        ("self_attn.output_proj", "attn1.to_out.0", "self_attn_output_proj"),
        ("cross_attn.q_proj", "attn2.to_q", "cross_attn_q_proj"),
        ("cross_attn.k_proj", "attn2.to_k", "cross_attn_k_proj"),
        ("cross_attn.v_proj", "attn2.to_v", "cross_attn_v_proj"),
        ("cross_attn.output_proj", "attn2.to_out.0", "cross_attn_output_proj"),
        ("mlp.layer1", "ff.net.0.proj", "mlp_layer1"),
        ("mlp.layer2", "ff.net.2", "mlp_layer2"),
    ]

    return [
        LoraConversionKeySet(omi, diffusers, legacy_diffusers_prefix=legacy, parent=key_prefix)
        for omi, diffusers, legacy in mappings
    ]


def convert_anima_lora_key_sets() -> list[LoraConversionKeySet]:
    keys = []

    transformer = LoraConversionKeySet(
        "lora_unet",
        "transformer",
        legacy_diffusers_prefix="lora_unet",
    )

    for block_prefix in __map_anima_blocks(transformer):
        keys += __map_transformer_block(block_prefix)

    return keys

After converting, the output in ComfyUI seems to be affected by the LoRA as expected.

[dxqb]

I've been training Anima LoRAs in OneTrainer and using the same dataset/settings in Kohya sd-scripts. Training itself works fine, but LoRAs saved from OneTrainer (preview branch) don't load correctly in ComfyUI when paired with the standard single-checkpoint Anima model (anima-base-v1.0.safetensors). I believe this is due to the LoRA key naming on export.

Comfy-Org/ComfyUI#14182

[Silvicultor]

I've been training Anima LoRAs in OneTrainer and using the same dataset/settings in Kohya sd-scripts. Training itself works fine, but LoRAs saved from OneTrainer (preview branch) don't load correctly in ComfyUI when paired with the standard single-checkpoint Anima model (anima-base-v1.0.safetensors). I believe this is due to the LoRA key naming on export.

Comfy-Org/ComfyUI#14182

Doesn't look like Comfyanon wants to merge this one and also keep in mind that other inference tools (at least the ones that aren't built upon Diffusers) would also have to make the same change to their code. So I say OneTrainer should include the above proposed conversion logic into it's code and settle for the de-facto standard already established. I know OT wants to use Diffusers keys whenever possible for consistency, and that's perfectly fine for all the models like Flux or Qwen, their original repos being Diffusers format, but this is a special case. Initial Anima release wasn't in Diffusers format, so it's hard to argue for the Diffusers keys.

[dxqb]

I've been training Anima LoRAs in OneTrainer and using the same dataset/settings in Kohya sd-scripts. Training itself works fine, but LoRAs saved from OneTrainer (preview branch) don't load correctly in ComfyUI when paired with the standard single-checkpoint Anima model (anima-base-v1.0.safetensors). I believe this is due to the LoRA key naming on export.

Comfy-Org/ComfyUI#14182

Doesn't look like Comfyanon wants to merge this one

if that is the case, they should close the PR. As for the other points, we already had this discussion on Discord.

[dxqb]

By the way, this PR already includes conversion code: https://github.com/dxqb/OneTrainer/blob/03b7156c49bac5c8f5a8f13de259357f94047d75/modules/model/AnimaModel.py#L31

It's just not used for LoRAs currently (only for full finetunes), because this was the consistent and accepted way to do things for all other models. If inference tools want to change that now, they should make that clear (by closing the PR, for example)

[Silvicultor]

Did a lot of testing with the current version of the PR in the last 2 weeks. Overall Anima runs very good and stable in OT.
What I tested:
-Normal LoRA training works
-Anima LoKr works
-Masked training works fine aside from normalizing mask area loss can cause NaNs sometimes
-Torch compile, transformer blocks + optimizer, fully functional after workaround code was added
-Output quality of the LoRAs is similar to what other training tools produce (e. g. SD scripts).

So hoping to see this in the master branch soon! The only thing from my perspective that is left to overcome is the LoRA key issue, but I know it’s being discussed right now. Until then I convert manually to Comfy format.