Add mlp support for qwen3vl series and little refactor

[Tcc0403]

Summary

This PR aims to fix #956, plus some refactors. Including:

• adds swiglu and geglu mlp patching for qwen3vl series, swiglu for text model, geglu for vision model
  • geglu is default to False as it is not close to torch's impl, leading to convergence test failure.
• modifies qk norm patching with LigerRMSNorm(row_mode=True)
• adds layernorm patching for vision model

Note that moe layers aren't patched since there will be a major change in transformers v5, see #958.

Testing Done

• Hardware Type:
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

[thad0ctor]

Summary

This PR aims to fix #956, plus some refactors. Including:

• adds swiglu and geglu mlp patching for qwen3vl series, swiglu for text model, geglu for vision model

  • geglu is default to False as it is not close to torch's impl, leading to convergence test failure.

• modifies qk norm patching with LigerRMSNorm(row_mode=True)

• adds layernorm patching for vision model

Note that moe layers aren't patched since there will be a major change in transformers v5, see #958.

Testing Done

• Hardware Type:
• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

I commented on #897 about a regression to fcle increasing training speed. I applied this path locally to change the cuda syncing and make the fcle fsdp2 share aware and it seems to fix the regression and squeeze a little more t/s. It may be worth pulling the thread on this further: f22ce38

[Tcc0403]

I commented on #897 about a regression to fcle increasing training speed. I applied this path locally to change the cuda syncing and make the fcle fsdp2 share aware and it seems to fix the regression and squeeze a little more t/s. It may be worth pulling the thread on this further: f22ce38

Thank you, that's an interesting fix! We are planning a 2026 Q1 roadmap, including fsdp2 (multi-gpu) aware testings, optimizations and so on. Feel free to open a PR so we can discuss how we integrate your work align with our roadmap!

Regarding .item() optimization, have you tried reading it directly as a tensor value in triton kernel instead of converting it to python value as we currently do? I wonder how this approach does compared with your approach.