The kl divergence collapses but the format reward becomes larger

[yuki-younai]

I had a problem where the KL divergence suddenly became very large during training and the format reward suddenly went up, causing the training to crash.
#255

[yuki-younai]

I used Qwen-2.5-1.5B-Instrcut for grpo training without changing any configuration files。