✨ Add vLLM guided decoding support to GRPO Trainer

[kldzj]

What does this PR do?

Adds the ability to pass vLLM's GuidedDecodingParams through to the llm.generate call.

Example:

from trl import GRPOConfig, GRPOTrainer
from vllm.sampling_params import GuidedDecodingParams

training_args = GRPOConfig(
    use_vllm = True,
    vllm_guided_decoding_params = GuidedDecodingParams(
        backend="outlines",
        regex="<reasoning>\n.*\n</reasoning>\n<answer>\n.*\n</answer>",
    ),
    # ...
)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[qgallouedec]

Thanks for contributing @kldzj! For the record, can you explain briefly what is the motivation behind using GuidedDecodingParams?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[kldzj]

@qgallouedec When using the GRPO trainer, we likely want the model to respond in a specific format, in the example above we enforce the <reasoning>\n...\n</reasoning>\n<answer>\n...\n</answer> format right away, without spending many training steps for the model to learn the correct format through our reward functions.

Let me know if there's any problem or flaw in my logic with this.

[qgallouedec]

It's very interesting
Regarding the implementation, it a bit annoying because GuidedDecodingParams isn't json serializable so it causes error. A fair alternative is to only do like this instead

@dataclass
 class GRPOConfig(TrainingArguments):
    ...
    vllm_guided_decoding_regex: Optional[str] = None

and

if args.vllm_guided_decoding_regex is not None:
    guided_decoding = GuidedDecodingParams(backend="outlines", regex= args.vllm_guided_decoding_regex)

it's less flexible but explicitly exposes the regex and probably easier for the user.

[kldzj]

@qgallouedec Made the suggested change. :)