feat: Added reward model according to paper.

[ahmeterdempmk]

This PR implements the reward model training pipeline for DeepSeek-R1, adding key functionality for preference learning and model comparison.

Changes

• Addedreward_model.py implementing reward model training pipeline
• IntroducedRewardModelScriptArguments class for handling training configuration
• Implemented dataset preparation withprepare_comparison_dataset function
• Added support for better/worse response comparison training
• Integrated with HuggingFace's Trainer for model training
• Added model saving and Hub pushing capabilities

Implementation Details

• Uses AutoModelForSequenceClassification for reward modeling
• Supports configurable dataset columns for comparisons
• Implements efficient batch processing with proper tokenization
• Handles model checkpointing and evaluation

Testing

The implementation has been tested with:

• Dataset preparation and tokenization
• Model training pipeline
• Hub integration for model sharing

Next Steps

• Add evaluation metrics for reward model performance
• Implement additional data preprocessing options
• Add documentation for training configuration

[qgallouedec]

Thanks! Any reason for not using the RewardTrainer?

[ahmeterdempmk]

Because the implementation focuses on direct preference comparison training with better/worse response pairs, which is effectively handled by the standard Trainer with our custom dataset preparation.

[qgallouedec]

Okay, but that's exactly what the RewardTrainer does, isn't it?

[qgallouedec]

Can you also explain which part of the paper this addition corresponds to? It seems to me that the idea behind the RL of DeepSeek-R1 is to free it from reward models.