Voxtral #20970
Voxtral #20970
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @patrickvonplaten, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request integrates the Voxtral audio-language model into vLLM, allowing for multimodal inference that processes both audio and text. It involves significant updates to the input handling, model architecture, and configuration system to support this new capability, building upon the mistral_common library for audio processing and tokenization.
Highlights
- New Model Support: This pull request introduces comprehensive support for the 'Voxtral' audio-language model, enabling multimodal inference capabilities within vLLM that combine both audio and text inputs.
- Enhanced Multimodal Input Handling: The system's input processing has been significantly upgraded to natively handle audio data alongside traditional text prompts. This includes advanced tokenization and embedding generation tailored for multimodal inputs.
- Deep
mistral_commonIntegration: The changes heavily leverage themistral_commonlibrary for core audio processing, tokenization, and adherence to chat completion and transcription protocols, ensuring seamless compatibility with Mistral's multimodal model ecosystem. - Whisper Encoder Reusability: The existing Whisper encoder implementation has been adapted and modified to function as a reusable, standalone component within the Voxtral model. This involved adjustments to its attention mechanisms and weight loading procedures.
- Flexible Configuration System: The configuration parsing logic has been extended to accurately identify and set up audio-language models like Voxtral. This includes remapping Mistral-specific multimodal arguments to vLLM's internal model configurations for proper initialization.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
mergify
bot
added
documentation
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Voxtral model, a multi-modal audio-language model. The changes are comprehensive and well-structured. I have provided feedback to improve code clarity, maintainability, and address a few dependency updates.
| # @cached_property | ||
| # def begin_transcript_token_id(self) -> int: | ||
| # return self._audio_processor.special_ids.begin_transcript | ||
|
|
||
| # @cached_property | ||
| # def end_transcript_token_id(self) -> int: | ||
| # return self._audio_processor.special_ids.end_transcript |
There was a problem hiding this comment.
Is this needed for transcription?
There was a problem hiding this comment.
Ah if it's commented out probably not -> I can kill it
This reverts commit 6a699ec.
Head branch was pushed to by a user without write access
DarkLight1337
removed request for
zou3519,
comaniac,
LucasWilkinson,
njhill,
zhuohan123,
youkaichao,
houseroad,
WoosukKwon,
alexm-redhat,
hmellor,
jeejeelee,
russellb and
tlrmchlsmth
Signed-off-by: Patrick von Platen <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Patrick von Platen <[email protected]>
Signed-off-by: Patrick von Platen <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> Signed-off-by: Himanshu Jaju <[email protected]>
Signed-off-by: Patrick von Platen <[email protected]> Co-authored-by: Cyrus Leung <[email protected]>
Needs: mistralai/mistral-common#119