Skip to content

Added vllm generation; Improved one error message #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

ytzi
Copy link
Contributor

@ytzi ytzi commented May 20, 2025

Adding a vllm generation backend that should be compatible with the original generation backend using Transformers pipeline.

Important difference compared to original backend:

  • Supports multi-gpu;
  • no batch_size parameter: vllm dynamically chooses the most optimal batch size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant