Skip to content

[Frontend] Add request_id to the Request object so they can be controlled better via external load balancers #21009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 25, 2025
Merged

[Frontend] Add request_id to the Request object so they can be controlled better via external load balancers #21009

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b0654d6
wip
kouroshHakha Jul 15, 2025
6ea6e6c
Merge branch 'main' of https://github.com/vllm-project/vllm into kh/a…
kouroshHakha Jul 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions vllm/entrypoints/openai/protocol.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1007,6 +1007,13 @@ class CompletionRequest(OpenAIBaseModel):
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."),
)
request_id: str = Field(
default_factory=lambda: f"{random_uuid()}",
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."),
)
Comment on lines +1010 to +1016
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The definition for the request_id field is duplicated across CompletionRequest, EmbeddingCompletionRequest, and EmbeddingChatRequest in this PR. A similar definition also exists in ChatCompletionRequest. This duplication can lead to maintenance issues in the future (e.g., if the description or default factory needs to be updated, it must be changed in multiple places).

To improve maintainability and adhere to the DRY (Don't Repeat Yourself) principle, I suggest defining the Field object once and reusing it in all relevant request classes.

For example, you could define a shared field at the module level:

# At an appropriate module-level scope
_REQUEST_ID_FIELD = Field(
    default_factory=lambda: f"{random_uuid()}",
    description=(
        "The request_id related to this request. If the caller does "
        "not set it, a random_uuid will be generated. This id is used "
        "through out the inference process and return in response."),
)

# In each request class
class CompletionRequest(OpenAIBaseModel):
    # ...
    request_id: str = _REQUEST_ID_FIELD
    # ...

This change should be applied to EmbeddingCompletionRequest and EmbeddingChatRequest as well.

logits_processors: Optional[LogitsProcessors] = Field(
default=None,
description=(
Expand Down Expand Up @@ -1251,6 +1258,13 @@ class EmbeddingCompletionRequest(OpenAIBaseModel):
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."),
)
request_id: str = Field(
default_factory=lambda: f"{random_uuid()}",
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."),
)
Comment on lines +1261 to +1267
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This is a duplicated definition of the request_id field. Please apply the same refactoring suggested for CompletionRequest to avoid code duplication and improve maintainability.


# --8<-- [end:embedding-extra-params]

Expand Down Expand Up @@ -1302,6 +1316,13 @@ class EmbeddingChatRequest(OpenAIBaseModel):
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."),
)
request_id: str = Field(
default_factory=lambda: f"{random_uuid()}",
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."),
)
Comment on lines +1319 to +1325
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This is a duplicated definition of the request_id field. Please apply the same refactoring suggested for CompletionRequest to avoid code duplication and improve maintainability.

# --8<-- [end:chat-embedding-extra-params]

@model_validator(mode="before")
Expand Down
4 changes: 3 additions & 1 deletion vllm/entrypoints/openai/serving_completion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,9 @@ async def create_completion(
return self.create_error_response(
"Echo is unsupported with prompt embeds.")

request_id = f"cmpl-{self._base_request_id(raw_request)}"
request_id = (
f"cmpl-"
f"{self._base_request_id(raw_request, request.request_id)}")
created_time = int(time.time())

request_metadata = RequestResponseMetadata(request_id=request_id)
Expand Down
5 changes: 3 additions & 2 deletions vllm/entrypoints/openai/serving_embedding.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,9 @@ async def create_embedding(
for the API specification. This API mimics the OpenAI Embedding API.
"""
model_name = self._get_model_name(request.model)
request_id = (f"{self.request_id_prefix}-"
f"{self._base_request_id(raw_request)}")
request_id = (
f"{self.request_id_prefix}-"
f"{self._base_request_id(raw_request, request.request_id)}")

ctx = EmbeddingServeContext(
request=request,
Expand Down