Add support for Prithvi in Online serving mode

[mgazz]

This PR builds on top of #20072 and it enables the execution of Prithvi in online serving mode.

This is achieved by increasing the level of support for models that skip the tokenizer initialisation.

A longer description of the what we are trying to achieve is available in #20234.

This supersedes #20307

Test Plan

The PR can be tested as following

pytest  tests/entrypoints/openai/test_skip_tokenizer.py

Additional information

Prithvi in serving mode can be started with the following command:

vllm serve --model='christian-pinto/Prithvi-EO-2.0-300M-TL-VLLM' --task embed --trust-remote-code --dtype float16 --skip-tokenizer-init --enforce-eager

The following script provides an example of prompt that can be used to perform an inference (the same is used during the test linked above):

import base64
import requests
import torch
import io
import numpy as np

torch.set_default_dtype(torch.float16)


def post_http_request(prompt: dict, api_url: str) -> requests.Response:
    headers = {"User-Agent": "Test Client","Content-Type": "application/json"}
    response = requests.post(api_url, headers=headers, json=prompt)
    return response


def decompress(output):
    np_result = np.frombuffer(
        base64.b64decode(output), dtype=np.float32)
    return np_result.reshape(1, 2, 512, 512)


def main():
    api_url = f"http://localhost:8000/pooling"
    model_name = 'christian-pinto/Prithvi-EO-2.0-300M-TL-VLLM'

    pixel_values = torch.full((6, 512, 512), 1.0,dtype=torch.float16)
    location_coords = torch.full((1, 2), 1.0,dtype=torch.float16)

    buffer_tiff = io.BytesIO()
    torch.save(pixel_values, buffer_tiff)
    buffer_tiff.seek(0)
    binary_data = buffer_tiff.read()
    base64_tensor_embedding = base64.b64encode(binary_data).decode('utf-8')

    buffer_coord = io.BytesIO()
    torch.save(location_coords, buffer_coord)
    buffer_coord.seek(0)
    binary_data = buffer_coord.read()
    base64_coord_embedding = base64.b64encode(binary_data).decode('utf-8')

    prompt={
        "model":model_name,
        "additional_data":{
            "prompt_token_ids": [1]
        },
        "encoding_format": "base64",
        "messages":[
            {
                "role": "user",
                "content": [
                        { "type": "image_embeds",
                        "image_embeds": {
                            "pixel_values": base64_tensor_embedding,
                            "location_coords": base64_coord_embedding,
                            },
                        }
                        ],
            }]}

    pooling_response = post_http_request(prompt=prompt, api_url=api_url)
    numpy_data = decompress(pooling_response.json()["data"][0]["data"])
    print(f"Returned result: {numpy_data}")


if __name__ == "__main__":
    main()

@DarkLight1337 @maxdebayser @njhill @christian-pinto

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This PR introduces support for Prithvi in online serving mode by allowing models to skip tokenizer initialization. The changes include modifications to the client, serving engine, serving pooling, and the Prithvi model itself. It's important to ensure that the tokenizer is checked for None before being used and that exceptions provide sufficient context for debugging.

[gemini-code-assist]

If self.tokenizer is None, this will raise an AttributeError. It's critical to ensure self.tokenizer is checked for None before being used here to prevent a crash. Consider adding a condition to skip this line if self.tokenizer is None.

        self.input_preprocessor = InputPreprocessor(self.model_config,
                                                    self.tokenizer if self.tokenizer else None)

[gemini-code-assist]

This exception lacks context about why the tokenizer is not initialized. Add more details to the exception message to aid debugging. For example, include the model name or a hint to check the --skip-tokenizer-init flag.

                raise Exception("Request must contain "
                                "additional_data['prompt_token_ids'] "
                                "when the tokenizer is not initialised. Check if '--skip-tokenizer-init' flag was used correctly for model {}".format(request.model))

[DarkLight1337]

Please also fix pre-commit

[DarkLight1337]

LGTM, thanks for adding support!

[mgazz]

@DarkLight1337 thank you for the support during the review process.

[DarkLight1337]

PTAL at the failing entrypoints test

[mgazz]

Apologies about the test. I will keep an eye on it.