[Frontend] Adding the "User Defined Custom Tool Calling" parser for the Llama models

[lulmer]

Description

The current Llama tool parsing in vLLM is based on the JSON based tool calling using the procedure given by Meta. However, another tool parsing strategy is mentioned on this same website : The User Defined Custom Tool Calling.

The gain is substantial : After testing this approach as a plugin on a private function calling benchmark (more than 120 different scenarios tested with a set of 30 complex and lengthy fintech tool definitions), I observed significantly higher function-calling accuracy compared to the current JSON-based tool parser. I also run some experiments on the BFCL benchmark (AST non-live bench) and could observe the same types of improvements:

Model	AST Summary	Simple AST	Python Simple AST	Java Simple AST	JavaScript Simple AST	Multiple AST	Parallel AST	Parallel Multiple AST
Meta-Llama-3.1-8B-Instruct-FC (json tool call)	43.33%	56.33%	39.00%	60.00%	70.00%	29.50%	30.00%	18.50%
Meta-Llama-3.1-8B-Instruct-(usr def custom tc)	72.98%	67.58%	89.75%	57.00%	56.00%	86.50%	78.50%	75.50%

This PR introduces a new Llama3UserDefinedCustomToolParser class that extends the ToolParser base class. The new parser allows for streaming support when using custom tools with the Llama models. It handles the extraction of tool calls and arguments from the model's response in streaming too, enabling real-time processing of tool calls.

The flow looks like this :

Main Changes

1. New Parser Class: The Llama3UserDefinedCustomToolParser class is added to handle streaming tool calls for Llama models.
2. New Chat Template: An example of chat template that works well with this parser can be found here : vllm/examples/tool_chat_template_llama3.1_usr_def_tool_call.jinja

Remarks

This is my first PR on the vLLM project and I believe there is still some stuff I need guidance on :

• What is the testing process ? I don't really know how to proceed
• This parser requires a chat template that is different from the one that has been written for the JSON based tool calling although I provided it in the example directory I don't know how to indicate to the user how to use it.
• There might be some edge case I am not covering when handling the streaming tool parsing.
• I noticed there is a refactoring issue for the frontend going on [RFC]: Refactor tool parsers to eliminate coding errors and allow more efficient implementations. #11522 and discussion over potential benefits of using FSM for tool parsers, I don't know how it could impact the work here but I am happy to discuss potential changes.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[paolovic]

Hi @lulmer ,
I fixed the causes for the failing tests here: https://github.com/paolovic/vllm/tree/feature/llama_tool_calling .
Do you want to incorporate them into your PR or should I create a separate one?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lulmer.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[lulmer]

Hi @lulmer , I fixed the causes for the failing tests here: https://github.com/paolovic/vllm/tree/feature/llama_tool_calling . Do you want to incorporate them into your PR or should I create a separate one?

Thank you for your indications @paolovic, I didn't noticed I needed to address linting issues, I updated my branch accordingly as it was simple. I rebased with the current branch as well.

[paolovic]

Hi @lulmer ,

no problem, please don't forget to sign-off your commits with -s, e.g.
git commit -m "this is an example commit" -s, otherwise DCO will fail.

As it did now.
I think you can fix it with the instructions given here under "Rebase the branch"

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lulmer.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[lulmer]

Hi @lulmer ,

no problem, please don't forget to sign-off your commits with -s, e.g. git commit -m "this is an example commit" -s, otherwise DCO will fail.

As it did now. I think you can fix it with the instructions given here under "Rebase the branch"

@paolovic Thank you, I followed the procedure you've pinpointed, now the DCO is passing, but a lot of unrelated files/labels have been added in this PR, I had to solve a merge conflict (I systematically chose the latest updates from main branch) and add back pre commits for
examples/offline_inference/mistral-small.py.
Also I've noticed v1/core/kv_cache_manager.py is now part of the modified files, I hope it is expected and won't be a problem for the review. Otherwise let me know what I should do next.

[paolovic]

Hi @lulmer ,
is it possible that you are, unintentionally, try to push an outdated version of v1/core/kv_cache_manager.py to vllm:main?
If so, please make sure to sync your fork like so:

1. Go to your fork
2. Sync fork (on the top right corner)
3. Update Branch (Green Button)

[lulmer]

Hi @lulmer , is it possible that you are, unintentionally, try to push an outdated version of v1/core/kv_cache_manager.py to vllm:main? If so, please make sure to sync your fork like so:

1. Go to your fork
2. Sync fork (on the top right corner)
3. Update Branch (Green Button)

I ensured that I synced my fork when I did the rebase (If you see the history, I got told by Mergebot), on this commit

When I go to my fork on the github UI, the right tab sync fork says my fork is up to date with the latest main branch, see below.

[lulmer]

@paolovic is there any additional steps I should do now ?

[paolovic]

@lulmer tbh: i would close this PR and create a new, clean one where v1/core/kv_cache_manager.py is not listed as a changed file.

furthermore, fix the failing checks. without them it won't be merged.

FYI: I cannot and won't do the code review.

[lulmer]

@paolovic finally, I managed to remove the changes from v1/core/kv_cache_manager.py. I think it is fine now

[paolovic]

@lulmer nice, super cool!

By the way could you provide an example for me how to use it?

edit: I guess something like this

vllm serve --model meta/llama... \
            --chat-template examples/usr_defined... \
            --enable-auto-tool-choice --tool-call-parser llama

[lulmer]

@lulmer nice, super cool!

By the way could you provide an example for me how to use it?

edit: I guess something like this

vllm serve --model meta/llama... \
            --chat-template examples/usr_defined... \
            --enable-auto-tool-choice --tool-call-parser llama

This is a way to use it !

I always developed it as an external plugin but I would assume it can be launched this way:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_user_defined_custom \
    --chat-template examples/tool_chat_template_llama3.1_usr_def_tool_call.jinja

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @lulmer.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork