feat: setup tokenizer on the provider model config #656
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0. Pulls message conversion utils into format module 1. Overhauls the provider tests which were flakey in CI 2. Removes moderation references for now, as we are investigating how to bring them in without the false positives 3. Removes cost tracking, as we don't want to keep up to date with the pricing details. We will track tokens instead
it was nice to have ollama testing the format as an integration in CI, but those are covered well by unit tests already and this increased test times significantly
salman1993
commented
Jan 20, 2025
crates/goose/build.rs
Outdated
| "Xenova/llama3-tokenizer", | ||
| "Xenova/gemma-2-tokenizer", | ||
| "Qwen/Qwen2.5-Coder-32B-Instruct", | ||
| ]; |
There was a problem hiding this comment.
not sure if it makes sense to embed 5 tokenizer files in the binary. maybe we just need 2 or 3? @baxen @michaelneale
There was a problem hiding this comment.
big +1 let's remove all we can!
There was a problem hiding this comment.
i just kept gpt-4o and claude tokenizer. removed the other 3 (fallback is to download so its okay)
baxen
approved these changes
Jan 20, 2025
crates/goose/build.rs
Outdated
| "Xenova/llama3-tokenizer", | ||
| "Xenova/gemma-2-tokenizer", | ||
| "Qwen/Qwen2.5-Coder-32B-Instruct", | ||
| ]; |
There was a problem hiding this comment.
big +1 let's remove all we can!
* origin/v1.0: fix: clean up providers (#650)
acekyd
pushed a commit
that referenced
this pull request
Jan 21, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
branched off this PR branch: #650 (will merge after that PR)
only one tokenizer gets used depending on the model, so we put tokenizer name on the provider and then try our best to map the tokenizer name. this way we can avoid loading up these tokenizers that aren't used