Skip to content

Commit 4c093b5

Browse files
committed
Add support for fine-tuned models in encoding_for_model
Identify models that can be fine-tuned in encoding_for_model. - See https://platform.openai.com/docs/models/model-endpoint-compatibility - See https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
1 parent 095924e commit 4c093b5

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

tiktoken/model.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@
88
# chat
99
"gpt-4-": "cl100k_base", # e.g., gpt-4-0314, etc., plus gpt-4-32k
1010
"gpt-3.5-turbo-": "cl100k_base", # e.g, gpt-3.5-turbo-0301, -0401, etc.
11+
# fine-tuned models
12+
"davinci": "r50k_base",
13+
"curie": "r50k_base",
14+
"babbage": "r50k_base",
15+
"ada": "r50k_base",
1116
}
1217

1318
MODEL_TO_ENCODING: dict[str, str] = {

0 commit comments

Comments
 (0)