OpenAI_Count_tokens_with_tiktoken.md

Template request | Bug report | Generate Data Product

Tags: #openai #tiktoken #count #token #tokens #cookbook

Author: Florent Ravenel

Description: This notebook shows how to count tokens used from a string with tiktoken to use OpenAI API.

References:

• OpenAI Cookbook
• How to count tokens with tiktoken

Input

Import libraries

try:
    import tiktoken
except:
    !pip install tiktoken --user
    import tiktoken

Setup Variables

• text_string: Given text string
• encoding_name: Encoding

text_string = "tiktoken is great!"
encoding_name = "cl100k_base"

Model

Count tokens

Count tokens by counting the length of the list returned by .encode()

def num_tokens_from_string(string: str, encoding_name: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

Output

Display result

num_tokens = num_tokens_from_string(text_string, encoding_name)
print("Tokens:", num_tokens)