We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
1 parent 3f1b056 commit 9b7f311Copy full SHA for 9b7f311
crates/bpe-openai/src/lib.rs
@@ -45,7 +45,7 @@ pub use bpe::*;
45
/// A byte-pair encoding tokenizer that supports a pre-tokenization regex.
46
/// The direct methods on this type pre-tokenize the input text and should
47
/// produce the same output as the tiktoken tokenizers. The type gives access
48
-/// to the regex and underlying bye-pair encoding if needed. Note that using
+/// to the regex and underlying byte-pair encoding if needed. Note that using
49
/// the byte-pair encoding directly does not take the regex into account and
50
/// may result in output that differs from tiktoken.
51
pub struct Tokenizer {
0 commit comments