Skip to content

Commit 9b7f311

Browse files
Fix typo
Co-authored-by: Alexander Neubeck <[email protected]>
1 parent 3f1b056 commit 9b7f311

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

crates/bpe-openai/src/lib.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ pub use bpe::*;
4545
/// A byte-pair encoding tokenizer that supports a pre-tokenization regex.
4646
/// The direct methods on this type pre-tokenize the input text and should
4747
/// produce the same output as the tiktoken tokenizers. The type gives access
48-
/// to the regex and underlying bye-pair encoding if needed. Note that using
48+
/// to the regex and underlying byte-pair encoding if needed. Note that using
4949
/// the byte-pair encoding directly does not take the regex into account and
5050
/// may result in output that differs from tiktoken.
5151
pub struct Tokenizer {

0 commit comments

Comments
 (0)