You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: crates/bpe-openai/README.md
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -2,7 +2,7 @@
2
2
3
3
Fast tokenizers for OpenAI token sets based on the [bpe](https://crates.io/crates/bpe) crate.
4
4
Serialized BPE instances are generated during build and lazily loaded at runtime as static values.
5
-
The overhead of loading the tokenizers is small because it happens only once per proces and only requires deserialization (as opposed to actually building the internal datastructures).
5
+
The overhead of loading the tokenizers is small because it happens only once per process and only requires deserialization (as opposed to actually building the internal data structures).
6
6
For convencience it re-exports the `bpe` crate so that depending on this crate is enough to use these tokenizers.
0 commit comments