Skip to content

Commit c66394f

Browse files
CodemodService Botfacebook-github-bot
authored andcommitted
Fix CQS signal performance-faster-string-find in fbcode/pytorch/tokenizers/src (#86)
Summary: Pull Request resolved: #86 Reviewed By: aelatawy Differential Revision: D77128438
1 parent cf543d0 commit c66394f

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/tiktoken.cpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ static Result<std::pair<std::string, uint64_t>> _parse(
5151
// Tiktoken format
5252
// https://github.com/openai/tiktoken/blob/main/tiktoken/load.py#L140 <base64
5353
// encoded token str> <rank>
54-
auto pos = line.find(" ");
54+
auto pos = line.find(' ');
5555
TK_CHECK_OR_RETURN_ERROR(
5656
pos != std::string::npos,
5757
ParseFailure,

0 commit comments

Comments
 (0)