Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Summary: X-link: pytorch-labs/tokenizers#21 Test Plan: ## OSS Build ``` cmake . -DCMAKE_INSTALL_PREFIX=cmake-out -DTOKENIZERS_BUILD_TEST=ON -Bcmake-out cmake --build cmake-out -j9 --target install ``` Test ``` (executorch) [[email protected] /data/users/lfq/tokenizers/cmake-out (lfq.tokenizer-test)]$ ctest Test project /data/users/lfq/tokenizers/cmake-out Start 1: test_base64 1/5 Test pytorch-labs/tokenizers#1: test_base64 ...................... Passed 0.00 sec Start 2: test_llama2c_tokenizer 2/5 Test pytorch-labs/tokenizers#2: test_llama2c_tokenizer ........... Passed 0.00 sec Start 3: test_pre_tokenizer 3/5 Test pytorch-labs/tokenizers#3: test_pre_tokenizer ............... Passed 0.73 sec Start 4: test_sentencepiece 4/5 Test pytorch-labs/tokenizers#4: test_sentencepiece ............... Passed 0.04 sec Start 5: test_tiktoken 5/5 Test pytorch-labs/tokenizers#5: test_tiktoken .................... Passed 3.32 sec 100% tests passed, 0 tests failed out of 5 Total Test time (real) = 4.10 sec ``` ## Internal ``` buck2 test fbsource//xplat/pytorch/tokenizers/test: buck2 test fbcode//pytorch/tokenizers/test: ``` Reviewed By: larryliu0820 Differential Revision: D69860352 Pulled By: lucylq
- Loading branch information