add local tokenizer option for automated testing without hf token

[sixiang-google]

No description provided.

[qihqi]

btw, For automated testing, one just need to create subdirectory structures in the checkpoint dir that mimics hf then it will read from there without redownloadi from HF.

Example

checkpoints/
checkpoints/meta-llama
checkpoints/meta-llama/Llama-2-7b-hf
checkpoints/meta-llama/Llama-2-7b-hf/hf_original
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/config.json
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/model-00001-of-00002.safetensors
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/generation_config.json
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/model-00002-of-00002.safetensors
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/tokenizer.model
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/model.safetensors.index.json
checkpoints/meta-llama/Llama-2-7b-hf/hf_original/pytorch_model.bin.index.json