fix: refactor set special tokens function and add unit tests.

[Luka-D]

Description of the change

Making a draft PR for everyone's thoughts on unit tests so far.

Missing Pad Token, LlamaTokenizerFast:

The first test uses a LlamaTokenizerFast tokenizer. This tokenizer is only missing a PAD token, however because it is a LlamaTokenizer, the function code automatically adds the bos, eos, unk and pad tokens to the special tokens dict. Then, the <pad> token is replaced with a <PAD> token, because the Llama tokenizer does not have a pad token specified.

EOS = PAD:

The second test uses a GPT2TokenizerFasttokenizer. This tokenizer is the case where the EOS token = PAD token, both of them are <|endoftext|>. So, the pad token in the tokenizer is set to <PAD> and the "pad_token": "<PAD>" is also added to the special tokens dict.

Missing Pad Token:

The third test uses a GPTNeoXTokenizerFasttokenizer. This tokenizer is another one that is hardcoded into the function to automatically add just a pad token to the special tokens dict. However, the tokenizer itself is also missing a pad token, so the function then replaces the <pad> token with the default <PAD> token.

Missing all tokens:

Added in 781ce58. This uses the IBM Granite tokenizer and removes all special tokens. The result is that the special tokens dict contains the PAD, EOS, BOS and UNK tokens.

Related issue number

Related to Issue #1515

How to verify the PR

You can run:
tox -e py -- tests/utils/test_tokenizer_data_utils.py

Was the PR tested

• I have added >=1 unit test(s) for every new method I have added.
• I have ensured all unit tests pass

[github-actions]

Thanks for making a pull request! 😃
One of the maintainers will review and advise on the next steps.

[Luka-D]

With commit 781ce58, I have added a new unit test for when a tokenizer has no special tokens. I figured it was easier to test a tokenizer that was missing all the tokens, instead of doing 4 unit tests for missing EOS, BOS, PAD etc. I can split it into 4 unit tests if you feel that is a better idea.

I also changed the conditional for the case of EOS = PAD token. Because both tokens are set to None, the conditional tokenizer.pad_token == tokenizer.eos_token would evaluate to True. I am unsure if we want to have the warning appear when both tokens are None, since I wouldn't really say they are identical, and the defaults that are added in the code above make them different anyways (PAD is set to <PAD> and EOS is set to </s>).
I have added if (tokenizer.pad_token is not None and tokenizer.pad_token == tokenizer.eos_token )to fix this.

Please let me know your thoughts on this. Thank you