"Chinese dot" breaks the fingerprint method

[reinoldus]

Hi everyone,

I noticed that if this "chinese dot" is included in the fingerprint hash "。" then the hash is always "ffffffffffffffff"

from trafilatura.deduplication import content_fingerprint, Simhash
content_a = Simhash("""行政長官岑浩""")


print(Simhash("""行政長官岑浩""").to_hex()) # 13dd8c82d4634a48
print(Simhash("""欢迎与我交流""").to_hex()) # 58429793861fa351
print(Simhash("""行政長官岑浩。""").to_hex()) #ffffffffffffffff
print(Simhash("""欢迎与我交流。""").to_hex()) #ffffffffffffffff

I hope the encoding is not all messed up.

[reinoldus]

I did some more debugging and it seems like the sample_tokens function returns an empty list, because due to the "chinese dot" not being detected as punctuation the isalnum check fails:

        if token.isalnum():
            tokens.append(token)

We have a couple of options here:

Use the "regex" lib to remove unicode punctuation

    import regex  
    strip_punct = regex.compile(r'\p{P}+')
    token = strip_punct.sub('', token)

or this approach:

import unicodedata
import sys

tbl = dict.fromkeys(i for i in range(sys.maxunicode)
                      if unicodedata.category(chr(i)).startswith('P'))
def remove_punctuation(text):
    return text.translate(tbl)

Or for this particular case we could just append the whole text if no tokens have been found (probably the worst option)

    if not tokens:
        tokens.append(inputstring)

Let me know if any of those options would work for you and I will open a PR.

[adbar]

Hi @reinoldus, thanks for the suggestions, it actually amounts to a tokenization issue.

I believe the most efficient approach is to add the character to string.punctuation in a global variable and then to replace this line in the deduplication module (just above):
token = token.strip(string.punctuation)

Feel free to draft a pull request on it, I could review and integrate it.