kgrams_counts bug

[wzy816]

Maybe it is trivial and I am wrong.

From the paper I think the count of a k-gram "word" is its occurrence in the corpus data not in its higher-order gram types. If this is the case,

kneser-ney/kneser_ney.py

Line 58 in 2740fba

new_order[suffix] += 1

should be changed to

new_order[suffix] += last_order[ngram]

But even this is troublesome. For example, ('?', '', '') in the last_order will add its suffix ('', '') to the new_order. But I think two pad symbol is not valid in a bigram model.

Therefore, I think a better way to do kgram count is to do each order independently and directly from corpus data.
And in the class KneserNeyLM definition, using highest_order gram ngrams as arg and in the example.py usinggut_ngrams need to be revised as well.

[robinn37]

It is the adjusted count or unique prefix count for low order here. I think +1 is correct here.