Adding new language models

[markov00]

Hi, I was interested to know how can be possible to create a new language model.
Let's say that I'm interested in adding an Italian language model, how can I start building that model?

Thanks

[manning]

In general, we'd be very interested in contributed language models! We don't have a guide on how to produce one at the moment, though some of the tools, like POS tagger, parser, NER do have some discussion of training models in their own documentation. The best you can do is look at examples like French or Spanish. You need to (i) have labeled data as needed for various tasks, (ii) write a few language-particular Java classes as needed for things like tokenization, different corpus format handling, recognizing punctuation, (iii) train models, (iv) provide config files (properties files) for configuring the language.

For Italian in particular, this has basically been done, see the larger Tint project: http://stanfordnlp.github.io/CoreNLP/human-languages.html#models-for-other-languages . It is my hope that at some point the appropriate models, resources, and code there could also be repackaged to provide a "pure" Stanford CoreNLP language pack for Italian.