Proposal - .NET tokenization library

[lqdev]

Data preparation in NLP usually involves some sort of tokenization. State of the art model architectures like transformers have their own way of preprocessing text. Libraries like those from HuggingFace have not only simplified the modeling process for complex NLP models, but also the preprocessing pipeline.

The idea is to create a tokenization library for .NET that makes it simpler to use downstream for modeling with other libraries such as TensorFlow.NET, TensorFlow.Keras, TorchSharp, DiffSharp, ML .NET, etc.

Two potential approaches:

• Standalone .NET library: With this approach, there are no dependencies and maintainers are in full control. However, almost everything would be built from scratch.
• HuggingFace Tokenizers .NET bindings: HuggingFace has created the tokenizers library. This library is built using Rust. The crate is then used to build bindings for other languages. Currently there are NodeJS & Python bindings. Building .NET bindings would follow the already established pattern (Tokenizers crate -> Rust dynamic library -> .NET class library). Although this adds a dependency on the Tokenizers library, the core components are already built and the .NET bindings would extend a well known and adopted library.

[dsyme]

I posted this in TorchSharp to link up discussions dotnet/TorchSharp#248

@NiklasGustafsson has been doing work there on samples requiring tokenization

[dsyme]

Also there is work along these lines in DiffSharp I think

[luisquintanilla]

Great! Thanks Don.

[WalternativE]

Any way to collaborate with the Curiosity AI folks on that one? They pride themselves on their tokenization as far as I can tell. Or would their implementation be to opinionated? https://github.com/curiosity-ai/catalyst

[luisquintanilla]

I don't see why not. I think Tokenizers from HuggingFace are more focused on transformer architectures, but not all NLP work is transformer-based or even model based so there's definitely opportunities to provide different options depending on the problem.

[WalternativE]

Ah, I see. My NLP knowledge is pretty rusty (think tokenization to build inverse indices, Lucene, a bit of Spacy) so I didn't know, that standard NLP tokenization and tokenization for transforms are so different. I'd be all for having a library all .NET programmers could use :)

[GeorgeS2019]

I suggest two categories of tokenizer

Those used by non transformer (e.g. Spacy etc.), which is probably available more common among .NET NLP
Those adhere closely to the HuggingFace. This is where the .NET community needs to collaboate.

[GeorgeS2019]

We are now sorting out here, please join the discussion

The list of tokenizers considered in TorchText.Data Utils.cs are

• spacy
• moses ( perhaps fast-mosestokenizer )
• toktok
• revtok
• subword (SentencePiece as used in Seq2SeqSharp)

[GeorgeS2019]

Not to hijack this discussion. Is there a interest of using BlingFire and making it more End-To-End available through TorchSharp?