Platform to implement a compiled variant of NoGraphs

[HeWeMel]

NoGraphs could be provided as compiled extension, additionally to the variant as pure Python source code. This could increase speed for applications, at least noticeable, and even significantly for cases where native data types are used.

But there are special problems:

• NoGraphs is highly generic (it resembles a template library in C++ with this respect).
• It calls functions defined outside the library very often (the callback functions).
• It returns results in a pipelined way, implemented as iterator.
• It is open to get its internal data structures replace by external ones
• NoGraphs should be reliable for users (instabilities that stem from the concept / platform used for the compiled version should be avoided)
• And the maintainer does not want to maintain a completely different second code base for the compiled version of the library :-)

What platform should be used for generating a compiled version of NoGraphs? Why is it suitable for NoGraphs? How could a good speed increase in general, and a strong speed increase for cases with native data types and list-based storage, be realized on this platform?

[HeWeMel]

Proposal: Cython

First, vague, arguments:

PRO:

• Stable. Often used. Documted.
• Fast interface from / to Python code outside compiled methods
• CI pipeline ready, for all mayor target systems.
• Support for direct access to native datatypes stored in arrays (lists, NumPy, …).
• Generic classes accessed by Python are fully supported, and classes with native fixed datatypes also, even mixed

Cons:

• Hard to debug
• Error-prone
• Code will tend to deviate more and more from standard Python

[HeWeMel]

Proposal: Mypyc

First, vague, arguments:

Pro:

• Migration path for existing code.
• Python ecosystem.
• Waiting for compilation is optional.
• Runtime type safety. Finds errors statically.
• NoGraphs might be well prepared, due to its fully typed code.

Cons:

• Alpha release
• Currently still important limitations

[HeWeMel]

Proposal: Rust

First, vague, arguments:

PRO:

• Well-designed and modern language.
• Resulting code is very reliable.
• Good infrastructure. Good support for many platforms.

CONTRA:

• No common code base with pure-Python version possible.
• Interface to Python might be slow in both directions
• Highly generic interface of NoGraphs might be impossible.

[leshabirukov]

https://softwareengineering.stackexchange.com/questions/33020/when-is-optimization-not-premature-and-therefore-not-evil