Wrap goalisa into a python module

[shendiaomo]

Is your feature request related to a problem? Please describe.
To migrate feature derivation into python, we need a mechanism to access MaxCompute though alisa

Describe the solution you'd like
step 1: wrap goalisa into a python module using https://github.com/go-python/gopy or similar tools.
step 2(optional): rewrite the alisa client in python.

Describe alternatives (optional)
None

Additional Notes

[shendiaomo]

This is critical to enable core code migration to python.

[lhw362950217]

I have made some progress this week. In short, the answer is, yes we can wrap go code into a Python module. Here I just make some simple notes, and quickly I will write a design doc and make a PR.

Experiment:

We only need this line of command to generate a Python module from existing go pkg.

gopy build -name=pydb -output=./pydb sqlflow.org/gomaxcompute sqlflow.org/gohive sqlflow.org/goalisa sqlflow.org/database  database/sql

Using gopy, I successfully converted our goalisa driver into a Python module. The idea behind the tool is described below:

1. generate a cgo file(just a go file with c code) which import the package we want to transfrom. In the cgo file, public structs and vars are marked with //export comment, so they will be exported to a c header file
2. use cgo tool to build a .so file as well as generate a .h file, all exported data structure are declared in this .h file
3. gopy will parse the given package and get same data structures declared in the .h file, then, gopy will use the metadata of these data structures to generate a build.py file
4. the build.py file is an starter for PyBindGen which is another tool to generate Python's .c wrapper file for given.h
5. finally, we have a .h file which export functions and structs from the .so file, and a .c file which wraps the functions and structs to python object, and a .so file with go code exported, so, just build them all as a dynamic linked library
6. this final dynamic linked library can be loaded as a Python module

I also try to build a module which wraps all our go driver, including MySQL, Hive, Alisa, ODPS, it just works.

What we can/can't do:

At first look, actually in most cases, it seems we can wrap almost anything to Python without writing a single line of code. However, the tool gopy is not so prefect now, it can not export var-len param funcs, and neither have a good support to export []interface{}. So, in rare case, we need to write some go code to make these feature available.

There are other drawbacks, wrapped type exported to Python are not easy to use. Say, returned []A from go code is just a Go handle, not a real Python slice of A. We have to call some function to get each A's field. I will figure out a way to deal with this. At the worst case, we can write a little bit of go code to wrap the funcs which can't export normally.

Conclusion:

As described above, I think our solution works. With slightly modification of existing code, we can wrap almost all our go code into a Python module. This is useful because we can reuse all the go drivers we previously wrote.

Todo:

• try to find a way to convert wrapped data to native Python objects

[typhoonzero]

@shendiaomo Can we only do step 2 so that we don't have to write some code that will eventually be deleted.

[shendiaomo]

@shendiaomo Can we only do step 2 so that we don't have to write some code that will eventually be deleted.

From your point of view, the better choice is that we only do step 1 and never do step 2?

[weiguoz]

I learned from @lhw362950217 the Python module is a Python3 version.
And I noticed the PAI tf1.15 only supports Python2. relative discussion
Does the Python version matter?