python3.PyImport_ImportModule(name) will emit a fatal error when called the second time. #29
Description
Describe what happened:
Environments:
On MacOS (Catalina Version 10.15.4)
Python3.7.6
Go1.13.8
I want to use go-python3 to invoke an algorithm written in Python3, but as described, an fatal error will generated when the second time I invoke this algorithm. From the output message, it seems that PyImport_ImportModule causes this error.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xa pc=0x91256a3]
runtime stack:
runtime.throw(0x4967a75, 0x2a)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:378 +0x47c
goroutine 41 [syscall]:
runtime.cgocall(0x4637740, 0xc000063c18, 0x48a4660)
/usr/local/go/src/runtime/cgocall.go:128 +0x5b fp=0xc000063be8 sp=0xc000063bb0 pc=0x4004d0b
github.com/DataDog/go-python3._Cfunc_PyImport_ImportModule(0x8061d90, 0x0)
_cgo_gotypes.go:3780 +0x4a fp=0xc000063c18 sp=0xc000063be8 pc=0x462c2fa
github.com/DataDog/go-python3.PyImport_ImportModule(0x49501f5, 0x8, 0x0)
/Users/zhao/go/pkg/mod/github.com/!data!dog/[email protected]/import.go:24 +0x87 fp=0xc000063c80 sp=0xc000063c18 pc=0x462e267
PPGServer/pkg/algo.ImportModule(0x4964926, 0x26, 0x49501f5, 0x8, 0x1)
/Users/zhao/go/src/PPGServer/pkg/algo/ppg.go:42 +0x4cb fp=0xc000063d98 sp=0xc000063c80 pc=0x46332db
PPGServer/pkg/algo.CalcPre(0xc0003560c0, 0xd, 0x0, 0x0)
....
Here is the sample code.
A wrapper of PyImport_ImportModule:
// ImportModule will import python module from given directory
func ImportModule(dir, name string) *python3.PyObject {
fmt.Println("python3.PyImport_ImportModule before")
sysModule := python3.PyImport_ImportModule("sys") // import sys
fmt.Println("python3.PyImport_ImportModule success")
path := sysModule.GetAttrString("path") // path = sys.path
ob := python3.PyList_GetItem(path, 1)
fmt.Println("check:", python3.PyUnicode_Check(ob))
fmt.Println("path:", python3.PyUnicode_AsUTF8(ob))
fmt.Println("sysModule.GetAttrString success")
python3.PyList_Insert(path, 0, python3.PyUnicode_FromString("/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages"))
python3.PyList_Insert(path, 0, python3.PyUnicode_FromString(dir))
fmt.Println("After module insert:", python3.PyUnicode_AsUTF8(python3.PyList_GetItem(path, 0)))
fmt.Println("module name:", name)
return python3.PyImport_ImportModule(name)
}
Each time the algorithm is called in a goroutine.
func CalcPre(dataFilePath string) (sbpI int, dbpI int) {
python3.Py_Initialize()
if !python3.Py_IsInitialized() {
fmt.Println("Error initializing the python interpreter")
os.Exit(1)
}
gstate = python3.PyGILState_Ensure()
fmt.Println("Py_Initialize success")
vbp := ImportModule("/Users/zhao/Desktop/lab/ppython", "value_bp")
fmt.Println("ImportModule success")
b := vbp.GetAttrString("estimate")
fmt.Printf("[FUNC] b = %#v\n", b)
bArgs := python3.PyTuple_New(1)
python3.PyTuple_SetItem(bArgs, 0, python3.PyUnicode_FromString(dataFilePath))
re := b.Call(bArgs, python3.Py_None)
sbp := python3.PyTuple_GetItem(re, 0)
dbp := python3.PyTuple_GetItem(re, 1)
defer func() {
python3.Py_Finalize()
fmt.Println("python3.Py_Finalize()")
}()
sbpI = python3.PyLong_AsLong(sbp)
dbpI = python3.PyLong_AsLong(dbp)
python3.PyGILState_Release(gstate)
return
}
func Calc(dataFilePath string) {
CalcPre(dataFilePath)
}
Sample caller like this: go Calc("aaa.csv").
Describe what you expected:
We can call go Calc() many times without a fatal error.
Steps to reproduce the issue:
Use the code above and environments above, put these code into a goroutine, like go Calc("aaa.csv").For simplicity, you may just remove the algorithm part and just have the skeleton remained.