Open
Description
c.f. #1553 (comment)
Currently, sql
package contains almost all core code for parsing, generating python code and executing. We need to put those features in a separated package structure for better code understanding:
parser
package is already moved underpkg
folderfeature_derivation
package is already moved underpkg
folder- tools like
pipe
,verifier
are already moved underpkg
folder - TODO: move
ir
to pkg folder - TODO: move
testdata
to pkg folder
We currently have two job execution mode: workflow mode and run in local mode.
- workflow mode, we generate a Couler python program, then execute the python program to generate a YAML file, then call functions under
pkg/argo
to submit it and monitor the job's status. Each step in the workflow is arepl
command. - repl mode, we generate some actual Python training/predicting code and execute the python code to get the result.
We may need to use a command step
instead of repl
to be more meaningful. For that w'll have
cmd/step
callspkg/step
to run a step or, in the future generate a step Python codepkg/step
contains:- run current step and get output (will only be used by repl if we generate python code for each step but not using the command
step
to run a single SQL statement) pkg/step/codegen
generate step python code
- run current step and get output (will only be used by repl if we generate python code for each step but not using the command
pkg/workflow
contains:- submit workflow and monitor the status
pkg/workflow/codegen
generate Couler/Fluid python codepkg/workflow/argo
submit, get status, get logs for argopkg/workflow/tekton
submit, get status, get logs for tekton