Skip to content

Commit fcd24fd

Browse files
authored
Fix deployment of spark_python_operator (#1033)
This properly packages the spark operation "kernelspec" and adds spark_python_operator to the corresponding list of exposed kernelspecs in the k8s deployment scripts. The operator-related scripts/templates have been moved to kernel-launchers/operators to more easily enable deployment and not clutter other Kubernetes kernelspecs with unrelated scripts/templates.
1 parent 3ac7277 commit fcd24fd

File tree

5 files changed

+4
-2
lines changed

5 files changed

+4
-2
lines changed

etc/Makefile

+2-1
Original file line numberDiff line numberDiff line change
@@ -52,9 +52,10 @@ TOREE_LAUNCHER_FILES:=$(shell find kernel-launchers/scala/toree-launcher/src -ty
5252
@echo ../build/kernelspecs/spark_python_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/python/scripts
5353
@echo ../build/kernelspecs/spark_R_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/R/scripts
5454
@echo ../build/kernelspecs/spark_scala_{conductor*,yarn*} | xargs -t -n 1 cp -r kernel-launchers/scala/lib
55-
# Container-based kernelspecs just get the container launchers
55+
# Container-based kernelspecs (and operators) just get the container launchers
5656
@echo ../build/kernelspecs/{python,R,scala,python_tf,python_tf_gpu}_kubernetes | xargs -t -n 1 cp -r kernel-launchers/kubernetes/*
5757
@echo ../build/kernelspecs/{python,R,scala,python_tf,python_tf_gpu}_docker | xargs -t -n 1 cp -r kernel-launchers/docker/*
58+
@echo ../build/kernelspecs/spark_python_operator | xargs -t -n 1 cp -r kernel-launchers/operators/*
5859
# Populate kernel resources. Because tensorflow is also python, it should be last.
5960
@echo ../build/kernelspecs/*R* | xargs -t -n 1 cp -r kernel-resources/ir/*
6061
@echo ../build/kernelspecs/*scala* | xargs -t -n 1 cp -r kernel-resources/apache_toree/*

etc/kubernetes/enterprise-gateway.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ spec:
184184
value: "60"
185185

186186
- name: EG_KERNEL_WHITELIST
187-
value: "'r_kubernetes','python_kubernetes','python_tf_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes'"
187+
value: "'r_kubernetes','python_kubernetes','python_tf_kubernetes','python_tf_gpu_kubernetes','scala_kubernetes','spark_r_kubernetes','spark_python_kubernetes','spark_scala_kubernetes','spark_python_operator'"
188188

189189
- name: EG_DEFAULT_KERNEL_NAME
190190
value: "python_kubernetes"

etc/kubernetes/helm/enterprise-gateway/values.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ kernel:
3636
- spark_r_kubernetes
3737
- spark_python_kubernetes
3838
- spark_scala_kubernetes
39+
- spark_python_operator
3940
# Default kernel name should be something from the whitelist
4041
defaultKernelName: python_kubernetes
4142

0 commit comments

Comments
 (0)