dataset-processing-volcano

a kata to process a parquet dataset on k8s cluster using Volcano with a custom queue

Run

steps:

1. create the pvc and copy the dataset to it
2. create configmap with script and its reuquirements
3. create the dataset queue
4. apply the job

1

kubectl apply -f ./manifests/pvc.yaml

then

bash copy.sh

2

kubectl create configmap extract-script --from-file=tasks/extract/extract_script.py --from-file=tasks/extract/requirements.txt

3

kubectl apply -f ./manifest/volcano/queue-dataset.yaml

4

kubectl apply -f ./manifest/volcano/job-extract-mnist.yaml

---