Skip to content

JKU KV Azure Course Project - AI/ML Dataset Processing Pipeline on Azure with ADF and Batch

License

Notifications You must be signed in to change notification settings

mahdikhashan/azure-dataset-processing-pipeline

Repository files navigation

Azure Dataset Processing Pipeline

architecture

Setup and deployment

Make sure nix package manager is available on host os and run the following command to setting up the re-producible environment:

nix-shell

it will setup local environments with Azure CLI, Python and requirements for jobs (scripts) to be tested on local.

  • add required python modules to requirements.txt for scripts in the scripts directory.

Deployment

Steps:

  • Deploy all resources to Azure
  • Generate Connection string for blob account
  • Copy Scripts to the scripts blob
  • Run Pipeline

Debug

  • while working on the project, I was in need to monitor steps and deubg/make sure everything is correctly set up. for further information, check commands in Makefile with debug prefix.

Issues

Documentation

About

JKU KV Azure Course Project - AI/ML Dataset Processing Pipeline on Azure with ADF and Batch

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published