diff --git a/.github/workflows/assess_new_production_model.yml b/.github/workflows/assess_new_production_model.yml index a1c4b97..06d0b6c 100644 --- a/.github/workflows/assess_new_production_model.yml +++ b/.github/workflows/assess_new_production_model.yml @@ -1,32 +1,32 @@ name: Deploy new production model on: push: - branches: ['main'] + jobs: - deploy: - runs-on: ubuntu-latest - name: Evaluate model and deploy to production if successful + helloworld: permissions: id-token: write contents: read + + runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - name: Configure AWS Credentials - uses: aws-actions/configure-aws-credentials@v1 - with: - role-to-assume: - aws-region: us-west-2 - - run: aws sts get-caller-identity - - name: Set up Python 3.x - uses: actions/setup-python@v1 - with: - python-version: '3.10' - - name: Install Outerbounds - run: | - python3 -m pip install --user outerbounds - - name: Test flow - env: - METAFLOW_HOME: /tmp/.metaflowconfig - run: | - - python evaluate_new_model_flow.py run --with card \ No newline at end of file + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: 3.11 + + - name: Install and Configure Outerbounds + run: |- + pip install -U outerbounds && \ + outerbounds service-principal-configure \ + --name \ + --deployment-domain \ + --perimeter \ + --github-actions + + - name: Run helloworld flow + run: |- + python evaluate_new_model_flow.py run --with card \ No newline at end of file diff --git a/README.md b/README.md index 36f9e07..a998dac 100644 --- a/README.md +++ b/README.md @@ -1,81 +1,22 @@ # GitHub Actions on Outerbounds Platform Demo A basic repo structure to run CI/CD jobs on Outerbounds platform. -## Related resources -[Github Actions x Outerbounds Service Principals](https://docs.google.com/document/d/1If-Nh4EY4cs5wDihWhnDglE-NKqu8Gv0-ZwXcw4cons/edit) -- This document describes how to set up a Github CI job that can run flows using Outerbounds Service Principals. - -## Workflows - - - -[](https://www.youtube.com/watch?v=XnW5MXzMEW8) - -### Engineer UX - -#### Initial Setup -This is something the person who handles cloud engineering/security will do once. - -##### 1. Find your AWS account ID as instructed [here](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.n2f7xpi062t8). -##### 2. Follow this section of the [Allow Github Actions Permissions to Assume your IAM Role](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.5cp00dpcus00) part of the [Service Principals guide](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit). -##### 3. [Create a new Permission Policy for Service Principals](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.p55n5nuncamf). -##### 4. [Create a Service Principal in Outerbounds UI](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.tdalusawlhk1). - -#### Modifying or adding a new CI/CD task in your action -This is a pattern that may require the end user who writes code that goes into the tasks of a FlowSpec run to communicate with the person who manages cloud engineering/security. It is the place where the security to run the action using the Outerbounds platform service principal connects to the logic that: -- runs the GitHub action CI job that runs the FlowSpec, -- deploys a new FlowSpec to a production branch or an experimental branch for A/B & multi-armed bandit scenarios. You might use the Metaflow [client API](https://docs.metaflow.org/api/client) to determine when the run has met some criteria. - -#### The workflow -##### 1. Identify the GitHub organization containing the repository where you want to add a new GitHub action. For example, this repository would be the “outerbounds” organization and “github-actions-on-obp-demo” repository. -##### 2. Go to step 5 and 6 of the [Create and Configure your IAM Role](https://docs.google.com/document/d/1If-Nh4EY4cs5wDihWhnDglE-NKqu8Gv0-ZwXcw4cons/edit) section, and follow instructions to add the action you want to the trust policy of your service principal IAM role. For example: here we define an action that runs when new code is pushed directly or merged to the main branch of the repository. - - - -##### 3. [Define the GitHub action](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.shunrk8q1a9d) in your GitHub repository with your FlowSpec code and its dependencies. - -There are two key pieces to look at in the example at `.github/workflows/assess_new_production_model.yml` to get this to work for your service principal. Descriptions are annotated inside `<>` in the following snippet. You will find the ARN in your AWS account in the IAM Role for the service pricipal, and you will find your Outerbounds configure command in the Outerbounds platform UI where you have connected the service principal to your account as a machine identity. -``` -name: Deploy new production model - -... - -jobs: - deploy: - - ... - - steps: - - uses: actions/checkout@v2 - - name: Configure AWS Credentials - uses: aws-actions/configure-aws-credentials@v1 - with: - role-to-assume: - - ... - - - name: Test flow - - ... - - run: | - - python evaluate_new_model_flow.py run --with card -``` - - -### Data Scientist UX -Our goal is to update the model used in the `Predict` workflow defined in `prediction_flow.py`. As a starting point for the CI/CD lifecycle, consider how a data scientist iterates locally or on a cloud workstation. - -This repository demonstrates how the data scientist can: -- take the result of such experimentation, +### Create a machine user in Outerbounds UI +- In the Outerbounds UI, go to the `Admin` panel on the left side navigation and select `Users`. +- Under `Machines` click the `Create New` button. +- Fill out the form, choosing the desired GitHub Actions form, and filling in the desired GitHub organization and repository. +- After submitting, click the row for the Machine User you created, and a code snippet will appear. +- Paste the command in actions file in `.github/workflows/` and modify it to run Metaflow code in the repository. + +### Write flows and run them on desired GitHub Actions +Our goal is to update the model used in the `Predict` workflow defined in `prediction_flow.py`. As a starting point for the CI/CD lifecycle, consider how a data scientist iterates. This repository demonstrates how to take the result of experimental, interactive development and use it to: - create a GitHub branch, -- let an automatic CI/CD process built with GitHub Actions validate the model's quality (using Outerbounds platform resources), -- and only if the new model code meets certain user-defined criteria, automatically deploy the newly trained model to be used in the production workflow that makes predictions accessed by other production applications. +- let an automatic CI/CD process built with GitHub Actions validate the model's quality (using Outerbounds platform resources), and +- if the new model code meets certain user-defined criteria, automatically deploy the newly trained model to be used in the production workflow that makes predictions accessed by other production applications. ### Deploy the `Predict` workflow to production -A data scientist or ML engineer would do this rarely, and typically less frequently than the model selction/architecture in `my_data_science_module.py` updates. +A data scientist or ML engineer would do this rarely, and typically less frequently than the model selection/architecture in `my_data_science_module.py` updates. This only needs to be done if the code in `predict_flow.py` file updates. ``` python predict_flow.py --production argo-workflows create @@ -87,13 +28,13 @@ This is a way to manually trigger a refresh of the production run that populates python predict_flow.py --production argo-workflows trigger ``` -### Development phase: Local iteration on `EvaluateNewModel` +### Local iteration on `EvaluateNewModel` Local/workstation testing: ``` python evaluate_new_model_flow.py run ``` -### Moving to production phase: a template for a CI/CD process using GitHub Actions +### CI/CD process using GitHub Actions When a data scientist is satisfied with what they see on local runs, then they can use GitHub commands like a regular software development workflow: ``` git switch -c 'my-new-model-branch' @@ -107,4 +48,4 @@ After the model is pushed to the remote branch of `my-new-model-branch`, the dat The GitHub Action in this template will do the following: 1. Run the `EvaluateNewModel` workflow defined in `evaluate_new_model_flow.py`. 2. If the `EvaluateNewModel` workflow produces a model that meets some user-defined criteria (e.g., beyond some performance metric threshold), then tag the Metaflow run in which the model was trained as a `deployment_candidate`. -3. If the upstream `EvaluateNewModel` run is tagged as a `deployment_candidate` and the model meets any other criteria you add to this template, then the production workflow will use a new version of the model in the `predict.py` flow in an on-going fashion. \ No newline at end of file +3. If the upstream `EvaluateNewModel` run is tagged as a `deployment_candidate` and the model meets any other criteria you add to this template, then the production workflow will use a new version of the model in the `predict.py` flow in an ongoing fashion. \ No newline at end of file