Skip to content

Commit

Permalink
copy paste instruction from ui
Browse files Browse the repository at this point in the history
  • Loading branch information
@emattia
emattia committed Jun 12, 2024
1 parent 6bfee4f commit 3adce7d
Show file tree
Hide file tree
Showing 2 changed files with 39 additions and 98 deletions.
48 changes: 24 additions & 24 deletions .github/workflows/assess_new_production_model.yml
View file
Original file line number Diff line number Diff line change
@@ -1,32 +1,32 @@
name: Deploy new production model
on:
push:
branches: ['main']

jobs:
deploy:
runs-on: ubuntu-latest
name: Evaluate model and deploy to production if successful
helloworld:
permissions:
id-token: write
contents: read

runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: <YOUR SERVICE PRINCIPAL ROLE ARN>
aws-region: us-west-2
- run: aws sts get-caller-identity
- name: Set up Python 3.x
uses: actions/setup-python@v1
with:
python-version: '3.10'
- name: Install Outerbounds
run: |
python3 -m pip install --user outerbounds
- name: Test flow
env:
METAFLOW_HOME: /tmp/.metaflowconfig
run: |
<YOUR OB CONFIGURE COMMAND FOR SERVICE PRINCIPALS>
python evaluate_new_model_flow.py run --with card
- name: Checkout repository
uses: actions/checkout@v2

- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: 3.11

- name: Install and Configure Outerbounds
run: |-
pip install -U outerbounds && \
outerbounds service-principal-configure \
--name <YOUR-SERVICE-PRINCIPAL-NAME> \
--deployment-domain <YOUR-OUTERBOUNDS-DEPLOYMENT-NAME> \
--perimeter <YOUR-OUTERBOUNDS-PERIMETER-NAME> \
--github-actions
- name: Run helloworld flow
run: |-
python evaluate_new_model_flow.py run --with card
89 changes: 15 additions & 74 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,81 +1,22 @@
# GitHub Actions on Outerbounds Platform Demo
A basic repo structure to run CI/CD jobs on Outerbounds platform.

## Related resources
[Github Actions x Outerbounds Service Principals](https://docs.google.com/document/d/1If-Nh4EY4cs5wDihWhnDglE-NKqu8Gv0-ZwXcw4cons/edit)
- This document describes how to set up a Github CI job that can run flows using Outerbounds Service Principals.

## Workflows

<img src="./static/lifecycle.png" style="display: block; float: left; max-width: 20%; height: auto; margin: auto; float: none!important;">

[<img src="./static/thumbnail.png" style="display: block; float: left; max-width: 20%; height: auto; margin: auto; float: none!important;">](https://www.youtube.com/watch?v=XnW5MXzMEW8)

### Engineer UX

#### Initial Setup
This is something the person who handles cloud engineering/security will do once.

##### 1. Find your AWS account ID as instructed [here](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.n2f7xpi062t8).
##### 2. Follow this section of the [Allow Github Actions Permissions to Assume your IAM Role](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.5cp00dpcus00) part of the [Service Principals guide](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit).
##### 3. [Create a new Permission Policy for Service Principals](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.p55n5nuncamf).
##### 4. [Create a Service Principal in Outerbounds UI](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.tdalusawlhk1).

#### Modifying or adding a new CI/CD task in your action
This is a pattern that may require the end user who writes code that goes into the tasks of a FlowSpec run to communicate with the person who manages cloud engineering/security. It is the place where the security to run the action using the Outerbounds platform service principal connects to the logic that:
- runs the GitHub action CI job that runs the FlowSpec,
- deploys a new FlowSpec to a production branch or an experimental branch for A/B & multi-armed bandit scenarios. You might use the Metaflow [client API](https://docs.metaflow.org/api/client) to determine when the run has met some criteria.

#### The workflow
##### 1. Identify the GitHub organization containing the repository where you want to add a new GitHub action. For example, this repository would be the “outerbounds” organization and “github-actions-on-obp-demo” repository.
##### 2. Go to step 5 and 6 of the [Create and Configure your IAM Role](https://docs.google.com/document/d/1If-Nh4EY4cs5wDihWhnDglE-NKqu8Gv0-ZwXcw4cons/edit) section, and follow instructions to add the action you want to the trust policy of your service principal IAM role. For example: here we define an action that runs when new code is pushed directly or merged to the main branch of the repository.

<img src="./static/trust-policy-git-action.png" style="display: block; float: left; max-width: 20%; height: auto; margin: auto; float: none!important;">

##### 3. [Define the GitHub action](https://docs.google.com/document/d/1O0ap2_hnz8VHQqIhiCDUruNCFNKxhiwt9JTWePlYAnc/edit#heading=h.shunrk8q1a9d) in your GitHub repository with your FlowSpec code and its dependencies.

There are two key pieces to look at in the example at `.github/workflows/assess_new_production_model.yml` to get this to work for your service principal. Descriptions are annotated inside `<>` in the following snippet. You will find the ARN in your AWS account in the IAM Role for the service pricipal, and you will find your Outerbounds configure command in the Outerbounds platform UI where you have connected the service principal to your account as a machine identity.
```
name: Deploy new production model
...
jobs:
deploy:
...
steps:
- uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: <YOUR SERVICE PRINCIPAL ROLE ARN>
...
- name: Test flow
...
run: |
<YOUR OB CONFIGURE COMMAND FOR SERVICE PRINCIPALS>
python evaluate_new_model_flow.py run --with card
```


### Data Scientist UX
Our goal is to update the model used in the `Predict` workflow defined in `prediction_flow.py`. As a starting point for the CI/CD lifecycle, consider how a data scientist iterates locally or on a cloud workstation.

This repository demonstrates how the data scientist can:
- take the result of such experimentation,
### Create a machine user in Outerbounds UI
- In the Outerbounds UI, go to the `Admin` panel on the left side navigation and select `Users`.
- Under `Machines` click the `Create New` button.
- Fill out the form, choosing the desired GitHub Actions form, and filling in the desired GitHub organization and repository.
- After submitting, click the row for the Machine User you created, and a code snippet will appear.
- Paste the command in actions file in `.github/workflows/` and modify it to run Metaflow code in the repository.

### Write flows and run them on desired GitHub Actions
Our goal is to update the model used in the `Predict` workflow defined in `prediction_flow.py`. As a starting point for the CI/CD lifecycle, consider how a data scientist iterates. This repository demonstrates how to take the result of experimental, interactive development and use it to:
- create a GitHub branch,
- let an automatic CI/CD process built with GitHub Actions validate the model's quality (using Outerbounds platform resources),
- and only if the new model code meets certain user-defined criteria, automatically deploy the newly trained model to be used in the production workflow that makes predictions accessed by other production applications.
- let an automatic CI/CD process built with GitHub Actions validate the model's quality (using Outerbounds platform resources), and
- if the new model code meets certain user-defined criteria, automatically deploy the newly trained model to be used in the production workflow that makes predictions accessed by other production applications.


### Deploy the `Predict` workflow to production
A data scientist or ML engineer would do this rarely, and typically less frequently than the model selction/architecture in `my_data_science_module.py` updates.
A data scientist or ML engineer would do this rarely, and typically less frequently than the model selection/architecture in `my_data_science_module.py` updates.
This only needs to be done if the code in `predict_flow.py` file updates.
```
python predict_flow.py --production argo-workflows create
Expand All @@ -87,13 +28,13 @@ This is a way to manually trigger a refresh of the production run that populates
python predict_flow.py --production argo-workflows trigger
```

### Development phase: Local iteration on `EvaluateNewModel`
### Local iteration on `EvaluateNewModel`
Local/workstation testing:
```
python evaluate_new_model_flow.py run
```

### Moving to production phase: a template for a CI/CD process using GitHub Actions
### CI/CD process using GitHub Actions
When a data scientist is satisfied with what they see on local runs, then they can use GitHub commands like a regular software development workflow:
```
git switch -c 'my-new-model-branch'
Expand All @@ -107,4 +48,4 @@ After the model is pushed to the remote branch of `my-new-model-branch`, the dat
The GitHub Action in this template will do the following:
1. Run the `EvaluateNewModel` workflow defined in `evaluate_new_model_flow.py`.
2. If the `EvaluateNewModel` workflow produces a model that meets some user-defined criteria (e.g., beyond some performance metric threshold), then tag the Metaflow run in which the model was trained as a `deployment_candidate`.
3. If the upstream `EvaluateNewModel` run is tagged as a `deployment_candidate` and the model meets any other criteria you add to this template, then the production workflow will use a new version of the model in the `predict.py` flow in an on-going fashion.
3. If the upstream `EvaluateNewModel` run is tagged as a `deployment_candidate` and the model meets any other criteria you add to this template, then the production workflow will use a new version of the model in the `predict.py` flow in an ongoing fashion.

0 comments on commit 3adce7d

Please sign in to comment.