Skip to content

Commit

Permalink
remove some setup files
Browse files Browse the repository at this point in the history
changed an auther to myself
  • Loading branch information
ezehlivinus committed Sep 1, 2022
1 parent ad46baa commit 3f18b1e
Show file tree
Hide file tree
Showing 20 changed files with 882 additions and 43 deletions.
1 change: 1 addition & 0 deletions <?xml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: kubernetes-release/release/&lt;</Details></Error>
30 changes: 30 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
FROM python:3.7.3-stretch

## Step 1:
# Create a working directory
WORKDIR /app

## Step 2:
# Copy source code to working directory
COPY . app.py /app/
# COPY ./model_data /app/model_data

## Step 3:
# Install packages from requirements.txt
# hadolint ignore=DL3013

RUN pip install --upgrade pip &&\
ls &&\
pip install --trusted-host pypi.python.org -r requirements.txt

## Step 4:
# Expose port 80
EXPOSE 80

## Step 5:
# Run app.py at container launch
CMD ["python", "app.py"]




33 changes: 33 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
## The Makefile includes instructions on environment setup and lint tests
# Create and activate a virtual environment
# Install dependencies in requirements.txt
# Dockerfile should pass hadolint
# app.py should pass pylint
# (Optional) Build a simple integration test

setup:
# Create python virtualenv & source it
# source ~/.devops/bin/activate
python3 -m venv ~/.devops

install:
# This should be run from inside a virtualenv
pip install --upgrade pip &&\
pip install -r requirements.txt
sudo wget -O /bin/hadolint https://github.com/hadolint/hadolint/releases/download/v1.16.3/hadolint-Linux-x86_64
sudo chmod +x /bin/hadolint

test:
# Additional, optional, tests could go here
#python -m pytest -vv --cov=myrepolib tests/*.py
#python -m pytest --nbval notebook.ipynb

lint:
# See local hadolint install instructions: https://github.com/hadolint/hadolint
# This is linter for Dockerfiles
hadolint Dockerfile
# This is a linter for Python source code linter: https://www.pylint.org/
# This should be run from inside a virtualenv
pylint --disable=R,C,W1203,W1202 app.py

all: install lint test
50 changes: 50 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
<include a CircleCI status badge, here>

## Project Overview

In this project, you will apply the skills you have acquired in this course to operationalize a Machine Learning Microservice API.

You are given a pre-trained, `sklearn` model that has been trained to predict housing prices in Boston according to several features, such as average rooms in a home and data about highway access, teacher-to-pupil ratios, and so on. You can read more about the data, which was initially taken from Kaggle, on [the data source site](https://www.kaggle.com/c/boston-housing). This project tests your ability to operationalize a Python flask app—in a provided file, `app.py`—that serves out predictions (inference) about housing prices through API calls. This project could be extended to any pre-trained machine learning model, such as those for image recognition and data labeling.

### Project Tasks

Your project goal is to operationalize this working, machine learning microservice using [kubernetes](https://kubernetes.io/), which is an open-source system for automating the management of containerized applications. In this project you will:
* Test your project code using linting
* Complete a Dockerfile to containerize this application
* Deploy your containerized application using Docker and make a prediction
* Improve the log statements in the source code for this application
* Configure Kubernetes and create a Kubernetes cluster
* Deploy a container using Kubernetes and make a prediction
* Upload a complete Github repo with CircleCI to indicate that your code has been tested

You can find a detailed [project rubric, here](https://review.udacity.com/#!/rubrics/2576/view).

**The final implementation of the project will showcase your abilities to operationalize production microservices.**

---

## Setup the Environment

* Create a virtualenv with Python 3.7 and activate it. Refer to this link for help on specifying the Python version in the virtualenv.
```bash
python3 -m pip install --user virtualenv
# You should have Python 3.7 available in your host.
# Check the Python path using `which python3`
# Use a command similar to this one:
python3 -m virtualenv --python=<path-to-Python3.7> .devops
source .devops/bin/activate
```
* Run `make install` to install the necessary dependencies

### Running `app.py`

1. Standalone: `python app.py`
2. Run in Docker: `./run_docker.sh`
3. Run in Kubernetes: `./run_kubernetes.sh`

### Kubernetes Steps

* Setup and Configure Docker locally
* Setup and Configure Kubernetes locally
* Create Flask app in Container
* Run via kubectl
73 changes: 73 additions & 0 deletions app.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
from flask import Flask, request, jsonify
from flask.logging import create_logger
import logging

import pandas as pd
from sklearn.externals import joblib
from sklearn.preprocessing import StandardScaler

app = Flask(__name__)
LOG = create_logger(app)
LOG.setLevel(logging.INFO)

def scale(payload):
"""Scales Payload"""

LOG.info(f"Scaling Payload: \n{payload}")
scaler = StandardScaler().fit(payload.astype(float))
scaled_adhoc_predict = scaler.transform(payload.astype(float))
return scaled_adhoc_predict

@app.route("/")
def home():
html = f"<h3>Sklearn Prediction Home</h3>"
return html.format(format)

@app.route("/predict", methods=['POST'])
def predict():
"""Performs an sklearn prediction
input looks like:
{
"CHAS":{
"0":0
},
"RM":{
"0":6.575
},
"TAX":{
"0":296.0
},
"PTRATIO":{
"0":15.3
},
"B":{
"0":396.9
},
"LSTAT":{
"0":4.98
}
result looks like:
{ "prediction": [ <val> ] }
"""

# Logging the input payload
json_payload = request.json
LOG.info(f"JSON payload: \n{json_payload}")
inference_payload = pd.DataFrame(json_payload)
LOG.info(f"Inference payload DataFrame: \n{inference_payload}")
# scale the input
scaled_payload = scale(inference_payload)
# get an output prediction from the pretrained model, clf
prediction = list(clf.predict(scaled_payload))
# TO DO: Log the output prediction value
LOG.info(f'Prediction values: \n {prediction}')

return jsonify({'prediction': prediction})

if __name__ == "__main__":
# load pretrained model as clf
clf = joblib.load("./model_data/boston_housing_prediction.joblib")
app.run(host='0.0.0.0', port=80, debug=True) # specify port=80
43 changes: 0 additions & 43 deletions config.yml

This file was deleted.

11 changes: 11 additions & 0 deletions docker_out.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[2022-09-01 18:49:06,364] INFO in app: JSON payload:
{'CHAS': {'0': 0}, 'RM': {'0': 6.575}, 'TAX': {'0': 296.0}, 'PTRATIO': {'0': 15.3}, 'B': {'0': 396.9}, 'LSTAT': {'0': 4.98}}
[2022-09-01 18:49:06,385] INFO in app: Inference payload DataFrame:
CHAS RM TAX PTRATIO B LSTAT
0 0 6.575 296.0 15.3 396.9 4.98
[2022-09-01 18:49:06,397] INFO in app: Scaling Payload:
CHAS RM TAX PTRATIO B LSTAT
0 0 6.575 296.0 15.3 396.9 4.98
[2022-09-01 18:49:06,401] INFO in app: Prediction values:
[20.35373177134412]
172.17.0.1 - - [01/Sep/2022 18:49:06] "POST /predict HTTP/1.1" 200 -
16 changes: 16 additions & 0 deletions kubernetes.out.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
^C(.devops) voclabs:~/environment/project-ml-microservice-kubernetes $ bash run_kubernes.sh
Error from server (AlreadyExists): pods "udapeople" already exists
NAME READY STATUS RESTARTS AGE
udapeople 1/1 Running 2 (44s ago) 3m34s
Forwarding from 127.0.0.1:8000 -> 80
Forwarding from [::1]:8000 -> 80
Handling connection for 8000


(.devops) voclabs:~/environment/project-ml-microservice-kubernetes $ bash make_prediction.sh Port: 8000
{
"prediction": [
20.35373177134412
]
}

28 changes: 28 additions & 0 deletions make_prediction.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/usr/bin/env bash

PORT=8000
echo "Port: $PORT"

# POST method predict
curl -d '{
"CHAS":{
"0":0
},
"RM":{
"0":6.575
},
"TAX":{
"0":296.0
},
"PTRATIO":{
"0":15.3
},
"B":{
"0":396.9
},
"LSTAT":{
"0":4.98
}
}'\
-H "Content-Type: application/json" \
-X POST http://localhost:$PORT/predict
Binary file added model_data/boston_housing_prediction.joblib
Binary file not shown.
Loading

0 comments on commit 3f18b1e

Please sign in to comment.