Garik_Hakobyan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Building basic functions with numpy ##\n",
    "\n",
    "Numpy is the main package for scientific computing in Python. It is maintained by a large community (www.numpy.org). In this exercise you will learn several key numpy functions such as np.exp, np.log, and np.reshape. You will need to know how to use these functions for future assignments.\n",
    "\n",
    "### 1.1 - sigmoid function, np.exp() ###\n",
    "\n",
    "Before using np.exp(), you will use math.exp() to implement the sigmoid function. You will then see why np.exp() is preferable to math.exp().\n",
    "\n",
    "**Exercise**: Build a function that returns the sigmoid of a real number x. Use math.exp(x) for the exponential function.\n",
    "\n",
    "**Reminder**:\n",
    "$sigmoid(x) = \\frac{1}{1+e^{-x}}$ is sometimes also known as the logistic function. It is a non-linear function used not only in Machine Learning (Logistic Regression), but also in Deep Learning.\n",
    "\n",
    "<img src=\"Sigmoid.png\" style=\"width:500px;height:228px;\">\n",
    "\n",
    "To refer to a function belonging to a specific package you could call it using package_name.function(). Run the code below to see an example with math.exp()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math\n",
    "\n",
    "def basic_sigmoid(x):\n",
    "    \"\"\"\n",
    "    Compute sigmoid of x.\n",
    "\n",
    "    Arguments:\n",
    "    x -- A scalar\n",
    "\n",
    "    Return:\n",
    "    s -- sigmoid(x)\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "    s = 1 / (1 + math.exp(-x))\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "basic_sigmoid(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "<table style = \"width:40%\">\n",
    "    <tr>\n",
    "    <td>** basic_sigmoid(3) **</td> \n",
    "        <td>0.9525741268224334 </td> \n",
    "    </tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, we rarely use the \"math\" library in deep learning because the inputs of the functions are real numbers. In deep learning we mostly use matrices and vectors. This is why numpy is more useful. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### One reason why we use \"numpy\" instead of \"math\" in Deep Learning ###\n",
    "x = [1, 2, 3]\n",
    "basic_sigmoid(x) # you will see this give an error when you run it, because x is a vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fact, if $ x = (x_1, x_2, ..., x_n)$ is a row vector then $np.exp(x)$ will apply the exponential function to every element of x. The output will thus be: $np.exp(x) = (e^{x_1}, e^{x_2}, ..., e^{x_n})$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# example of np.exp\n",
    "x = np.array([1, 2, 3])\n",
    "print(np.exp(x)) # result is (exp(1), exp(2), exp(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, if x is a vector, then a Python operation such as $s = x + 3$ or $s = \\frac{1}{x}$ will output s as a vector of the same size as x."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# example of vector operation\n",
    "x = np.array([1, 2, 3])\n",
    "print (x + 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Any time you need more info on a numpy function, we encourage you to look at [the official documentation](https://docs.scipy.org/doc/numpy-1.10.1/reference/generated/numpy.exp.html). \n",
    "\n",
    "You can also create a new cell in the notebook and write `np.exp?` (for example) to get quick access to the documentation.\n",
    "\n",
    "**Exercise**: Implement the sigmoid function using numpy. \n",
    "\n",
    "**Instructions**: x could now be either a real number, a vector, or a matrix. The data structures we use in numpy to represent these shapes (vectors, matrices...) are called numpy arrays. You don't need to know more for now.\n",
    "$$ \\text{For } x \\in \\mathbb{R}^n \\text{,     } sigmoid(x) = sigmoid\\begin{pmatrix}\n",
    "    x_1  \\\\\n",
    "    x_2  \\\\\n",
    "    ...  \\\\\n",
    "    x_n  \\\\\n",
    "\\end{pmatrix} = \\begin{pmatrix}\n",
    "    \\frac{1}{1+e^{-x_1}}  \\\\\n",
    "    \\frac{1}{1+e^{-x_2}}  \\\\\n",
    "    ...  \\\\\n",
    "    \\frac{1}{1+e^{-x_n}}  \\\\\n",
    "\\end{pmatrix}\\tag{1} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np # this means you can access numpy functions by writing np.function() instead of numpy.function()\n",
    "\n",
    "def sigmoid(x):\n",
    "    \"\"\"\n",
    "    Compute the sigmoid of x\n",
    "\n",
    "    Arguments:\n",
    "    x -- A scalar or numpy array of any size\n",
    "\n",
    "    Return:\n",
    "    s -- sigmoid(x)\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "    s = 1 / (1 + np.exp(-x))\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "sigmoid(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "<table>\n",
    "    <tr> \n",
    "        <td> **sigmoid([1,2,3])**</td> \n",
    "        <td> array([ 0.73105858,  0.88079708,  0.95257413]) </td> \n",
    "    </tr>\n",
    "</table> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 - Sigmoid gradient\n",
    "\n",
    "As you've seen in lecture, you will need to compute gradients to optimize loss functions using backpropagation. Let's code your first gradient function.\n",
    "\n",
    "**Exercise**: Implement the function sigmoid_grad() to compute the gradient of the sigmoid function with respect to its input x. The formula is: $$sigmoid\\_derivative(x) = \\sigma'(x) = \\sigma(x) (1 - \\sigma(x))\\tag{2}$$\n",
    "You often code this function in two steps:\n",
    "1. Set s to be the sigmoid of x. You might find your sigmoid(x) function useful.\n",
    "2. Compute $\\sigma'(x) = s(1-s)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid_derivative(x):\n",
    "    \"\"\"\n",
    "    Compute the gradient (also called the slope or derivative) of the sigmoid function with respect to its input x.\n",
    "    You can store the output of the sigmoid function into variables and then use it to calculate the gradient.\n",
    "    \n",
    "    Arguments:\n",
    "    x -- A scalar or numpy array\n",
    "\n",
    "    Return:\n",
    "    ds -- Your computed gradient.\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 2 lines of code)\n",
    "    s = 1 / (1 + np.exp(-x))\n",
    "    ds = s * (1 - s)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([1, 2, 3])\n",
    "print (\"sigmoid_derivative(x) = \" + str(sigmoid_derivative(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "\n",
    "\n",
    "<table>\n",
    "    <tr> \n",
    "        <td> **sigmoid_derivative([1,2,3])**</td> \n",
    "        <td> [ 0.19661193  0.10499359  0.04517666] </td> \n",
    "    </tr>\n",
    "</table> \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 - Reshaping arrays ###\n",
    "\n",
    "Two common numpy functions used in deep learning are [np.shape](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.shape.html) and [np.reshape()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.reshape.html). \n",
    "- X.shape is used to get the shape (dimension) of a matrix/vector X. \n",
    "- X.reshape(...) is used to reshape X into some other dimension. \n",
    "\n",
    "For example, in computer science, an image is represented by a 3D array of shape $(length, height, depth = 3)$. However, when you read an image as the input of an algorithm you convert it to a vector of shape $(length*height*3, 1)$. In other words, you \"unroll\", or reshape, the 3D array into a 1D vector.\n",
    "\n",
    "<img src=\"image2vector.png\" style=\"width:500px;height:300;\">\n",
    "\n",
    "**Exercise**: Implement `image2vector()` that takes an input of shape (length, height, 3) and returns a vector of shape (length\\*height\\*3, 1). For example, if you would like to reshape an array v of shape (a, b, c) into a vector of shape (a*b,c) you would do:\n",
    "``` python\n",
    "v = v.reshape((v.shape[0]*v.shape[1], v.shape[2])) # v.shape[0] = a ; v.shape[1] = b ; v.shape[2] = c\n",
    "```\n",
    "- Please don't hardcode the dimensions of image as a constant. Instead look up the quantities you need with `image.shape[0]`, etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GRADED FUNCTION: image2vector\n",
    "def image2vector(image):\n",
    "    \"\"\"\n",
    "    Argument:\n",
    "    image -- a numpy array of shape (length, height, depth)\n",
    "    \n",
    "    Returns:\n",
    "    v -- a vector of shape (length*height*depth, 1)\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 1 line of code)\n",
    "    v = np.reshape(image, (image.shape[0] * image.shape[1] * image.shape[2],1))\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a 3 by 3 by 2 array, typically images will be (num_px_x, num_px_y,3) where 3 represents the RGB values\n",
    "image = np.array([[[ 0.67826139,  0.29380381],\n",
    "        [ 0.90714982,  0.52835647],\n",
    "        [ 0.4215251 ,  0.45017551]],\n",
    "\n",
    "       [[ 0.92814219,  0.96677647],\n",
    "        [ 0.85304703,  0.52351845],\n",
    "        [ 0.19981397,  0.27417313]],\n",
    "\n",
    "       [[ 0.60659855,  0.00533165],\n",
    "        [ 0.10820313,  0.49978937],\n",
    "        [ 0.34144279,  0.94630077]]])\n",
    "\n",
    "print (\"image2vector(image) = \" + str(image2vector(image)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "\n",
    "\n",
    "<table style=\"width:100%\">\n",
    "     <tr> \n",
    "       <td> **image2vector(image)** </td> \n",
    "       <td> [[ 0.67826139]\n",
    " [ 0.29380381]\n",
    " [ 0.90714982]\n",
    " [ 0.52835647]\n",
    " [ 0.4215251 ]\n",
    " [ 0.45017551]\n",
    " [ 0.92814219]\n",
    " [ 0.96677647]\n",
    " [ 0.85304703]\n",
    " [ 0.52351845]\n",
    " [ 0.19981397]\n",
    " [ 0.27417313]\n",
    " [ 0.60659855]\n",
    " [ 0.00533165]\n",
    " [ 0.10820313]\n",
    " [ 0.49978937]\n",
    " [ 0.34144279]\n",
    " [ 0.94630077]]</td> \n",
    "     </tr>\n",
    "    \n",
    "   \n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 - Normalizing rows\n",
    "\n",
    "Another common technique we use in Machine Learning and Deep Learning is to normalize our data. It often leads to a better performance because gradient descent converges faster after normalization. Here, by normalization we mean changing x to $ \\frac{x}{\\| x\\|} $ (dividing each row vector of x by its norm).\n",
    "\n",
    "For example, if $$x = \n",
    "\\begin{bmatrix}\n",
    "    0 & 3 & 4 \\\\\n",
    "    2 & 6 & 4 \\\\\n",
    "\\end{bmatrix}\\tag{3}$$ then $$\\| x\\| = np.linalg.norm(x, axis = 1, keepdims = True) = \\begin{bmatrix}\n",
    "    5 \\\\\n",
    "    \\sqrt{56} \\\\\n",
    "\\end{bmatrix}\\tag{4} $$and        $$ x\\_normalized = \\frac{x}{\\| x\\|} = \\begin{bmatrix}\n",
    "    0 & \\frac{3}{5} & \\frac{4}{5} \\\\\n",
    "    \\frac{2}{\\sqrt{56}} & \\frac{6}{\\sqrt{56}} & \\frac{4}{\\sqrt{56}} \\\\\n",
    "\\end{bmatrix}\\tag{5}$$ Note that you can divide matrices of different sizes and it works fine: this is called broadcasting and you're going to learn about it in part 5.\n",
    "\n",
    "\n",
    "**Exercise**: Implement normalizeRows() to normalize the rows of a matrix. After applying this function to an input matrix x, each row of x should be a vector of unit length (meaning length 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalizeRows(x):\n",
    "    \"\"\"\n",
    "    Implement a function that normalizes each row of the matrix x (to have unit length).\n",
    "    \n",
    "    Argument:\n",
    "    x -- A numpy matrix of shape (n, m)\n",
    "    \n",
    "    Returns:\n",
    "    x -- The normalized (by row) numpy matrix. You are allowed to modify x.\n",
    "    \"\"\"\n",
    "    \n",
    "    ### START CODE HERE ### (≈ 2 lines of code)\n",
    "    # Compute x_norm as the norm 2 of x. Use np.linalg.norm(..., ord = 2, axis = ..., keepdims = True)\n",
    "    x_norm = np.linalg.norm(x, axis=1, keepdims = True)\n",
    "    \n",
    "    # Divide x by its norm.\n",
    "    x = x / x_norm\n",
    "    ### END CODE HERE ###\n",
    "\n",
    "    return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.array([\n",
    "    [0, 3, 4],\n",
    "    [1, 6, 4]])\n",
    "print(\"normalizeRows(x) = \" + str(normalizeRows(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Expected Output**: \n",
    "\n",
    "<table style=\"width:60%\">\n",
    "    <tr> \n",
    "        <td> **normalizeRows(x)** </td>\n",
    "        <td> [[ 0.          0.6         0.8]\n",
    " [ 0.13736056  0.82416338  0.54944226]]</td> \n",
    "    </tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Implement Logistic Regression\n",
    "\n",
    "1. Define a **log_reg function** with arguments X, y and alpha (features, target and the learning rate of the gradient descent):\n",
    "\n",
    "- this function should implement the **gradient descent** based on X and y and the logistic cost function that you will specify. \n",
    "\n",
    "*Remember that through forward and backward propagation of the computational graph, the gradient descent should update the weights of the model. You can implement this either by loops or by vectorization (during the next class we will cover the vectorized implementation of this).\n",
    "\n",
    "2. Define a **RandomGridSearch** function that takes as input arguments the estimator (an example could be log_reg()), the number of cross validation folds, and the possible values of the hyperparameters, in this case it is the learning_rate of the gradient descent in a list format (example alpha_grid = [0.0001, 0.001, 0.01, 0.1, 1, 2, 5]). \n",
    "\n",
    "- the final form could be as follows: \n",
    "    RandomGridSearch(log_reg(), X, y, alpha, cv = 3, alpha_grid = [0.0001, 0.001, 0.01, 0.1, 1, 2, 5]).\n",
    "    \n",
    "- this function should randomly select several hyperparameter values from alpha_grid and perform cross validations for each of those values. For each alpha_grid value this function should get a list of cross validation scores. \n",
    "- use **ROC AUC score** as a score metric, you can find roc_auc_score function in scikit learn's metrics module useful for this task. \n",
    "\n",
    "*Note that the number of scores will depend on the value of \"cv\" you specify in the function. At the end, this function should return the best hyperparameter value based on the best average cross validation score.\n",
    "Using this hyperparameter value you can train your final model on the whole training set by using log_reg function with X, y, and alpha = best hyperparameter from RandomGridSearch.\n",
    "\n",
    "3. Using all of your defined functions train a logistic regression model on **Santander Customer Satisfaction** problem that you can find on Kaggle. \n",
    "- You can take several features of your choice from the dataset for training the model (for example up to 5-7 features). The target is binary. \n",
    "- Be careful with the target, **if it is imbalanced** you can use oversampling, undersampling, class weighting in cost function or any other method of dealing with the imbalanced target that you may find useful (you can do your research here). \n",
    "- After training the model, report the main performance metrics of your model: **ROC AUC score, precision and recall, F1 score**, as well as the **ROC curve and confusion matrix**.\n",
    "- Write a short description about the trained model and its performance in a way you see it. Describe the features you have used, which of them are the most important in terms of predicting the target and interpret in your words the performance metrics of the model. Use 5-7 sentences in maximum.\n",
    "\n",
    "*Note: You are free to use your preferred coding design with your preferred data structures for this assignment. The most important thing is to get the correct results. We will check it in the future by using the scikit learns logistic function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "import kaggle\n",
    "import os\n",
    "import zipfile\n",
    "\n",
    "from sklearn.preprocessing import StandardScaler\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extractind_data(file_name,file_path):\n",
    "   \n",
    "    !kaggle competitions download -c santander-customer-satisfaction -f $file_name -p $file_path --force\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading train.csv.zip to ..\\data\\Santander_Customer_Satisfaction\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0.00/4.05M [00:00<?, ?B/s]\n",
      " 25%|##4       | 1.00M/4.05M [00:04<00:14, 228kB/s]\n",
      " 49%|####9     | 2.00M/4.05M [00:06<00:07, 276kB/s]\n",
      " 74%|#######4  | 3.00M/4.05M [00:09<00:03, 308kB/s]\n",
      " 99%|#########8| 4.00M/4.05M [00:11<00:00, 332kB/s]\n",
      "100%|##########| 4.05M/4.05M [00:11<00:00, 364kB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading test.csv.zip to ..\\data\\Santander_Customer_Satisfaction\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "  0%|          | 0.00/4.02M [00:00<?, ?B/s]\n",
      " 25%|##4       | 1.00M/4.02M [00:02<00:08, 365kB/s]\n",
      " 50%|####9     | 2.00M/4.02M [00:05<00:05, 384kB/s]\n",
      " 75%|#######4  | 3.00M/4.02M [00:07<00:02, 422kB/s]\n",
      " 99%|#########9| 4.00M/4.02M [00:08<00:00, 464kB/s]\n",
      "100%|##########| 4.02M/4.02M [00:08<00:00, 472kB/s]\n"
     ]
    }
   ],
   "source": [
    "train_file_name = 'train.csv'\n",
    "test_file_name = 'test.csv'\n",
    "raw_data_path = os.path.join(os.path.pardir, 'data', 'Santander_Customer_Satisfaction')\n",
    "\n",
    "extractind_data(train_file_name, raw_data_path)\n",
    "extractind_data(test_file_name, raw_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will unzip the files so that you can see them..\n",
    "Dataset = \"train.csv\"\n",
    "\n",
    "with zipfile.ZipFile(raw_data_path +'\\\\' + Dataset +\".zip\",\"r\") as z:\n",
    "    z.extractall(raw_data_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(raw_data_path+'\\\\train.csv')\n",
    "columns = ['var3','var15','saldo_medio_var44_hace3','saldo_medio_var44_ult1','saldo_medio_var44_ult3','var38']\n",
    "\n",
    "p0 = data[data['TARGET']==0].head(3000)\n",
    "p1 = data[data['TARGET']==1]\n",
    "df = pd.concat([p0,p1])\n",
    "\n",
    "X = df[columns].values\n",
    "y = df['TARGET'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "scale = StandardScaler()\n",
    "x_scaled = scale.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "class my_logit_regressor():\n",
    "    \n",
    "    def __init__(self, n_iter = 500, eta = 0.001):\n",
    "        \n",
    "        self.n_iter =n_iter\n",
    "#         self.eta = eta\n",
    "        \n",
    "    def fit(self, X, y, eta):\n",
    "        self.theta = np.zeros(1 + (X.shape[1]))\n",
    "        self.loss = []\n",
    "        \n",
    "        for i in range(self.n_iter):\n",
    "            \n",
    "            zeta = self.net_input(X)\n",
    "            pred = self.sigmoid(zeta)\n",
    "            error = y - pred \n",
    "            self.theta[1:] += eta * X.T.dot(error) \n",
    "            self.theta[0] = eta * error.sum()\n",
    "#             loss = self.cost(pred,y)\n",
    "        \n",
    "#             self.loss.append(loss)\n",
    "#         return self.loss   \n",
    "        \n",
    "    def net_input(self, X):\n",
    "        return np.dot(X, self.theta[1:]) + self.theta[0]\n",
    "        \n",
    "    def sigmoid(self, zeta):        \n",
    "        return 1/(1 + np.exp(-np.clip(zeta, -100,100)))\n",
    "    \n",
    "    def predict(self, X):\n",
    "        return np.where(self.net_input(X) > 0, 1, 0)\n",
    "   \n",
    "    def cost(self, z, y):\n",
    "        return -(np.dot(y,np.log(z)) +  np.dot(1 - y, np.log(1-z)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementing RandomGridSearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import f1_score\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import recall_score\n",
    "from sklearn.metrics import precision_score\n",
    "from sklearn.metrics import roc_curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "class myRandomGridSearch:\n",
    "    def __init__(self, estimator, alpha, cv, n_iter = 100):\n",
    "        self.estimator = estimator\n",
    "        self.cv = cv\n",
    "        self.alpha = alpha\n",
    "        self.n_iter = n_iter\n",
    "        \n",
    "    def fit(self, X, y):\n",
    "        \n",
    "        for alpha in self.alpha:\n",
    "           \n",
    "            split = StratifiedShuffleSplit(n_splits = self.cv, test_size=0.3,  random_state= 42)\n",
    "            split.get_n_splits(X, y)\n",
    "            \n",
    "            for train_index, test_index in split.split(X,y):\n",
    "                \n",
    "                X_train, X_test = X[train_index], X[test_index]\n",
    "                y_train, y_test = y[train_index], y[test_index]\n",
    "                \n",
    "                \n",
    "                self.estimator.fit(X_train, y_train, alpha)\n",
    "                \n",
    "                \n",
    "                pred =  self.estimator.predict(X_test)\n",
    "                \n",
    "            self.modelReport(y_test, pred, alpha)\n",
    "\n",
    "                \n",
    "    def modelReport(self, y, pred, alpha):\n",
    "        \n",
    "        print('Model Learning Rate : ', alpha , \n",
    "              '\\nF1 Score:          ', f1_score(y, pred, labels=np.unique(pred)),\n",
    "              '\\nROC Score:         ', roc_auc_score(y, pred),\n",
    "              '\\nRecall Score:      ', recall_score(y, pred),\n",
    "              '\\nPrecision Score:   ', precision_score(y, pred, zero_division=1),\n",
    "              '\\nConfusion Matrix:\\n', confusion_matrix(y, pred),\n",
    "              '\\n\\n\\n***************************************\\n\\n\\n')\n",
    "#         print('Confusion Matrix:', roc_curve(y, pred))\n",
    "\n",
    "           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Learning Rate :  1e-05 \n",
      "F1 Score:           0.603003003003003 \n",
      "ROC Score:          0.6335179032853452 \n",
      "Recall Score:       0.5559246954595792 \n",
      "Precision Score:    0.6587926509186351 \n",
      "Confusion Matrix:\n",
      " [[640 260]\n",
      " [401 502]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  0.0001 \n",
      "F1 Score:           0.6071856287425149 \n",
      "ROC Score:          0.6362864525655223 \n",
      "Recall Score:       0.5614617940199336 \n",
      "Precision Score:    0.6610169491525424 \n",
      "Confusion Matrix:\n",
      " [[640 260]\n",
      " [396 507]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  0.001 \n",
      "F1 Score:           0.608955223880597 \n",
      "ROC Score:          0.6368364710225176 \n",
      "Recall Score:       0.5647840531561462 \n",
      "Precision Score:    0.6606217616580311 \n",
      "Confusion Matrix:\n",
      " [[638 262]\n",
      " [393 510]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  0.01 \n",
      "F1 Score:           0.6724267468069121 \n",
      "ROC Score:          0.5155703211517165 \n",
      "Recall Score:       0.991140642303433 \n",
      "Precision Score:    0.5088118249005117 \n",
      "Confusion Matrix:\n",
      " [[ 36 864]\n",
      " [  8 895]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  0.1 \n",
      "F1 Score:           0.6749622926093516 \n",
      "ROC Score:          0.5211258767072721 \n",
      "Recall Score:       0.991140642303433 \n",
      "Precision Score:    0.5117209834190967 \n",
      "Confusion Matrix:\n",
      " [[ 46 854]\n",
      " [  8 895]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  1 \n",
      "F1 Score:           0.6744624669935873 \n",
      "ROC Score:          0.5205721668512366 \n",
      "Recall Score:       0.9900332225913622 \n",
      "Precision Score:    0.511441647597254 \n",
      "Confusion Matrix:\n",
      " [[ 46 854]\n",
      " [  9 894]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n",
      "Model Learning Rate :  5 \n",
      "F1 Score:           0.6744624669935873 \n",
      "ROC Score:          0.5205721668512366 \n",
      "Recall Score:       0.9900332225913622 \n",
      "Precision Score:    0.511441647597254 \n",
      "Confusion Matrix:\n",
      " [[ 46 854]\n",
      " [  9 894]] \n",
      "\n",
      "\n",
      "***************************************\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "logit_model = my_logit_regressor()\n",
    "rg = myRandomGridSearch(estimator = logit_model,  cv=5, alpha = [0.00001, 0.0001, 0.001, 0.01, 0.1, 1, 5])\n",
    "rg.fit(x_scaled ,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Six featurs are used in this model taken from Santander_Customer_Satisfaction data. \n",
    "implementing grid search, we can see that in balanced data, we can get F1 score 0.60 when alpha is equal 0.001, but when alpha is becoming bigger the model does not yield accurate scores, it overfits. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing with the Sklearn's results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(x_scaled, y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight='balanced', dual=False,\n",
       "                   fit_intercept=True, intercept_scaling=1, l1_ratio=None,\n",
       "                   max_iter=100, multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit = LogisticRegression(class_weight='balanced')\n",
    "logit.fit(X_train ,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = logit.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "F1 Score:           0.6216657593903103 \n",
      "ROC Score:          0.6476385252498498 \n",
      "Recall Score:       0.5991605456453305 \n",
      "Precision Score:    0.6459276018099548 \n",
      "Confusion Matrix:\n",
      " [[717 313]\n",
      " [382 571]]\n"
     ]
    }
   ],
   "source": [
    "print(    '\\nF1 Score:          ', f1_score(y_test, pred),\n",
    "          '\\nROC Score:         ', roc_auc_score(y_test, pred),\n",
    "          '\\nRecall Score:      ', recall_score(y_test, pred),\n",
    "          '\\nPrecision Score:   ', precision_score(y_test, pred),\n",
    "          '\\nConfusion Matrix:\\n', confusion_matrix(y_test, pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ROC for Logistic regression model\n",
    "y_pred_grd = logit.predict_proba(X_test)[:, 1]\n",
    "fpr_grd, tpr_grd, _ = roc_curve(y_test, pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdeVhU1RvA8e8RFDfEfcV9A0REQM19zS33pVzLIhWXrCxTs1z7mbumYWZlmmZWmEtpaauaVia474gLoAiobCLrnN8fg4QIOirDsLyf5+GRmXvm3ndmcN457z33HKW1RgghRP5VwNIBCCGEsCxJBEIIkc9JIhBCiHxOEoEQQuRzkgiEECKfk0QghBD5nCQCIYTI5yQRiDxHKXVJKXVHKRWjlApRSq1VShVP16aFUuo3pVS0UipSKfW9UsopXZsSSqllSqkrKfvyT7ldNnufkRDmJYlA5FU9tdbFAVegMTD17galVHNgN7ANqAzUBI4C+5VStVLaFAJ+BRoAXYESQAvgBtDUXEErpazNtW8hMiOJQORpWusQYBfGhHDXAuALrfUHWutorfVNrfU7wN/AzJQ2zwPVgL5a61Naa4PWOlRrPUdrvTOjYymlGiilflZK3VRKXVdKvZ1y/1ql1Htp2rVTSgWluX1JKTVZKXUMuK2Uekcp5ZNu3x8opZan/G6nlPpMKXVNKRWslHpPKWX1hC+VyMckEYg8TSllD3QD/FNuF8X4zf7bDJp/Azyd8nsn4CetdYyJx7EFfgF+wtjLqIOxR2GqwcAzQElgPdBdKVUiZd9WwLPAxpS264CklGM0BjoDLz/CsYS4hyQCkVdtVUpFA4FAKDAj5f7SGP/ur2XwmGvA3fp/mUzaZKYHEKK1Xqy1jkvpafzzCI9frrUO1Frf0VpfBvyAPinbOgCxWuu/lVIVMCa217TWt7XWocBSYNAjHEuIe0giEHlVH621LdAOcOC/D/hbgAGolMFjKgHhKb/fyKRNZqoCFx4rUqPAdLc3YuwlAAzhv95AdaAgcE0pFaGUigA+Bso/wbFFPieJQORpWus9wFpgUcrt28BfwMAMmj/Lf+WcX4AuSqliJh4qEKidybbbQNE0tytmFGq6298C7VJKW335LxEEAvFAWa11yZSfElrrBibGKcR9JBGI/GAZ8LRS6u4J4ynAC0qpCUopW6VUqZSTuc2BWSlt1mP80N2slHJQShVQSpVRSr2tlOqewTF+ACoqpV5TStmk7LdZyrYjGGv+pZVSFYHXHhaw1joM+AP4HLiotT6dcv81jCOeFqcMby2glKqtlGr7GK+LEIAkApEPpHyofgG8m3L7T6AL0A/jeYDLGE+6ttJan09pE4/xhPEZ4GcgCjiIscR0X+1fax2N8URzTyAEOA+0T9m8HuPw1EsYP8S/NjH0jSkxbEx3//NAIeAUxlKXD49WxhLiHkoWphFCiPxNegRCCJHPSSIQQoh8ThKBEELkc5IIhBAin8t1E1yVLVtW16hRw9JhCCFEruLr6xuutS6X0bZclwhq1KjBoUOHLB2GEELkKkqpy5ltk9KQEELkc5IIhBAin5NEIIQQ+VyuO0eQkcTERIKCgoiLi7N0KELkeIULF8be3p6CBQtaOhSRQ+SJRBAUFIStrS01atRAKWXpcITIsbTW3Lhxg6CgIGrWrGnpcEQOYbbSkFJqjVIqVCl1IpPtSim1PGVB8GNKKbfHPVZcXBxlypSRJCDEQyilKFOmjPSexT3MeY5gLcZFvzPTDaib8jMK+OhJDiZJQAjTyP8VkZ7ZEoHWei9w8wFNemNcQFxrrf8GSiqlZCpdIYRI50xQODO+Pci569Fm2b8lRw1V4d7l+YJS7ruPUmqUUuqQUupQWFhYtgT3qKysrHB1dcXZ2ZmePXsSERGRuu3kyZN06NCBevXqUbduXebMmUPa6b9//PFHPDw8cHR0xMHBgTfffNMST+GBBg8ejIuLC0uXLn2sx69du5bx48c/cRyrVq3iiy++yHT7H3/8wYEDB0xun96D3secoHv37jkuJmEeMfFJfHMokM7zf6Trh/+w7tB1/r4Q/vAHPgZLJoKM+qcZLo6gtV6ttfbQWnuUK5fhFdIWV6RIEY4cOcKJEycoXbo03t7eANy5c4devXoxZcoUzp07x9GjRzlw4AArV64E4MSJE4wfP54NGzZw+vRpTpw4Qa1atbI0tqSkpCd6fEhICAcOHODYsWO8/vrr2XLMzHh5efH8889nuj19InhY+/Qyex+fVFa9Hjt37qRkyZJZsi+R8xgMmgP+4Uz8+gge7/3MWz7HOOl/hQInfmBJ++I838I8J/gtmQiCMC74fZc9cNVCsWSp5s2bExwcDMDGjRtp2bIlnTt3BqBo0aJ8+OGHzJs3D4AFCxYwbdo0HBwcALC2tmbs2LH37TMmJoYXX3yRhg0b4uLiwubNmwEoXrx4ahsfHx9GjBgBwIgRI5g4cSLt27dn0qRJ1KhR455vknXq1OH69euEhYXRv39/mjRpQpMmTdi/f/99x+7cuTOhoaG4urqyb98+jhw5wlNPPYWLiwt9+/bl1q1bALRr1463336btm3b8sEHH5j0Wn311Vc0bNgQZ2dnJk+enHr/Z599Rr169WjXrh0jR45M7U3MnDmTRYsWAbB8+XKcnJxwcXFh0KBBXLp0iVWrVrF06dLUWNO29/f3p1OnTjRq1Ag3NzcuXHjwWvNp30eAhQsX0qRJE1xcXJgxY0bq/XPmzMHBwYGnn36awYMHpx4v/euR2Wu9Z88eXF1dcXV1pXHjxkRHR3Pt2jXatGmT2jvZt28fYJxiJTzc+K1wyZIlODs74+zszLJlywC4dOkSjo6OjBw5kgYNGtC5c2fu3Llj0nshLOdS+G0W7z5L6wW/M+TTf/j59HWSzh/g+peTGFo6gJPfLqFf1/YP39FjsuTw0e3AeKXUJqAZEJmyHusTmfX9SU5djXri4NJyqlyCGT1NWxs8OTmZX3/9FU9PT8BYFnJ3d7+nTe3atYmJiSEqKooTJ07wxhtvPHS/c+bMwc7OjuPHjwOkfvg+yLlz5/jll1+wsrLCYDCwZcsWXnzxRf755x9q1KhBhQoVGDJkCK+//jqtWrXiypUrdOnShdOnT9+zn+3bt9OjRw+OHDkCgIuLCytWrKBt27ZMnz6dWbNmpX4QRUREsGfPnoe/UMDVq1eZPHkyvr6+lCpVis6dO7N161aaNm3KnDlz8PPzw9bWlg4dOtCoUaP7Hj9v3jwuXryIjY0NERERlCxZEi8vL4oXL55aXvv1119T2w8dOpQpU6bQt29f4uLiMBgMmcaW/n3cvXs358+f5+DBg2it6dWrF3v37qVo0aJs3ryZw4cPk5SUhJub2z3vd9rXI7PXetGiRXh7e9OyZUtiYmIoXLgwq1evpkuXLkybNo3k5GRiY2Pvic/X15fPP/+cf/75B601zZo1o23btpQqVYrz58/z1Vdf8cknn/Dss8+yefNmhg0bZtJ7IrJPdFwiO49fw8c3iH8v3aKAgmbV7ZjctT6dG1Tkxx9iqfpmbzw8PMwei9kSgVLqK6AdUFYpFQTMAAoCaK1XATuB7oA/EAu8aK5YssOdO3dwdXXl0qVLuLu78/TTTwPGcduZjdJ4lNEbv/zyC5s2bUq9XapUqYc+ZuDAgVhZWQHw3HPPMXv2bF588UU2bdrEc889l7rfU6dOpT4mKiqK6OhobG1tM9xnZGQkERERtG1rXCv9hRdeYODAganb7+7XFP/++y/t2rXjbrlv6NCh7N27F4C2bdtSunTp1Odx7ty5+x7v4uLC0KFD6dOnD3369HngsaKjowkODqZv376A8aKqjGT2Pu7evZvdu3fTuHFjwNhDO3/+PNHR0fTu3ZsiRYoA0LNnz3v2l/b1yOy1btmyJRMnTmTo0KH069cPe3t7mjRpwksvvURiYiJ9+vTB1dX1nv3++eef9O3bl2LFigHQr18/9u3bR69evahZs2Zqe3d3dy5duvTA10Zkn2SD5q8LN/DxDeSnkyHEJRqoXa4Yk7vWR1/8h3fffJ4O8+ZR2HVk6t9qdjBbItBaD37Idg2My+rjmvrNPavdrS1HRkbSo0cPvL29mTBhAg0aNEj9cLsrICCA4sWLY2trS4MGDfD19c3wG29amSWUtPelHxt+90MCjGUOf39/wsLC2Lp1K++88w4ABoOBv/76K/WD7EmlPebDZLZetqnraO/YsYO9e/eyfft25syZw8mTJx/5WOll9j5qrZk6dSqjR4++p/3DTp6nfT0ye62nTJnCM888w86dO3nqqaf45ZdfaNOmDXv37mXHjh0MHz6cSZMm3XOu40HPx8bGJvV3KysrKQ3lAAFhMWz2C+I7v2CuRcZRorA1/d3sGeBuTxmiGTNmTOr737Jly2yPT+YaymJ2dnYsX76cRYsWkZiYyNChQ/nzzz/55ZdfAOM3zgkTJvDWW28BMGnSJObOnZv6jddgMLBkyZL79tu5c2c+/PDD1Nt3S0MVKlTg9OnTqaWfzCil6Nu3LxMnTsTR0ZEyZcpkuN+75Z8HPb9SpUql1qzXr1+f2jt4VM2aNWPPnj2Eh4eTnJzMV199Rdu2bWnatCl79uzh1q1bJCUlpZ4PSctgMBAYGEj79u1ZsGABERERxMTEYGtrS3T0/UPsSpQogb29PVu3bgUgPj7+vnJL+ueZ9n3s0qULa9asISYmBoDg4GBCQ0Np1aoV33//PXFxccTExLBjx45M95nZa33hwgUaNmzI5MmT8fDw4MyZM1y+fJny5cszcuRIPD098fPzu2dfbdq0YevWrcTGxnL79m22bNlC69atH/Bqi+wWFZfIxn+u0P+jA3RYvIeP/rhA/Yq2fDikMQendeJ/fRtyZv9PODs788cff7Bs2TL+/PNPnJycsj3WPDHFRE7TuHFjGjVqxKZNmxg+fDjbtm3jlVdeYdy4cSQnJzN8+PDUk58uLi4sW7aMwYMHExsbi1KKZ5555r59vvPOO4wbNw5nZ2esrKyYMWMG/fr1Y968efTo0YOqVavi7Oyc+kGVkeeee44mTZqwdu3a1PuWL1/OuHHjcHFxISkpiTZt2rBq1aoHPr9169bh5eVFbGwstWrV4vPPPzfpdVm7dm3qBzHA33//zfvvv0/79u3RWtO9e3d69+4NwNtvv02zZs2oXLkyTk5O2NnZ3bOv5ORkhg0bRmRkJFprXn/9dUqWLEnPnj0ZMGAA27ZtY8WKFfc8Zv369YwePZrp06dTsGBBvv322weO0Er/Pp4+fZrmzZsDxpP0GzZsoEmTJvTq1YtGjRpRvXp1PDw87ov1rsxe62XLlvH7779jZWWFk5MT3bp1Y9OmTSxcuJCCBQtSvHjx+4bAurm5MWLECJo2bQrAyy+/TOPGjaUMZGHJBs1+/3B8fIPYdTKE+CQDdcsXZ2o3B/o0rkKFEveWJEuVKkWzZs1YvXq1Raf8UKZ2mXMKDw8PnX5hmtOnT+Po6GihiIQ5xMTEULx4cZKSkujbty8vvfRSttZMH8XdWGNjY2nTpg2rV6/Gze2xZ0zJFvJ/Jmv5hxpLP1v8ggmJisOuSEF6NarMAHd7XOztUku4SUlJLF26lISEBKZNmwY8+DxiVlJK+WqtMzzzLD0CkSPNnDmTX375hbi4ODp37vzQk8GWNGrUKE6dOkVcXBwvvPBCjk8CImtExiby/bGrbPYL4vCVCKwKKNrWK8f0nk50dCyPjbXVPe2PHj2Kp6cnvr6+PPvss6kJICdM+SGJQORId8fi5wYbN260dAgimyQbNPvOh+HjG8TuU9dJSDJQv4It07o70rtxZcrb3j8aLT4+nvfee4958+ZRunRpvv32W/r3758jEsBdeSYRZFf3SojcLreVg3OC89ej8Ukp/YRGx1OyaEGGNK3GAHd7GlQu8cDPnvPnzzN//nyGDBnCkiVLUgdq5CR5IhEULlyYGzduyFTUQjzE3fUIMruOQvwnIjaB749exccvmKOBxtJP+/rlGOBuT3uH+0s/acXExLBt2zaGDh2Ks7MzZ86cyfKpY7JSnkgE9vb2BAUFkVMnpBMiJ7m7Qpm4X1KygX3njaN+fj51nYRkAw4VbXnnGUd6u1ahnK3NQ/fx888/M2rUKC5fvoybmxuOjo45OglAHkkEBQsWlNWWhBCP7WxItHHUz+FgwqLjKV2sEEOfqkZ/t4eXfu66desWb775JmvWrKFevXrs2bMn14zMyhOJQAghHtWt2wlsP2oc9XMsKBLrAor2DuWNpZ/65Slkbfr1tsnJybRs2ZJz584xdepUpk+fnqvKb5IIhBD5RmKygT1nw9jsF8Qvp6+TmKxxqlSC6T2c6O1amTLFH176SSs8PJzSpUtjZWXF3LlzqVatWq4cPiyJQAiR550JicLnUBBbjwQTHpNAmWKFeL55Dfq72eNUucQj709rzfr163nttdeYN28eo0aNytHXujyMJAIhRJ5083YC244Es9kviBPBURS0UnR0qEB/d3va1S9HQavHm2rt8uXLjB49ml27dtGiRQvatGmTxZFnP0kEQog8IzHZwB9nw/DxDeS3M6EkJmucq5RgZk8nerlWoXSxQk+0/w0bNjBmzBi01qxYsYKxY8dSoEDun7tTEoEQItc7dTUKH98gth0J5sbtBMoWt2FEixr0d7fHoeKjl34yU65cOVq2bMnHH39M9erVs2y/liaJQAiRK4XHxLPtyFV8fIM4fS2KQlYF6ORUnv5u9rSp9/iln7QSExNZvHgxiYmJvPvuu3Tp0oXOnTvnuQtXJREIIXKNhCQDv50JZbNfEL+fCSXJoGlkb8fs3g3o6VKZUk9Y+knr8OHDeHp6cvjwYQYNGpSjJonLapIIhBA5mtaak2lKP7diEylna4Nnq5r0d7enXoWMl1V9XHFxccyePZsFCxZQtmxZNm/eTL9+/bL0GDmNJAIhRI4UFh3PtiPB+PgGcSYkmkJWBXi6QQUGuNvTuk5ZrLOg9JMRf39/Fi1axPPPP8/ixYtNWh88t5NEIITIMeKTkvntdCg+vkH8cS6MZIPGtWpJ5vRxpqdLJUoWzbrST1oxMTFs2bKF4cOH4+zszNmzZ/PVtDWSCIQQFqW15kRwFD6+gWw7epWI2EQqlLBhZOtaDHCvQp3yWVv6SW/Xrl2MGjWKwMBAPDw8cHR0zFdJACQRCCEsJDQqjq0ppZ9z12MoZF2ALg0qMsDdnlZ1ymJVwLwnZW/cuMHEiRP54osvcHBwYN++fblmkrisJolACJFt4hKT+fV0KD6+gew9H06yQeNWrST/6+tMD5fK2BUpmC1x3J0kzt/fn2nTpvHOO+/kqknispokAiGEWWmtORoUyWbfILYfvUrknUQqlijM6Da16O9uT+1yxbMtlrCwMMqUKYOVlRXz58+nevXquLq6ZtvxcypJBEIIs7geFceWw8bSj39oDDbWBejqbCz9tKht/tJPWlpr1q5dy8SJE5k3bx6jR4+md+/e2Xb8nE4SgRAiy8QlJvPzqev4+Aax73wYBg0e1Usxr19DurtUokTh7Cn9pHXp0iVGjRrFzz//TOvWrWnfvn22x5DTSSIQQjwRrTWHAyPY7BvE90evEhWXRGW7woxtV4f+7vbULFvMYrGtX7+eMWPGoJRi5cqVjB49Ok9MEpfVJBEIIR5LSGQc3x0Owsc3iICw2xQuWIBuzpUY4G5P81plKJCNpZ/MVKhQgTZt2rBq1SqqVatm6XByLKW1tnQMj8TDw0MfOnTI0mEIkS/FJSaz62QIPr5B/OkfjtbQtEZpBrjb061hRWwtUPpJKzExkQULFpCcnMz06dMtGktOo5Ty1Vp7ZLRNegRCiAfSWuN35RY+vkH8cPQa0fFJVClZhFc61KW/WxWql7Fc6SctPz8/XnrpJY4ePcqQIUNSJ4kTDyeJQAiRoasRd1JH/VwMv02RglZ0a2gc9fNUzZxR+gG4c+cOs2bNYtGiRZQrV44tW7bk6mUjLcGsiUAp1RX4ALACPtVaz0u3vRqwDiiZ0maK1nqnOWMSQmTuTsJ/pZ/9F4yln2Y1SzO2XW26NaxEcZuc990xICCAJUuWMGLECBYuXJgvJonLamZ7V5VSVoA38DQQBPyrlNqutT6Vptk7wDda64+UUk7ATqCGuWISQtxPa82hy7fwORTEjuPXiIlPomrpIrzasS79GttTrUxRS4d4n6ioKL777jtGjBhBgwYNOH/+fJ5aMSy7mTO9NwX8tdYBAEqpTUBvIG0i0MDddeTsgKtmjEcIkUbQrVi2+BkXd790I5aihazo3tA46qdpjdI5pvST3s6dO/Hy8iI4OJhmzZrh6OgoSeAJmTMRVAEC09wOApqlazMT2K2UegUoBnTKaEdKqVHAKECGgAnxBGITkvjphLH0c+DCDQCa1yrDKx3q0tW5IsVyYOnnrvDwcF5//XU2bNiAk5MT+/fvz7eTxGU1c77rGX2dSD9WdTCwVmu9WCnVHFivlHLWWhvueZDWq4HVYBw+apZohcijtNYcvHgTH98gdh6/xu2EZKqVLsrEp+vRt3EVqpbOeaWf9O5OEhcQEMD06dN5++23sbGxsXRYeYY5E0EQUDXNbXvuL/14Al0BtNZ/KaUKA2WBUDPGJUS+EHgzlu9SSj9XbsZSrJAVPVwq09/dniY1SuWKoZXXr1+nXLlyWFlZsWjRIqpXr46Li4ulw8pzzJkI/gXqKqVqAsHAIGBIujZXgI7AWqWUI1AYCDNjTELkabfjk/jxRAg+voH8HXATpaBF7TK8/nRdujSoSNFCObf0k5bWmjVr1vDGG28wb948vLy86Nmzp6XDyrPM9lehtU5SSo0HdmEcGrpGa31SKTUbOKS13g68AXyilHodY9lohM5tlzoLYWEGg+aflNLPjyeuEZuQTI0yRXmzcz36utlTpWQRS4f4SAICAhg5ciS//fYbbdu2pVOnDE8diixk1q8HKdcE7Ex33/Q0v58CWpozBiHyqis3YtnsF8RmvyCCbt3B1saa3q6V6e9mj3v13FH6SW/dunWMHTsWKysrVq1axciRI2WSuGyQO/qJQggAYuKT2Hn8Gj6+QRy8aCz9tKpTlkld6tPZqSJFCllZOsQnUrlyZTp06MBHH32Evb29pcPJN2TSOSFyOINB83fAjZTSTwh3EpOpVbYY/d3t6du4CpVzWeknrYSEBObNm4fBYGDmzJmWDidPk0nnhMiFLoXf5ju/IDb7BRMccQfbwtb0davCAHd7GlctmStLP2n9+++/vPTSS5w4cYLhw4fLJHEWJIlAiBwkOi4xtfTz76VbFFDQqm45JndzoLNTBQoXzN2lH4DY2FimT5/O0qVLqVSpEtu3b5cRQRYmiUAIC0s2aP66cAMf30B+OhlCXKKB2uWKMbmrA30bV6GiXWFLh5ilLl68yIoVKxg5ciTz58/Hzs7O0iHle5IIhLCQgLAYNvsFscUvmKuRcZQobM0Ad3v6u9njmgdKP2lFRkby3Xff8eKLL9KgQQP8/f2pWrXqwx8osoUkAiGyUVRcIj8cvcZmvyB8LxtLP23rlePtZxzp5Jg3Sj/p7dixg9GjR3Pt2jWaN2+Og4ODJIEcRhKBEGaWbNDs9w/HxzeIXSdDiE8yULd8caZ2M5Z+ypfIW6Wfu8LCwnjttdfYuHEjzs7OfPfddzg4OFg6LJEBSQRCmIl/6H+ln5CoOOyKFOS5JlUZ4G5Pwyp2ear0k15ycjKtWrXi4sWLzJo1iylTplCoUCFLhyUyIYlAiCwUGZvI98eustkviMNXIrAqoGhXrxwzejrRwbE8NtZ5r/STVkhICOXLl8fKyorFixdTo0YNnJ2dLR2WeAhJBEI8oWSDZt/5MHx8g9h96joJSQbqV7BlWndHejeuTHnbvFn6SctgMPDJJ58wadIk5s+fz5gxY+jRo4elwxImemgiUEoVAV4DqmutvZRSdYC6WusfzR6dEDnY+evR+KSUfkKj4ylZtCBDmlZjgLs9DSqXyNOln7T8/f0ZOXIkf/zxBx06dKBLly6WDkk8IlN6BGuA40CrlNtXgW8BSQQi34mITeD7o1fx8QvmaKCx9NO+fnkGuFehvUPeL/2k9/nnnzN27FgKFSrEJ598gqenZ75JgHmJKYmgrtZ6sFJqIIDWOlbJOy3ykaRkA/vOG0f9/HzqOgnJBhwq2vLOM470dq1COdv8u1JWtWrV6NKlC97e3lSpUsXS4YjHZEoiSEhZOUwDpCw0k2DWqITIAc6GRBtH/RwOJiw6ntLFCjH0qbuln/x5NWx8fDzvv/8+BoOB2bNn07FjRzp27GjpsMQTMiURzAF+AuyVUuuAtsDLZo1KCAu5dTuB7UeNo36OBUViXUDRwaE8A9ztaVe/PIWs8+/c+P/88w+enp6cPHmSF154QSaJy0Memgi01j8qpQ4BLTAuSD9Jay1rCos8IzHZwJ6zYWz2C+KX09dJTNY0qFyC6T2c6O1amTLF82/pB+D27du8++67LFu2jCpVqvDDDz/wzDPPWDoskYVMGTW0W2vdGdiWwX1C5FpnQqLwORTE1iPBhMckULZ4IZ5vXoP+bvY4VS5h6fByjMuXL7Ny5Uq8vLyYN28eJUrIa5PXZJoIlFKFMC4mX0EpZYuxNwBQAqiWDbEJkeVu3k5g25FgNvsFcSI4ioJWio4OFRjgbk/b+uUoaJV/Sz9pRURE4OPjw8svv4yTkxP+/v6yYlge9qAewThgIlAeOMl/iSAKWGXmuITIMonJBv44G4aPbyC/nQklMVnTsIods3o1oGejypQuJlMfpLVt2zbGjBlDaGgorVq1wsHBQZJAHpdpItBaLwWWKqVe01ovy8aYhMgSp65G4eMbxLYjwdy4nUDZ4jaMaFGD/u72OFSU8kZ6oaGhTJgwga+//hoXFxe2b98uk8TlE6acLF6mlHIAnDCWiu7ev9GcgQnxOMJj4tl25Co+vkGcvhZFIasCdHIyjvppU7cc1lL6yVBycjItW7bkypUrvPfee7z11lsULFjQ0mGJbGLKyeJ3gM6AA7AL6AL8CUgiEDlCQpKB386EstkviN/PhJJk0DSyt2NOb2Ppp2RRKf1k5urVq1SsWBErKys++OADatRP+Y4AACAASURBVNSogZOTk6XDEtnMlOsIngNcAT+t9XClVCXgY/OGJcSDaa05mab0cys2kfK2Nni2qkl/d3vqVbC1dIg5msFg4OOPP2by5MnMmzePsWPH0r17d0uHJSzElERwR2udrJRKShk9FALUMnNcQmQoLDqebUeC8fEN4kxINIWsCvB0A+Oon9Z1ykrpxwTnzp1j5MiR7N27l06dOtGtWzdLhyQszJREcFgpVRLj5HOHMI4a8jNrVEKkEZ+UzG+nU0o/Z8NINmhcq5bkvT7O9HSpjF1RqWWb6rPPPmP8+PEULlyYNWvWMGLECLk6WDw4EaRMLjdTax0BeCuldgEltNaSCIRZaa05ERyFj28g245eJSI2kQolbBjVphb93eypU764pUPMlWrUqEG3bt3w9vamUqVKlg5H5BBKa/3gBkr5aq3dsymeh/Lw8NCHDh2ydBjCTEKj49h62Fj6OXc9BhvrAnRuUJEB7va0qlMWqwLy7fVRxMfHM2fOHADee+89C0cjLCnls9wjo22mlIYOKqXcpBcgzCUuMZlfU0o/e84ZSz9u1Uoyt29DnnGphF0RKf08jgMHDuDp6cmZM2d46aWXZJI4kSlTEkErYKRS6gJwG+MVxlpr7WbWyESeprXmWFAkPr5BbD96lcg7iVSyK4xX21r0c7Ondjkp/TyumJgYpk2bxooVK6hatSo//fSTrBomHsiURNDncXeulOoKfABYAZ9qredl0OZZYCbG9Q6Oaq2HPO7xRM53PSqOLSmlH/9QY+mnm3NF+rvb06K2lH6ywpUrV/j4448ZN24cc+fOxdZWhtKKBzPlyuILj7NjpZQV4A08DQQB/yqltmutT6VpUxeYCrTUWt9SSpV/nGOJnC0uMZmfT11ns18Qe8+FYdDgUb0U8/o1pLtLJUoUltLPk7p16xbffvsto0aNwsnJiYCAACpXrmzpsEQuYUqP4HE1Bfy11gEASqlNQG/gVJo2IwFvrfUtAFnnIG85eTWSjf9c4fujV4mKS6KyXWHGta9DPzd7apYtZunw8owtW7YwduxYwsLCaNu2LfXr15ckIB6JORNBFSAwze0goFm6NvUAlFL7MZaPZmqtf0q/I6XUKGAUGNdIFTnf4Su3ePbjv7AqoOjuXIn+7vY0r1WGAlL6yTIhISG88sor+Pj44Orqyo4dO6hfv76lwxK5kEmJQCllj3ER+9+VUjaAtdb69sMelsF96ceqWgN1gXaAPbBPKeWcct3Cfw/SejWwGozDR02JWVhOeEw8Y7/0o0KJwmwb1zLfr/BlDsnJybRu3ZrAwEDmzp3Lm2++KZPEicdmyqRzLwHjATugNlAdWAl0eshDg4CqaW7bA1czaPO31joRuKiUOosxMfxrUvQix0lKNvDKxsPcvJ3A5jEtJAlksaCgICpXroyVlRXLly+nZs2aMlW0eGKmTMwyAXgK49QSaK3PYVys5mH+BeoqpWqmrHY2CNiers1WoD2AUqosxlJRgGmhi5xo0e5z/BVwg/f6OONcxc7S4eQZBoOBFStW4ODgwEcffQRAt27dJAmILGFKIojTWifcvZEyGuihhV6tdRLGnsQu4DTwjdb6pFJqtlKqV0qzXcANpdQp4Hdgktb6xqM+CZEz/HTiGqv2XGBIs2oM9Kj68AcIk5w5c4Y2bdowYcIEWrVqRY8ePSwdkshjTDlHsF8p9RZQWCnVHuMSlj+YsnOt9U5gZ7r7pqf5XWNcDnOiyRGLHOlCWAxvfnuMRvZ2zOgp89lnlU8//ZTx48dTtGhR1q1bx/Dhw+XqYJHlTOkRvAVEA2eAV4FfgWnmDErkLrfjk/Ba70sh6wKsHOaOjbWVpUPKM2rXrk3Pnj05ffo0zz//vCQBYRam9Ai6Y7wq+CNzByNyH601kzcf40JYDOs9m1GlZBFLh5SrxcXFMXv2bADmzp1L+/btad++vYWjEnmdKT2CZwF/pdTnSqkuKecIhABgzf5L/HDsGm92qU/LOmUtHU6utn//flxdXXn//fcJCwvjYTMDC5FVHpoItNbDMY7m+R54CQhQSq0yd2Ai5zt48SZzd56ms1MFxrStbelwcq3o6GheeeUVWrduTXx8PLt27eKTTz6RMpDINiat66e1jge2AWsxDgt91owxiVwgNCqOcRv9qFa6KIuebSQfWk8gKCiITz/9lFdeeYXjx4/TuXNnS4ck8pmHJgKlVCel1KfABWAY8AVQ0dyBiZwrMdnAuI1+xMQlsWqYu0wa9xhu3LiRej2Ao6MjAQEBfPDBBxQvLtNvi+xnSo/AC/gJcNRaD9Vab097XYHIf97feYZ/L91iXv+G1K8oUxw/Cq01Pj4+ODk5MWHCBM6ePQsgy0YKizLlHMEArbWP1vpOdgQkcrbtR6+yZv9FRrSoQW/XKpYOJ1e5du0a/fv3Z+DAgVStWpVDhw7JJHEiR8h0+KhSao/Wuq1S6hb3ThZ3d4Wy0maPTuQo565HM9nnGB7VS/F2d0dLh5Or3J0kLjg4mAULFvD6669jbW3OyX+FMN2D/hLvDl6WMYGCqLhEvNb7UszGGu+hbhSyNmmcQb4XGBhIlSpVsLKywtvbm5o1a1KvXj1LhyXEPTL936y1NqT8+pnWOjntD/BZ9oQncgKtNW9+c5TLN2PxHtKYCiUKWzqkHC85OZnly5ffM0lcly5dJAmIHMmUvqlL2hspF5Q1MU84IidatSeA3aeu884zjjSrVcbS4eR4p0+fxtPTk7/++otu3brRs2dPS4ckxANl2iNQSk1OOT/gopS6mfJzCwgj3URyIu/a7x/Owl1neMalEp6talo6nBxv9erVuLq6cu7cOdavX8+OHTtkVT2R4z2o0LsAKAcsTfm3HFBWa11aaz0pO4ITlnU14g6vfHWYWuWKs6C/i1w0ZoK6devSt29fTp06xbBhw+Q1E7nCg0pDdbTW55VS64EGd++8+4ettT5m5tiEBcUnJTP2Sz8SkgysGuZOMRsZ4ZKRO3fuMHPmTJRSzJs3TyaJE7nSg/53TwE8Ae8MtmmgjVkiEjnCnB9OcSQwglXD3KhTXq52zcjevXt5+eWXOX/+PF5eXmitpQcgcqVME4HW2jPl39bZF47ICXx8g9jw9xVGt61FV2e54jW9qKgopkyZwkcffUStWrX49ddf6dChg6XDEuKxmTLXUD+llG3K71OUUt8opRqZPzRhCSevRjJty3Ga1yrDpM5y1WtGrl69ytq1a5k4cSLHjh2TJCByPVOuCpqptY5WSrUAegJfAx+bNyxhCZGxiXht8KVU0UKsGNIYayu5aOyu8PBwVq5cCYCDgwMXL15k8eLFFCtWzMKRCfHkTPmfnpzybw9gpdZ6M2BjvpCEJRgMmte+PkxIZBzeQ90oW1zeYjBeTPf111/j5OTEa6+9xrlz5wCoUKGChSMTIuuYkgiuKaW8gUHATqVUIRMfJ3KRFb/58/vZMKb3cMK9eilLh5MjXL16lT59+jBo0CCqV6+Or6+vXBks8iRTxgQ+i3Hd4hVa61tKqcoYRxSJPOL3s6Es+/Uc/RpXYdhT1S0dTo6QnJxMmzZtCA4OZtGiRbz66qsySZzIsx76l621jlFKnQLaKaXaAfu01j+aPTKRLQJvxvLapiPUr2DL//o2zPfDHy9fvoy9vT1WVlasXLmSWrVqUadOHUuHJYRZmTJqaDzwDVAt5ecbpdRYcwcmzC8uMRmvDb4YtObj4e4UKWRl6ZAsJjk5mSVLluDo6Jg6SVznzp0lCYh8wZS+7iigqdY6BkApNRc4AKw0Z2DCvLTWvLv1BCevRvHZCx5UL5N/R7+cOHECT09PDh48SI8ePejTp4+lQxIiW5ly0lcBiWluJ6bcJ3KxTf8G8q1vEBM61KGjY/4dAbNq1Src3NwICAhg48aNbN++HXt7e0uHJUS2MqVHsB74Wym1GWMC6AOsM2tUwqyOBkYwY9tJ2tQrx6ud8ucomLvTQTg6OjJw4ECWLVtGuXLlLB2WEBahtNYPb6RUE+DuVBP7tNb/mjWqB/Dw8NCHDh2y1OFzvZu3E+ixfB9KKX54pRWlihWydEjZKjY2lunTp2NlZcX8+fMtHY4Q2UYp5au19shom6nXA8Sn/NxJ+VfkQskGzYSvDhN+O4FVw9zzXRL4448/cHFxYfHixcTExGDKlyAh8gNTRg1NA74CKgH2wEal1FRzByay3pKfz/Knfzjv9Xamob2dpcPJNpGRkYwePTp1eujffvsNb2/vfD9UVoi7TDlHMAxw11rHAiil/gf4Au+bMzCRtXafDMH79wsMalKVZ5tUtXQ42eratWts2LCBN998k1mzZlG0aFFLhyREjmJKaegy9yYMayDAlJ0rpboqpc4qpfyVUplejayUGqCU0kqpDOtX4slcDL/NG98cpWEVO2b2avDwB+QBYWFhrFixAjBOEnfp0iUWLlwoSUCIDJiSCGKBk0qpT5VSnwDHgQil1BKl1JLMHpSyyL030A1wAgYrpZwyaGcLTAD+eZwnIB4sNiEJr/W+WFkpPhrmRuGCefuiMa01GzduxNHRkTfeeCN1kjgZESRE5kwpDe1I+bnrbxP33RTw11oHACilNgG9gVPp2s3BuD7ymybuV5hIa83U745zLjSadS82xb5U3v42HBgYyJgxY9ixYwfNmjXjs88+k0nihDCBKXMNffaY+64CBKa5HQQ0S9tAKdUYqKq1/kEplWkiUEqNwniFM9WqVXvMcPKfdQcuse3IVd7sXI829fL2N+KkpCTatWtHSEgIS5cu5ZVXXsHKKm/3foTIKuacTjGjIRmp4/WUUgWApcCIh+1Ia70aWA3G6wiyKL487dClm7y34zSdHMsztl3enS/n0qVLVK1aFWtraz7++GNq1apFrVq1LB2WELmKOdcVCALSDk+xB66muW0LOAN/KKUuAU8B2+WE8ZMLjY5j3EY/qpQqwuJnXSlQIO8Nk0xKSmLRokU4OjqmrhzWqVMnSQJCPAaTewRKKRut9aNcTPYvUFcpVRMIxriwzZC7G7XWkUDZNPv/A3hTay2XDT+BxGQD4zceJvJOImtfbIpdkYKWDinLHTt2DE9PTw4dOkTv3r3p37+/pUMSIlcz5YKypkqp48D5lNuNlFIrHvY4rXUSMB7YBZwGvtFan1RKzVZK9XrCuEUmFvx0hoMXb/J+v4Y4Viph6XCy3MqVK3F3d+fy5ct8/fXXbNmyhcqVK1s6LCFyNVN6BMsxrle8FUBrfVQp1d6UnWutdwI70903PZO27UzZp8jcjmPX+GTfRV5oXp2+jfPWDJp3J4lzdnZm0KBBLF26lLJlyz78gUKIhzIlERTQWl9Odzl+cmaNhWX4h0YzyecobtVKMu2Z+y7XyLVu377NO++8g7W1NQsXLqRNmza0adPG0mEJkaeYcrI4UCnVFNBKKSul1GvAOTPHJR5BdFwio9b7UrSQFSuHulPI2pxjALLPr7/+SsOGDVm2bBnx8fEySZwQZmLKJ8YYYCLGZSqvYxzdM8acQQnTaa15y+cYl2/EsmKwGxXtCls6pCcWERHByy+/TKdOnbC2tmbv3r0sX75cJokTwkxMuaAsFOOIH5EDfbIvgB9PhPB2dwea1y5j6XCyxPXr19m0aROTJ09mxowZFClSxNIhCZGnPTQRpMwvdF+fXGs9yiwRCZMduBDOvB/P0M25IiNb5+7x83c//F999VXq16/PpUuX5GSwENnElNLQL8CvKT/7gfLI4jQWFxIZx4SvDlOzbDEWDmyUa8smWms2bNiAk5MTb731FufPnweQJCBENjKlNPR12ttKqfXAz2aLSDxUQpKBsV/6cichmU2jnqK4jTlnCjGfK1eu4OXlxY8//kjz5s357LPPqFu3rqXDEiLfeZxPkJpA9awORJjufztO4XclAu8hbtQpb2vpcB7L3UniQkNDWb58OWPHjpVJ4oSwEFPOEdziv3MEBYCbQKaLzAjz2nI4iHV/XWZk65o841LJ0uE8soCAAKpXr461tTWffPIJtWvXpkaNGpYOS4h87YHnCJSx8NwIKJfyU0prXUtr/U12BCfudfpaFFO/O06zmqWZ3NXB0uE8kqSkJObPn4+TkxPe3t4AdOzYUZKAEDnAA3sEWmutlNqitXbProBExiLvJOK1wRe7IgX5cIgb1la556KxI0eO4OnpiZ+fH3379mXgwIGWDkkIkYYpnyYHlVJuZo9EZMpg0LzxzRGCb91h5VA3ytnaWDokk3344Yc0adKE4OBgfHx8+O6776hUKfeVtITIyzLtESilrFNmEG0FjFRKXQBuY1xwRmutJTlkk5V/+PPL6VBm9WqAe/XSlg7HJHcniXNxcWHo0KEsWbKE0qVzR+xC5DcPKg0dBNyAPtkUi8jA3nNhLP75HH1cK/N885w/WCsmJoZp06ZRsGBBFi1aJJPECZELPKg0pAC01hcy+smm+PK1wJuxTNh0mPoVbJnbr2GOv2hs9+7dODs7s2LFChITE2WSOCFyiQf1CMoppSZmtlFrvcQM8YgUcYnJjP3Sj+RkzUfD3ClaKOdeNHbr1i0mTpzI2rVrqV+/Pnv37qVVq1aWDksIYaIHfbpYAcXJeBF6YWYzt5/keHAkq4e7U7NsMUuH80ChoaH4+PgwdepUpk+fTuHCuX8GVCHykwclgmta69nZFolI9fW/V9j0byDj2temc4OKlg4nQyEhIXz11Ve8/vrrqZPElSmTN2Y/FSK/eeg5ApG9jgdF8u62k7SqU5aJT9e3dDj30Vqzbt06nJycmDp1auokcZIEhMi9HpQIOmZbFAKAW7cT8NrgS7niNiwf3BirAjkrF1+6dImuXbsyYsQInJycOHLkiEwSJ0QekGlpSGt9MzsDye+SDZpXvz5CWHQ833o1p3SxQpYO6R5JSUm0b9+e8PBwvL298fLyokCB3HN1sxAiczl3KEo+88Ev59h7Loz3+zWkUdWSlg4nlb+/PzVr1sTa2po1a9ZQq1YtqlfP+dczCCFMJ1/pcoBfT19n+W/+POthz6AmVS0dDgCJiYnMnTuXBg0apE4S1759e0kCQuRB0iOwsEvht3nt6yM4VynB7N7OOeKiMT8/Pzw9PTly5AgDBw7kueees3RIQggzkh6BBd1JSMZrgy9WBRQfDXWncEHLL8yyfPlymjZtSkhICN999x3ffPMNFSpUsHRYQggzkkRgIVpr3t5ynLPXo1n2nCtVSxe1eDwAjRs35vnnn+fUqVP07dvXojEJIbKHlIYsZMPfl9lyOJiJT9ejXf3yFosjOjqaqVOnYmNjw+LFi2ndujWtW7e2WDxCiOwnPQIL8Ltyi9k/nKKDQ3nGt69jsTh++uknnJ2dWblyJVprmSROiHxKEkE2C4+JZ+wGPyrZFWHps64UsMBFYzdu3OCFF16gW7duFCtWjP3797NkyZIccaJaCJH9JBFko6RkA69sPMyt2AQ+GuaGXdGCFonjxo0bbNmyhXfffZfDhw/TvHlzi8QhhMgZzJoIlFJdlVJnlVL+SqkpGWyfqJQ6pZQ6ppT6VSmVpwepL9x9lr8CbjC3b0MaVLbL1mNfu3aNRYsWobWmXr16XL58mdmzZ2Njk3uWvRRCmIfZEoFSygrwBroBTsBgpZRTumaHAQ+ttQvgAywwVzyW9uPxa3y8J4BhT1Wjv7t9th1Xa82aNWtwdHTk3Xffxd/fH4BSpUplWwxCiJzNnD2CpoC/1jpAa50AbAJ6p22gtf5dax2bcvNvIPs+IbORf2gMk3yO4Vq1JO/2SJ8LzefixYt07twZT09PGjVqxNGjR2WSOCHEfcw5fLQKEJjmdhDQ7AHtPYEfM9qglBoFjAKoVq1aVsWXLW7HJ+G1wRcb6wJ8NMwNG+vsuWgsKSmJDh06cOPGDT766CNGjRolk8QJITJkzkSQ0RCUDMcnKqWGAR5A24y2a61XA6sBPDw8cs0YR601b20+RkBYDBs8m1HJrojZj3n+/Hlq1aqFtbU1n3/+ObVr16Zq1Zwxf5EQImcy51fEICDtJ5A9cDV9I6VUJ2Aa0EtrHW/GeLLdZ39eZMexa7zV1YEWdcqa9ViJiYm89957ODs78+GHHwLQrl07SQJCiIcyZ4/gX6CuUqomEAwMAoakbaCUagx8DHTVWoeaMZZs90/ADd7/8QxdGlRgdJtaZj3WoUOH8PT05NixYwwaNIjBgweb9XhCiLzFbD0CrXUSMB7YBZwGvtFan1RKzVZK9UppthAoDnyrlDqilNpurniy0/WoOMZtPEz10kVZNLCRWS/U+uCDD2jWrBnh4eFs27aNr776ivLlLTdlhRAi9zHrXENa653AznT3TU/zeydzHt8SEpMNjPvSj9iEJDaObIZtYfNcNKa1RimFh4cHnp6eLFiwgJIlc86CNkKI3EMmnctic3ee5tDlW6wY3Jh6FWyzfP9RUVFMnjyZwoULs3TpUlq2bEnLli2z/DhCiPxDxhNmoW1Hgvl8/yVealmTno0qZ/n+d+7cSYMGDVi9ejXW1tYySZwQIktIIsgiZ0OimbL5OE1qlGJqd4cs3Xd4eDjDhg3jmWeewc7OjgMHDrBw4UKZJE4IkSUkEWSBqLhEvDb4UrywNd5D3CholbUv661bt/j++++ZMWMGfn5+NGv2oOvyhBDi0cg5gidkMGje+OYogTdj+WrUU5QvUThL9hscHMyXX37JpEmTqFu3LpcvX5aTwUIIs5AewRNatfcCP5+6ztvdHWlSo/QT709rzSeffIKTkxMzZ87kwoULAJIEhBBmI4ngCfx5PpxFu87Ss1FlXmxZ44n3d+HCBTp27MioUaNwc3Pj2LFj1KljuRXMhBD5g5SGHlNwxB0mbDpMnfLFmdev4ROfuE1KSqJjx47cvHmTjz/+mJdfflkmiRNCZAtJBI8hPimZsRt8SUgysGqYO8VsHv9lPHv2LLVr18ba2pp169ZRu3Zt7O3z5GzcQogcSr5yPobZ35/iaFAkiwY2ola54o+1j4SEBGbNmkXDhg3x9vYGoG3btpIEhBDZTnoEj+jbQ4F8+c8VxrSrTVfnio+1j4MHD+Lp6cmJEycYMmQIQ4cOzeIohRDCdNIjeAQngiN5Z+sJWtYpwxtP13usfSxbtozmzZunXhvw5ZdfUraseaeoFkKIB5FEYKKI2ATGfOlL6WKFWD6oMdaPeNHY3ekgmjZtysiRIzl58iQ9evQwR6hCCPFIpDRkAoNB89rXRwiJjOOb0c0pU9zG5MdGRkby1ltvUaRIEZYtW0aLFi1o0aKFGaMVQohHIz0CEyz/7Tx/nA1jRs8GNK5WyuTHff/99zg5OfHpp59iY2Mjk8QJIXIkSQQP8fuZUD749Tz93ewZ2qyaSY8JCwtjyJAh9OrVizJlyvD3338zf/58mSROCJEjSSJ4gCs3Ynl102EcK5bgf32dTf4gj4yMZOfOncyaNYtDhw7RpEkTM0cqhBCPT84RZCIuMRmvDb4ArBrmTuGCVg9sHxgYyIYNG5gyZQp16tTh8uXL2NnZZUeoQgjxRKRHkAGtNdO2nODUtSg+GNSYamWKZtrWYDCwatUqGjRowHvvvZc6SZwkASFEbiGJIAMbD15hs18Qr3asS3uHzBeCP3/+PB06dGDMmDE0bdqU48ePyyRxQohcR0pD6RwJjGDW9lO0q1+OVzvWzbRdUlISTz/9NBEREXz22We8+OKLcjJYCJErSSJI40ZMPGM3+FK+hA3LnnOlQIH7P9hPnz5N3bp1sba2Zv369dSuXZvKlbN+fWIhhMguUhpKkWzQTNh0mBu3E1g1zJ2SRQvdsz0+Pp4ZM2bg4uLChx9+CEDr1q0lCQghcj3pEaRYvPss+/1vsHCAC85V7j3R+/fff+Pp6cmpU6cYPnw4w4cPt1CUQgiR9aRHAOw6GcLKPy4wpFk1BnpUvWfb4sWLadGiBdHR0ezcuZMvvviCMmXKWChSIYTIevk+EQSExfDmN0dpZG/HjJ5OqfcbDAYAmjdvjpeXFydOnKBbt26WClMIIcwmX5eGYhOS8Nrgi7WVYuUwd2ysrYiIiOCNN96gaNGirFixQiaJE0Lkefm2R6C1Zsrm4/iHxrBisBtVShZh69atODk5sW7dOmxtbWWSOCFEvpBvewRrD1xi+9GrTOpSn3p2Bp599lm+/fZbXF1d+eGHH3Bzc7N0iEIIkS3yZY/g30s3+d+O0zztVIExbWsTFRXFzz//zP/+9z8OHjwoSUAIka/kux5BaHQc4770o4JtQSpf+Rml3KlTpw5XrlzB1tbW0uEJIUS2M2uPQCnVVSl1Vinlr5SaksF2G6XU1ynb/1FK1TBnPInJBsZ/6cetmDucXPUqi97/b5I4SQJCiPzKbIlAKWUFeAPdACdgsFLKKV0zT+CW1roOsBSYb654ACZv/IuDl25xddsimta35+TJkzJJnBAi3zNnj6Ap4K+1DtBaJwCbgN7p2vQG1qX87gN0VGaauW3b4SC+OxlB/PFdeL/1Irt27aJGjRrmOJQQQuQq5jxHUAUITHM7CGiWWRutdZJSKhIoA4SnbaSUGgWMAqhWzbTlItMra1sY9wrWLB0/mWr2VR5rH0IIkReZMxFk9M0+/cB8U9qgtV4NrAbw8PB4rMH9LeuUpeXrXR7noUIIkaeZszQUBKSduMceuJpZG6WUNWAH3DRjTEIIIdIxZyL4F6irlKqplCoEDAK2p2uzHXgh5fcBwG9aLucVQohsZbbSUErNfzywC7AC1mitTyqlZgOHtNbbgc+A9Uopf4w9gUHmikcIIUTGzHpBmdZ6J7Az3X3T0/weBww0ZwxCCCEeLF9OMSGEEOI/kgiEECKfk0QghBD5nCQCIYTI51RuG62plAoDLj/mw8uS7qrlfECec/4gzzl/eJLnXF1rXS6jDbkuETwJpdQhrbWHpePITvKc8wd5zvmDuZ6zlIaEECKfk0QghBD5XH5LBKstHYAFyHPOoM8GUgAAB5xJREFUH+Q55w9mec756hyBEEKI++W3HoEQQoh0JBEIIUQ+lycTgVKqq1LqrFLKXyk1JYPtNkqpr1O2/6OUqpH9UWYtE57zRKXUKaXUMaXUr0qp6paIMys97DmnaTdAKaWVUrl+qKEpz1kp9WzKe31SKbUxu2PMaib8bVdTSv2ulDqc8vfd3RJxZhWl1P/bO/cYq6orDn8/YRCf+EAT46MjFRREpUpUTGynwZhGEwhqOxpHwFeCRmtIaYzBB5GkiqY1odZgGw3UpIq0aAdSMzUVRRkG8AGIGhUpURpTH6nE9wNX/9hr9DJz78yZ3Lnncu9dX7Jz19lnn7PXumfmrrP2PmftByW9J2lLif2StNC/j82STi27UzOrq0JKef0WMAoYBmwCxvVocy2wyOWLgaXV1jsHm38K7OvyNY1gs7c7AFgNdAETq613Dtd5NPAScLBvH15tvXOw+Y/ANS6PA7ZXW+8ybf4xcCqwpcT+84AnSCs8ngmsK7fPeowITge2mtk2M/sKeASY2qPNVGCJy38FJksqtmxmrdCvzWa2ysw+880u0opxtUyW6wwwH7gL+CJP5SpEFpuvBv5gZv8DMLP3ctZxsMliswEHujyC3ish1hRmtpq+V2qcCvzZEl3AQZKOKKfPenQERwLvFGzv8LqibczsG2AncGgu2lWGLDYXciXpjqKW6ddmST8CjjazlXkqVkGyXOcxwBhJayR1SfpZbtpVhiw2zwPaJO0grX9yfT6qVY2B/r/3S0UXpqkSxe7sez4jm6VNLZHZHkltwETgJxXVqPL0abOkvYB7gJl5KZQDWa7zUNLwUAsp6ntW0ngz+6jCulWKLDZfAiw2s99KmkRa9XC8mX1befWqwqD/ftVjRLADOLpg+yh6h4rftZE0lBRO9hWK7elksRlJ5wBzgSlm9mVOulWK/mw+ABgPPC1pO2kstb3GJ4yz/m3/3cy+NrN/A6+THEOtksXmK4FHAcxsLTCclJytXsn0/z4Q6tERbABGSzpW0jDSZHB7jzbtwAyXLwKeMp+FqVH6tdmHSe4nOYFaHzeGfmw2s51mNtLMms2smTQvMsXMnq+OuoNClr/tx0kPBiBpJGmoaFuuWg4uWWx+G5gMIGksyRG8n6uW+dIOTPenh84EdprZu+WcsO6GhszsG0nXAR2kJw4eNLNXJN0OPG9m7cADpPBxKykSuLh6GpdPRpvvBvYHlvm8+NtmNqVqSpdJRpvriow2dwDnSnoV2AX82sw+rJ7W5ZHR5l8Bf5I0mzREMrOWb+wkPUwa2hvp8x63AU0AZraINA9yHrAV+Ay4vOw+a/j7CoIgCAaBehwaCoIgCAZAOIIgCIIGJxxBEARBgxOOIAiCoMEJRxAEQdDghCMI9lgk7ZK0saA099G2uVS2xryRNFHSQpdbJJ1VsG+WpOk56jKh1rNxBpWn7t4jCOqKz81sQrWVGCj+0lr3i2stwCdAp+9bNNj9SRrqObOKMYGUUuQfg91vUD9ERBDUFH7n/6ykF72cVaTNiZLWexSxWdJor28rqL9f0pAix26XtMDbrZd0nNf/QGkdh+71HI7x+p9L2iJpk6TVXtciaaVHMLOA2d7n2ZLmSZojaayk9T3s2uzyaZKekfSCpI5imSUlLZb0O0mrgAWSTpfUqZSTv1PS8f4m7u1Aq/ffKmk/pXz3G7xtsYytQaNR7dzbUaKUKqQ3Yzd6eczr9gWGuzya9HYpQDOevx34PXCpy8OAfYCxwAqgyevvA6YX6XM7MNfl6cBKl1cAM1y+Anjc5ZeBI10+yD9bCo6bB8wpOP93227XKJdvBG4mvUHaCRzm9a2kt2l76rkYWAkM8e0DgaEunwP8zeWZwL0Fx/0GaOvWF3gD2K/a1zpKdUsMDQV7MsWGhpqAeyVNIDmKMUWOWwvMlXQUsNzM3pQ0GTgN2OApNvYBSuVcerjg8x6XJwEXuPwQaY0DgDXAYkmPAssHYhwpUdovgDtJP/itwPGkZHlPup5DgFJ5ZJaZ2S6XRwBLPPoxPCVBEc4Fpkia49vDgWOA1waoe1BHhCMIao3ZwH+BU0hDm70WnDGzv0haB5wPdEi6ipS6d4mZ3ZShDysh92pjZrMkneF9bXQHlZWlpNxPy9Op7E1JJwGvmNmkDMd/WiDPB1aZ2TQfknq6xDECLjSz1wegZ1DnxBxBUGuMAN61lGv+MtId825IGgVsM7OFpEyNJwP/Ai6SdLi3OUSl121uLfhc63In3ycnvBR4zs/zQzNbZ2a3Ah+we3pggI9JKbF7YWZvkaKaW0hOAVLa6MOU8uojqUnSiSX0LGQE8B+XZ/bRfwdwvTzcUMpKGzQ44QiCWuM+YIakLtKw0KdF2rQCWyRtBE4gLev3KmkM/p8+KfskUGp5v709oriBFIEA/BK43I+9zPcB3C3pZX90dTVpTd1CVgDTuieLi/S1FGjj+3z6X5FSoy+QtIk0j9BrQrwIdwF3SFrD7s5xFTCue7KYFDk0AZtd5/kZzh3UOZF9NAgKUFrEZqKZfVBtXYIgLyIiCIIgaHAiIgiCIGhwIiIIgiBocMIRBEEQNDjhCIIgCBqccARBEAQNTjiCIAiCBuf/lAL8sRJrmHcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(1)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "\n",
    "\n",
    "plt.plot(fpr_grd, tpr_grd, label='ROC curve for Logistic Regression')\n",
    "\n",
    "plt.xlabel('False positive rate')\n",
    "plt.ylabel('True positive rate')\n",
    "plt.title('ROC curve')\n",
    "plt.legend(loc='best')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "neural-networks-deep-learning",
   "graded_item_id": "XHpfv",
   "launcher_item_id": "Zh0CU"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}