feature-tuning-credit-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Context:\n",
    "    \n",
    "The dataset contains 1000 entries with 20 categorial/symbolic attributes prepared by Prof. Hofmann. In this dataset, \n",
    "each entry represents a person who takes a credit by a bank. Each person is classified as good or bad credit risks according to \n",
    "the set of attributes. The link to the dataset can be found below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset:\n",
    "https://www.kaggle.com/renaldydermawan25/credit-data/version/1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some Attribute information:\n",
    "    \n",
    "Age (numeric)\n",
    "\n",
    "Sex (text: male, female)\n",
    "\n",
    "Job (numeric: 0 - unskilled and non-resident, 1 - unskilled and resident, 2 - skilled, 3 - highly skilled)\n",
    "\n",
    "Housing (text: own, rent, or free)\n",
    "\n",
    "Saving accounts (text - little, moderate, quite rich, rich)\n",
    "\n",
    "Checking account (numeric, in DM - Deutsch Mark)\n",
    "\n",
    "Credit amount (numeric, in DM)\n",
    "\n",
    "Duration (numeric, in month)\n",
    "\n",
    "Purpose (text: car, furniture/equipment, radio/TV, domestic appliances, repairs, education, business, vacation/others)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objective:\n",
    "    \n",
    "To Guage Feature Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import important library\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from IPython.core.interactiveshell import InteractiveShell\n",
    "# InteractiveShell.ast_node_interactivity = \"all\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Read input file and understand the data\n",
    "# \"default\" is my dependent variable\n",
    "df_pre = pd.read_csv(\"credit_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 21)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pre.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000 entries, 0 to 999\n",
      "Data columns (total 21 columns):\n",
      "default                       1000 non-null int64\n",
      "account_check_status          1000 non-null object\n",
      "duration_in_month             1000 non-null int64\n",
      "credit_history                1000 non-null object\n",
      "purpose                       1000 non-null object\n",
      "credit_amount                 1000 non-null int64\n",
      "savings                       1000 non-null object\n",
      "present_emp_since             1000 non-null object\n",
      "installment_as_income_perc    1000 non-null int64\n",
      "personal_status_sex           1000 non-null object\n",
      "other_debtors                 1000 non-null object\n",
      "present_res_since             1000 non-null int64\n",
      "property                      1000 non-null object\n",
      "age                           1000 non-null int64\n",
      "other_installment_plans       1000 non-null object\n",
      "housing                       1000 non-null object\n",
      "credits_this_bank             1000 non-null int64\n",
      "job                           1000 non-null object\n",
      "people_under_maintenance      1000 non-null int64\n",
      "telephone                     1000 non-null object\n",
      "foreign_worker                1000 non-null object\n",
      "dtypes: int64(8), object(13)\n",
      "memory usage: 164.2+ KB\n"
     ]
    }
   ],
   "source": [
    "df_pre.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>default</th>\n",
       "      <th>account_check_status</th>\n",
       "      <th>duration_in_month</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings</th>\n",
       "      <th>present_emp_since</th>\n",
       "      <th>installment_as_income_perc</th>\n",
       "      <th>personal_status_sex</th>\n",
       "      <th>...</th>\n",
       "      <th>present_res_since</th>\n",
       "      <th>property</th>\n",
       "      <th>age</th>\n",
       "      <th>other_installment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>credits_this_bank</th>\n",
       "      <th>job</th>\n",
       "      <th>people_under_maintenance</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>6</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>1169</td>\n",
       "      <td>unknown/ no savings account</td>\n",
       "      <td>.. &gt;= 7 years</td>\n",
       "      <td>4</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>real estate</td>\n",
       "      <td>67</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>yes, registered under the customers name</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0 &lt;= ... &lt; 200 DM</td>\n",
       "      <td>48</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>5951</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>female : divorced/separated/married</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>real estate</td>\n",
       "      <td>22</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>no checking account</td>\n",
       "      <td>12</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>(vacation - does not exist?)</td>\n",
       "      <td>2096</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>real estate</td>\n",
       "      <td>49</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled - resident</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>42</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>radio/television</td>\n",
       "      <td>7882</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>if not A121 : building society savings agreeme...</td>\n",
       "      <td>45</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>24</td>\n",
       "      <td>delay in paying off in the past</td>\n",
       "      <td>car (new)</td>\n",
       "      <td>4870</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>3</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>unknown / no property</td>\n",
       "      <td>53</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   default account_check_status  duration_in_month  \\\n",
       "0        0               < 0 DM                  6   \n",
       "1        1    0 <= ... < 200 DM                 48   \n",
       "2        0  no checking account                 12   \n",
       "3        0               < 0 DM                 42   \n",
       "4        1               < 0 DM                 24   \n",
       "\n",
       "                                      credit_history  \\\n",
       "0  critical account/ other credits existing (not ...   \n",
       "1           existing credits paid back duly till now   \n",
       "2  critical account/ other credits existing (not ...   \n",
       "3           existing credits paid back duly till now   \n",
       "4                    delay in paying off in the past   \n",
       "\n",
       "                        purpose  credit_amount                      savings  \\\n",
       "0           domestic appliances           1169  unknown/ no savings account   \n",
       "1           domestic appliances           5951                 ... < 100 DM   \n",
       "2  (vacation - does not exist?)           2096                 ... < 100 DM   \n",
       "3              radio/television           7882                 ... < 100 DM   \n",
       "4                     car (new)           4870                 ... < 100 DM   \n",
       "\n",
       "    present_emp_since  installment_as_income_perc  \\\n",
       "0       .. >= 7 years                           4   \n",
       "1  1 <= ... < 4 years                           2   \n",
       "2  4 <= ... < 7 years                           2   \n",
       "3  4 <= ... < 7 years                           2   \n",
       "4  1 <= ... < 4 years                           3   \n",
       "\n",
       "                   personal_status_sex  ... present_res_since  \\\n",
       "0                        male : single  ...                 4   \n",
       "1  female : divorced/separated/married  ...                 2   \n",
       "2                        male : single  ...                 3   \n",
       "3                        male : single  ...                 4   \n",
       "4                        male : single  ...                 4   \n",
       "\n",
       "                                            property age  \\\n",
       "0                                        real estate  67   \n",
       "1                                        real estate  22   \n",
       "2                                        real estate  49   \n",
       "3  if not A121 : building society savings agreeme...  45   \n",
       "4                              unknown / no property  53   \n",
       "\n",
       "   other_installment_plans   housing credits_this_bank  \\\n",
       "0                     none       own                 2   \n",
       "1                     none       own                 1   \n",
       "2                     none       own                 1   \n",
       "3                     none  for free                 1   \n",
       "4                     none  for free                 2   \n",
       "\n",
       "                           job people_under_maintenance  \\\n",
       "0  skilled employee / official                        1   \n",
       "1  skilled employee / official                        1   \n",
       "2         unskilled - resident                        2   \n",
       "3  skilled employee / official                        2   \n",
       "4  skilled employee / official                        2   \n",
       "\n",
       "                                   telephone foreign_worker  \n",
       "0  yes, registered under the customers name             yes  \n",
       "1                                       none            yes  \n",
       "2                                       none            yes  \n",
       "3                                       none            yes  \n",
       "4                                       none            yes  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pre.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['default', 'account_check_status', 'duration_in_month',\n",
       "       'credit_history', 'purpose', 'credit_amount', 'savings',\n",
       "       'present_emp_since', 'installment_as_income_perc',\n",
       "       'personal_status_sex', 'other_debtors', 'present_res_since', 'property',\n",
       "       'age', 'other_installment_plans', 'housing', 'credits_this_bank', 'job',\n",
       "       'people_under_maintenance', 'telephone', 'foreign_worker'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_pre.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>default</th>\n",
       "      <th>account_check_status</th>\n",
       "      <th>duration_in_month</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>savings</th>\n",
       "      <th>present_emp_since</th>\n",
       "      <th>installment_as_income_perc</th>\n",
       "      <th>personal_status_sex</th>\n",
       "      <th>...</th>\n",
       "      <th>present_res_since</th>\n",
       "      <th>property</th>\n",
       "      <th>age</th>\n",
       "      <th>other_installment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>credits_this_bank</th>\n",
       "      <th>job</th>\n",
       "      <th>people_under_maintenance</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>6</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>1169</td>\n",
       "      <td>unknown/ no savings account</td>\n",
       "      <td>.. &gt;= 7 years</td>\n",
       "      <td>4</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>real estate</td>\n",
       "      <td>67</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>yes, registered under the customers name</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0 &lt;= ... &lt; 200 DM</td>\n",
       "      <td>48</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>5951</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>female : divorced/separated/married</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>real estate</td>\n",
       "      <td>22</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>1</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>no checking account</td>\n",
       "      <td>12</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>(vacation - does not exist?)</td>\n",
       "      <td>2096</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>real estate</td>\n",
       "      <td>49</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled - resident</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>42</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>radio/television</td>\n",
       "      <td>7882</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>if not A121 : building society savings agreeme...</td>\n",
       "      <td>45</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>24</td>\n",
       "      <td>delay in paying off in the past</td>\n",
       "      <td>car (new)</td>\n",
       "      <td>4870</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>3</td>\n",
       "      <td>male : single</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>unknown / no property</td>\n",
       "      <td>53</td>\n",
       "      <td>none</td>\n",
       "      <td>for free</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>2</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   default account_check_status  duration_in_month  \\\n",
       "0        0               < 0 DM                  6   \n",
       "1        1    0 <= ... < 200 DM                 48   \n",
       "2        0  no checking account                 12   \n",
       "3        0               < 0 DM                 42   \n",
       "4        1               < 0 DM                 24   \n",
       "\n",
       "                                      credit_history  \\\n",
       "0  critical account/ other credits existing (not ...   \n",
       "1           existing credits paid back duly till now   \n",
       "2  critical account/ other credits existing (not ...   \n",
       "3           existing credits paid back duly till now   \n",
       "4                    delay in paying off in the past   \n",
       "\n",
       "                        purpose  credit_amount                      savings  \\\n",
       "0           domestic appliances           1169  unknown/ no savings account   \n",
       "1           domestic appliances           5951                 ... < 100 DM   \n",
       "2  (vacation - does not exist?)           2096                 ... < 100 DM   \n",
       "3              radio/television           7882                 ... < 100 DM   \n",
       "4                     car (new)           4870                 ... < 100 DM   \n",
       "\n",
       "    present_emp_since  installment_as_income_perc  \\\n",
       "0       .. >= 7 years                           4   \n",
       "1  1 <= ... < 4 years                           2   \n",
       "2  4 <= ... < 7 years                           2   \n",
       "3  4 <= ... < 7 years                           2   \n",
       "4  1 <= ... < 4 years                           3   \n",
       "\n",
       "                   personal_status_sex  ... present_res_since  \\\n",
       "0                        male : single  ...                 4   \n",
       "1  female : divorced/separated/married  ...                 2   \n",
       "2                        male : single  ...                 3   \n",
       "3                        male : single  ...                 4   \n",
       "4                        male : single  ...                 4   \n",
       "\n",
       "                                            property age  \\\n",
       "0                                        real estate  67   \n",
       "1                                        real estate  22   \n",
       "2                                        real estate  49   \n",
       "3  if not A121 : building society savings agreeme...  45   \n",
       "4                              unknown / no property  53   \n",
       "\n",
       "   other_installment_plans   housing credits_this_bank  \\\n",
       "0                     none       own                 2   \n",
       "1                     none       own                 1   \n",
       "2                     none       own                 1   \n",
       "3                     none  for free                 1   \n",
       "4                     none  for free                 2   \n",
       "\n",
       "                           job people_under_maintenance  \\\n",
       "0  skilled employee / official                        1   \n",
       "1  skilled employee / official                        1   \n",
       "2         unskilled - resident                        2   \n",
       "3  skilled employee / official                        2   \n",
       "4  skilled employee / official                        2   \n",
       "\n",
       "                                   telephone foreign_worker  \n",
       "0  yes, registered under the customers name             yes  \n",
       "1                                       none            yes  \n",
       "2                                       none            yes  \n",
       "3                                       none            yes  \n",
       "4                                       none            yes  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Randomly select 50% data for this use case\n",
    "from sklearn.model_selection import train_test_split\n",
    "out_data,df = train_test_split(df_pre, train_size = 0.5, random_state = 5)\n",
    "df_pre.shape\n",
    "df_pre.columns\n",
    "df_pre.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>account_check_status</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>savings</th>\n",
       "      <th>present_emp_since</th>\n",
       "      <th>personal_status_sex</th>\n",
       "      <th>other_debtors</th>\n",
       "      <th>property</th>\n",
       "      <th>other_installment_plans</th>\n",
       "      <th>housing</th>\n",
       "      <th>job</th>\n",
       "      <th>telephone</th>\n",
       "      <th>foreign_worker</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>544</td>\n",
       "      <td>no checking account</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>car (new)</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>.. &gt;= 7 years</td>\n",
       "      <td>male : single</td>\n",
       "      <td>none</td>\n",
       "      <td>real estate</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>unskilled - resident</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>515</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>critical account/ other credits existing (not ...</td>\n",
       "      <td>car (new)</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>female : divorced/separated/married</td>\n",
       "      <td>none</td>\n",
       "      <td>if not A121 : building society savings agreeme...</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>none</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>193</td>\n",
       "      <td>no checking account</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>4 &lt;= ... &lt; 7 years</td>\n",
       "      <td>male : married/widowed</td>\n",
       "      <td>none</td>\n",
       "      <td>real estate</td>\n",
       "      <td>none</td>\n",
       "      <td>rent</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>business</td>\n",
       "      <td>... &lt; 100 DM</td>\n",
       "      <td>... &lt; 1 year</td>\n",
       "      <td>female : divorced/separated/married</td>\n",
       "      <td>none</td>\n",
       "      <td>if not A121 : building society savings agreeme...</td>\n",
       "      <td>none</td>\n",
       "      <td>rent</td>\n",
       "      <td>skilled employee / official</td>\n",
       "      <td>none</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>279</td>\n",
       "      <td>no checking account</td>\n",
       "      <td>existing credits paid back duly till now</td>\n",
       "      <td>domestic appliances</td>\n",
       "      <td>100 &lt;= ... &lt; 500 DM</td>\n",
       "      <td>1 &lt;= ... &lt; 4 years</td>\n",
       "      <td>male : single</td>\n",
       "      <td>none</td>\n",
       "      <td>if not A121/A122 : car or other, not in attrib...</td>\n",
       "      <td>none</td>\n",
       "      <td>rent</td>\n",
       "      <td>management/ self-employed/ highly qualified em...</td>\n",
       "      <td>yes, registered under the customers name</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    account_check_status                                     credit_history  \\\n",
       "544  no checking account  critical account/ other credits existing (not ...   \n",
       "515               < 0 DM  critical account/ other credits existing (not ...   \n",
       "193  no checking account           existing credits paid back duly till now   \n",
       "11                < 0 DM           existing credits paid back duly till now   \n",
       "279  no checking account           existing credits paid back duly till now   \n",
       "\n",
       "                 purpose              savings   present_emp_since  \\\n",
       "544            car (new)         ... < 100 DM       .. >= 7 years   \n",
       "515            car (new)         ... < 100 DM  4 <= ... < 7 years   \n",
       "193  domestic appliances         ... < 100 DM  4 <= ... < 7 years   \n",
       "11              business         ... < 100 DM       ... < 1 year    \n",
       "279  domestic appliances  100 <= ... < 500 DM  1 <= ... < 4 years   \n",
       "\n",
       "                     personal_status_sex other_debtors  \\\n",
       "544                        male : single          none   \n",
       "515  female : divorced/separated/married          none   \n",
       "193               male : married/widowed          none   \n",
       "11   female : divorced/separated/married          none   \n",
       "279                        male : single          none   \n",
       "\n",
       "                                              property  \\\n",
       "544                                        real estate   \n",
       "515  if not A121 : building society savings agreeme...   \n",
       "193                                        real estate   \n",
       "11   if not A121 : building society savings agreeme...   \n",
       "279  if not A121/A122 : car or other, not in attrib...   \n",
       "\n",
       "    other_installment_plans housing  \\\n",
       "544                    none     own   \n",
       "515                    none     own   \n",
       "193                    none    rent   \n",
       "11                     none    rent   \n",
       "279                    none    rent   \n",
       "\n",
       "                                                   job  \\\n",
       "544                               unskilled - resident   \n",
       "515                        skilled employee / official   \n",
       "193                        skilled employee / official   \n",
       "11                         skilled employee / official   \n",
       "279  management/ self-employed/ highly qualified em...   \n",
       "\n",
       "                                     telephone foreign_worker  \n",
       "544                                       none            yes  \n",
       "515                                       none             no  \n",
       "193                                       none            yes  \n",
       "11                                        none            yes  \n",
       "279  yes, registered under the customers name             yes  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Lets build a Ensemble model but need to modify the dataset first\n",
    "obj_df=df.select_dtypes(include=['object'])\n",
    "obj_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "dff=pd.get_dummies(df, drop_first=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500, 49)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dff.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "credit_amount                                            duration_in_month                                                             0.583234\n",
       "credit_history_existing credits paid back duly till now  credit_history_critical account/ other credits existing (not at this bank)   -0.670328\n",
       "savings_unknown/ no savings account                      savings_... < 100 DM                                                         -0.609828\n",
       "other_debtors_none                                       other_debtors_guarantor                                                      -0.690725\n",
       "housing_rent                                             housing_own                                                                  -0.756613\n",
       "job_unskilled - resident                                 job_skilled employee / official                                              -0.661808\n",
       "dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Lets check for highly correlated variables\n",
    "cor=dff.corr()\n",
    "cor.loc[:,:] = np.tril(cor,k=-1)\n",
    "cor=cor.stack()\n",
    "cor[(cor > 0.55) | (cor < -0.55)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop irrelavent data\n",
    "dff=dff.drop([\"foreign_worker_yes\",\"job_unskilled - resident\",\"housing_rent\"],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((350, 45), (150, 45), (350,), (150,))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Split Train/Test data 70:30 ratio\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "y = dff['default']\n",
    "X = dff.loc[:, dff.columns != 'default']\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split (X, y, test_size=0.3, random_state=42,)\n",
    "X_train.shape, X_test.shape, y_train.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Build RF Model\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rf = RandomForestClassifier(n_jobs=2,n_estimators=500,criterion=\"entropy\",random_state=9999)\n",
    "rfm = rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install imblearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn.utils\n",
    "from imblearn.over_sampling import SMOTE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "upsampler = SMOTE(random_state=7, k_neighbors=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_upsampled, y_train_upsampled = upsampler.fit_resample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>duration_in_month</th>\n",
       "      <th>credit_amount</th>\n",
       "      <th>installment_as_income_perc</th>\n",
       "      <th>present_res_since</th>\n",
       "      <th>age</th>\n",
       "      <th>credits_this_bank</th>\n",
       "      <th>people_under_maintenance</th>\n",
       "      <th>account_check_status_&lt; 0 DM</th>\n",
       "      <th>account_check_status_&gt;= 200 DM / salary assignments for at least 1 year</th>\n",
       "      <th>account_check_status_no checking account</th>\n",
       "      <th>...</th>\n",
       "      <th>other_debtors_none</th>\n",
       "      <th>property_if not A121/A122 : car or other, not in attribute 6</th>\n",
       "      <th>property_real estate</th>\n",
       "      <th>property_unknown / no property</th>\n",
       "      <th>other_installment_plans_none</th>\n",
       "      <th>other_installment_plans_stores</th>\n",
       "      <th>housing_own</th>\n",
       "      <th>job_skilled employee / official</th>\n",
       "      <th>job_unemployed/ unskilled - non-resident</th>\n",
       "      <th>telephone_yes, registered under the customers name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>36</td>\n",
       "      <td>8086</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>42</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>45</td>\n",
       "      <td>4746</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>1164</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>51</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>1391</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>745</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>529</td>\n",
       "      <td>23</td>\n",
       "      <td>3612</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>530</td>\n",
       "      <td>32</td>\n",
       "      <td>4011</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>531</td>\n",
       "      <td>10</td>\n",
       "      <td>1307</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>532</td>\n",
       "      <td>35</td>\n",
       "      <td>5167</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>29</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>533</td>\n",
       "      <td>10</td>\n",
       "      <td>626</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>534 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     duration_in_month  credit_amount  installment_as_income_perc  \\\n",
       "0                   36           8086                           2   \n",
       "1                   45           4746                           4   \n",
       "2                    8           1164                           3   \n",
       "3                    9           1391                           2   \n",
       "4                    9            745                           3   \n",
       "..                 ...            ...                         ...   \n",
       "529                 23           3612                           2   \n",
       "530                 32           4011                           3   \n",
       "531                 10           1307                           3   \n",
       "532                 35           5167                           3   \n",
       "533                 10            626                           4   \n",
       "\n",
       "     present_res_since  age  credits_this_bank  people_under_maintenance  \\\n",
       "0                    4   42                  4                         1   \n",
       "1                    2   25                  2                         1   \n",
       "2                    4   51                  2                         2   \n",
       "3                    1   27                  1                         1   \n",
       "4                    2   28                  1                         1   \n",
       "..                 ...  ...                ...                       ...   \n",
       "529                  3   30                  2                         1   \n",
       "530                  3   44                  1                         1   \n",
       "531                  3   27                  1                         1   \n",
       "532                  2   29                  1                         1   \n",
       "533                  1   26                  1                         1   \n",
       "\n",
       "     account_check_status_< 0 DM  \\\n",
       "0                              0   \n",
       "1                              0   \n",
       "2                              1   \n",
       "3                              0   \n",
       "4                              0   \n",
       "..                           ...   \n",
       "529                            0   \n",
       "530                            0   \n",
       "531                            0   \n",
       "532                            1   \n",
       "533                            0   \n",
       "\n",
       "     account_check_status_>= 200 DM / salary assignments for at least 1 year  \\\n",
       "0                                                    0                         \n",
       "1                                                    0                         \n",
       "2                                                    0                         \n",
       "3                                                    0                         \n",
       "4                                                    1                         \n",
       "..                                                 ...                         \n",
       "529                                                  0                         \n",
       "530                                                  0                         \n",
       "531                                                  0                         \n",
       "532                                                  0                         \n",
       "533                                                  0                         \n",
       "\n",
       "     account_check_status_no checking account  ...  other_debtors_none  \\\n",
       "0                                           0  ...                   1   \n",
       "1                                           0  ...                   1   \n",
       "2                                           0  ...                   1   \n",
       "3                                           0  ...                   1   \n",
       "4                                           0  ...                   1   \n",
       "..                                        ...  ...                 ...   \n",
       "529                                         0  ...                   0   \n",
       "530                                         0  ...                   1   \n",
       "531                                         0  ...                   0   \n",
       "532                                         0  ...                   1   \n",
       "533                                         0  ...                   1   \n",
       "\n",
       "     property_if not A121/A122 : car or other, not in attribute 6  \\\n",
       "0                                                    1              \n",
       "1                                                    0              \n",
       "2                                                    0              \n",
       "3                                                    0              \n",
       "4                                                    0              \n",
       "..                                                 ...              \n",
       "529                                                  0              \n",
       "530                                                  0              \n",
       "531                                                  0              \n",
       "532                                                  0              \n",
       "533                                                  0              \n",
       "\n",
       "     property_real estate  property_unknown / no property  \\\n",
       "0                       0                               0   \n",
       "1                       0                               0   \n",
       "2                       0                               1   \n",
       "3                       1                               0   \n",
       "4                       1                               0   \n",
       "..                    ...                             ...   \n",
       "529                     0                               0   \n",
       "530                     0                               0   \n",
       "531                     0                               0   \n",
       "532                     0                               0   \n",
       "533                     0                               0   \n",
       "\n",
       "     other_installment_plans_none  other_installment_plans_stores  \\\n",
       "0                               1                               0   \n",
       "1                               1                               0   \n",
       "2                               0                               0   \n",
       "3                               0                               0   \n",
       "4                               1                               0   \n",
       "..                            ...                             ...   \n",
       "529                             1                               0   \n",
       "530                             1                               0   \n",
       "531                             1                               0   \n",
       "532                             1                               0   \n",
       "533                             1                               0   \n",
       "\n",
       "     housing_own  job_skilled employee / official  \\\n",
       "0              1                                0   \n",
       "1              1                                0   \n",
       "2              0                                0   \n",
       "3              1                                1   \n",
       "4              1                                0   \n",
       "..           ...                              ...   \n",
       "529            1                                1   \n",
       "530            0                                1   \n",
       "531            1                                0   \n",
       "532            0                                1   \n",
       "533            1                                0   \n",
       "\n",
       "     job_unemployed/ unskilled - non-resident  \\\n",
       "0                                           0   \n",
       "1                                           0   \n",
       "2                                           0   \n",
       "3                                           0   \n",
       "4                                           0   \n",
       "..                                        ...   \n",
       "529                                         0   \n",
       "530                                         0   \n",
       "531                                         0   \n",
       "532                                         0   \n",
       "533                                         0   \n",
       "\n",
       "     telephone_yes, registered under the customers name   \n",
       "0                                                    1    \n",
       "1                                                    0    \n",
       "2                                                    1    \n",
       "3                                                    1    \n",
       "4                                                    0    \n",
       "..                                                 ...    \n",
       "529                                                  0    \n",
       "530                                                  0    \n",
       "531                                                  0    \n",
       "532                                                  0    \n",
       "533                                                  0    \n",
       "\n",
       "[534 rows x 45 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train_upsampled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    267\n",
       "0    267\n",
       "Name: default, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_upsampled.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds = rf.predict_proba(X_test)[:,1]\n",
    "y_pred = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate Confusion Matrix\n",
    "\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import accuracy_score\n",
    "def calculate_confusion_matrix(y_true, y_pred):\n",
    "    cm=confusion_matrix(y_true, y_pred)\n",
    "    print(cm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[96  0]\n",
      " [43 11]]\n",
      "0.7133333333333334\n"
     ]
    }
   ],
   "source": [
    "calculate_confusion_matrix(y_test, y_pred)\n",
    "print(accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1MAAAJcCAYAAAAcmf79AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd7hmVXn38e+PIl0IQgigOIoi0nUGFUGFiA2NooKEoIiNEAvRxBgTjWISE3gxdtEggVHgNUoVASkSulJmYGCGagL4EkUBlSqgwP3+sdeRh8NzyuwpZ8r3c13Pdfaz9ir3LjOce9bam1QVkiRJkqT5s8JUByBJkiRJSyOTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJGkZk+SWJA8kuW/gs9EC9rlzkv9dWDFOcsyZSf55cY45liQHJTlmquOQtGQxmZIkadn0J1W15sDnZ1MZTJKVpnL8BbE0xy5p0TKZkiRpOZLkRUl+mOSuJFcl2Xlg3zuSXJfk3iQ3JfnzVr4G8H1go8GZrtEzR6Nnr9oM2d8muRq4P8lKrd0JSe5IcnOSAycZ97Qk1WK8NcmvkxyQZPskV7fj+fJA/f2SXJzkS0nuTnJ9kpcP7N8oySlJfpXkv5O8Z2DfQUmOT3JMknuAA4C/B/Zqx37VeOdr8Fwk+esktye5Lck7BvavluTfkvykxXdRktUmcY32a2Pd287fPpM5f5IWDf+lRZKk5USSjYHTgLcBZwAvB05IsnlV3QHcDrwOuAl4KfD9JJdX1RVJXgMcU1VPHehvMsPuDbwWuBN4FPge8N1W/lTgB0luqKozJ3kYLwSe3eI7pR3HrsDKwJVJjquq8wfqHg+sB7wJODHJM6rqV8C3gGuAjYDNgbOT3FRV57S2bwD2BPYFVml9PKuq3joQy5jnq+3/I2BtYGPgFcDxSU6uql8DnwG2BF4M/LzF+uh41wj4DfBFYPuquiHJhsC6kzxvkhYBZ6YkSVo2ndxmNu5KcnIreytwelWdXlWPVtXZwCxgN4CqOq2q/qc65wNnAS9ZwDi+WFW3VtUDwPbA+lX1j1X126q6Cfg68Kfz0d8/VdWDVXUWcD/wraq6vap+ClwIPG+g7u3A56vqd1X1beAG4LVJngbsBPxt62sOcARdAjPiR1V1cjtPDwwLZBLn63fAP7bxTwfuA56TZAXgncBfVtVPq+qRqvphVT3EBNeILiHdKslqVXVbVV0zH+dO0kJmMiVJ0rJp96pap312b2VPB/YcSLLuoksqNgRI8pokl7Slb3fR/QK/3gLGcevA9tPplgoOjv/3wAbz0d8vBrYfGPJ9zYHvP62qGvj+E7qZqI2AX1XVvaP2bTxG3ENN4nz9sqoeHvj+mxbfesCqwP8M6XbMa1RV9wN70S07vC3JaW3GStIUMZmSJGn5cStw9ECStU5VrVFVBydZBTiBbvnZBlW1DnA6MLKWr4b0dz+w+sD3PxpSZ7DdrcDNo8Zfq6p2G9JuYdg4j1+LuAnws/ZZN8lao/b9dIy4n/B9EudrPHcCDwKbDtk35jUCqKozq+oVdAnw9XQze5KmiMmUJEnLj2OAP0nyqiQrJlm1vSjhqcCT6J4NugN4uD0j9cqBtr8AnpJk7YGyOcBuSdZN8kfABycY/zLgnvZSitVaDFsl2X6hHeHj/SFwYJKVk+wJPJduCd2twA+Bf23nYBvgXcCx4/T1C2BaW6IHE5+vMVXVo8CRwGfbizBWTLJDS9DGvEZJNkjy+nQvBHmIbtngI/N5TiQtRCZTkiQtJ1oS8Qa6pXV30M2C/A2wQlvydiDwHeDXwJ/RveBhpO31dC9tuKktP9sIOBq4CriF7nmhb08w/iPAnwDbATfTzdAcQfeShkXhUrqXVdwJfBrYo6p+2fbtDUyjm6U6Cfhkez5pLMe1n79McsVE52sSPgzMBS4HfgUcQncdxrxG7fPXLeZfAS8D3jsfY0payPL4pcSSJElLvyT7Ae+uqp2mOhZJyy5npiRJkiSpB5MpSZIkSerBZX6SJEmS1IMzU5IkSZLUw0pTHYCk5c96661X06ZNm+owJEmSJjR79uw7q2r9YftMpiQtdtOmTWPWrFlTHYYkSdKEkvxkrH0u85MkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqYaWpDkDS8mf2bEimOoplR9VURyBJ0vLJmSlJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmdIikeS8JDPms81+STZaWPWWdkmmJZm3CPt/fZKPLkD7+b7GkiRJyxKTqeVQkpWmOoYx7AdMJkmabD2No6pOqaqDpzoOSZKkpZXJ1FKozVhcn+QbSa5OcnyS1du+6UnOTzI7yZlJNmzl5yX5lyTnA3+ZZM8k85JcleSCVmfVJEclmZvkyiS7tPL9kpyY5IwkP07yfwZi+WqSWUmuSfKpSca/YpKZbfy5ST6UZA9gBnBskjlJVkvyiSSXt3qHpzOs3i1J1mt9z0hyXtt+Waszpx3PWmPEs2GSC1q9eUle0spfmeRHSa5IclySNZOsneSGJM9pdb6V5D3jHOt9SQ5p1+MHSV7QrsVNSV4/cD0vbONckeTFY5yzQ9v5uDrJn0/mXA+0PzDJta3tf7ay/ZJ8uW3PTPLFJD9sse3RyldIcli7vqcmOX1k36j+n3CuhtTZv90rs+CO+QlfkiRpiWQytfR6DnB4VW0D3AO8N8nKwJeAPapqOnAk8OmBNutU1cuq6t+ATwCvqqptgde3/e8DqKqtgb2BbyRZte3bDtgL2BrYK8nTWvnHqmoGsA3wsiTbTCL27YCNq2qrNtZRVXU8MAvYp6q2q6oHgC9X1fZVtRWwGvC6MeqN5cPA+6pqO+AlwFh1/ww4s9XbFpjTkrOPA7tW1fPbmH9VVXcD7wdmJvlT4A+q6uvjxLAGcF67HvcC/wy8Angj8I+tzu3AK9o4ewFfHNLPu4C7q2p7YHvgPUmeMbpSS3aGzdp9FHheu18OGCPWDYGdgNcBIzNWbwKm0V33dwM7DBlz6LkaXa+qDq+qGd39sv4YIUiSJC09ltTlXprYrVV1cds+BjgQOAPYCjg7CcCKwG0Dbb49sH0xXULwHeDEVrYTXTJGVV2f5CfAZm3fOS2RIMm1wNOBW4G3JNmf7l7aENgCuHqC2G8CnpnkS8BpwFlj1NslyUeA1YF1gWuA703Q96CLgc8mORY4sar+d4x6lwNHtmT05Kqak+Rl7VgubufyScCPAKrq7CR7Al+hS77G81u66wIwF3ioqn6XZC5dkgKwMvDlJNsBj/DYOR/0SmCbgVmhtYFnAzcPVqqq3caI42q62byTgZPHqHNyVT0KXJtkg1a2E3BcK/95knOHtHsRY5wrSZKkZZnJ1NKrhnwPcE1VPWH2oLn/95WrDkjyQuC1dDMx27X2Y3loYPsRYKU2M/JhYPuq+nWSmcCqwxo/LtCu7rbAq+hmw94CvHOwTpsROwyYUVW3JjlonL4f5rFZ1t/XqaqDk5wG7AZckmTXqrp+SDwXJHkp3bk4OsmhwK+Bs6tq79H1k6wAPJdupmtdYKwkDeB3VTVyrR6lnceqejSPPbv2IeAXdInZCsCDQ/oJ8IGqOnOcscbzWuCldLOQ/5BkyyF1Bq9xRv0cTxjjXEmSJC3LXOa39NokyUjStDdwEXADsP5IeZKVx/ilmSSbVtWlVfUJ4E7gacAFwD5t/2bAJq3PsTyZLkG7u81kvGYygbdlYStU1QnAPwDPb7vuBUaeaxpJiu5sz98MPqczWA/gFmB6237zqGOcW1WH0C0923yMeJ4O3N6W6/1Hi+cSYMckz2p1Vm/nBLrk5zq68z4yo7Ug1gZua7M/b6ObURztTOAvRsZKslmSNSbTeUv+nlZV5wIfAdYBnvBM0xguAt7cnp3aANh5SJ3xzpUkSdIyy5mppdd1wNuT/DvwY+CrVfXbtgzsi0nWpru+n6dbHjfaoUmeTTercA5wFXA98LW2BO1hYL+qeqgt3XqCqroqyZWt/5voltVNxsbAUe2XfIC/az9ntvEfoHs25+t0S+NuoVuKxxj1PgX8R5K/By4dqPfBdC/ReAS4Fvj+GPHsDPxNkt8B9wH7VtUdSfYDvpVklVbv4+1cvBt4QVXdm+7lHR8HPjnJYx/mMOCEtnTwXAZmEAccQbcs8Ip0QdwB7D66UpLTgXdX1c8GilcEjmn3RIDPVdVdY13XUU4AXg7MA26kO793D1YY61y1+pIkScusPLYCSUuLJNOAU9uLGaRFKsmaVXVfkqcAlwE7VtXPF6zPGdVNFmph8K9xSZIWnSSz2wvXnsCZKUkTOTXJOnQvlvinBU2kJEmSlhUmU0uhqrqF7q19S7wklwKrjCp+W1XNnYJYtgaOHlX8UFW9cCH0vcQc58JWVTtPdQySJElLIpMpLVILI1FZWFpis90i6nuJOU5JkiQtHr7NT5IkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBF1BIWuymT4dZ/m+mJEnSUs6ZKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sG3+Ula7GbPhmSqo1i2VE11BJIkLX+cmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKY0qyTpL3DnzfOcmpi2CcA5Ls26Pd4+KboO597ee0JPPmd6z5jGtakj9bgPb3Lcx4JEmStGiYTGk86wCTSlYmI8lKw8qr6mtV9c0eXS7U+BaiaUDvZEqSJElLB5Mp/V6Sv0oyr30+CBwMbJpkTpJDW7U1kxyf5PokxyZJazs9yflJZic5M8mGrfy8JP+S5HzgL8cY96AkHx6of0iSy5LcmOQlrXzLVjYnydVJnj06viRrJjknyRVJ5iZ5wwTHu1+Sk5N8L8nNSd7fzsGVSS5Jsm6rt2mSM9qxXZhk81Y+M8kXk/wwyU1J9mhdHwy8pMX1oXHG/m7r94YknxxSZ+jxtJmv65J8Pck1Sc5Kslrbd2CSa9s5+s9xjv2gJEe2831TkgMH9o2+DyYac+j5GTLm/klmJZkFd4x3aSRJkpYKQ2cKtPxJMh14B/BCIMClwFuBrapqu1ZnZ+B5wJbAz4CLgR2TXAp8CXhDVd2RZC/g08A7W/frVNXL5iOclarqBUl2Az4J7AocAHyhqo5N8iRgReCjo+JbCXhjVd2TZD3gkiSnVFWNM9ZW7ZhWBf4b+Nuqel6SzwH7Ap8HDgcOqKofJ3khcBjwx639hsBOwObAKcDxLa4PV9XrJjjOF7TxfwNcnuS0qpo1sP/BYcfT9j0b2Luq3pPkO8CbgWPa2M+oqoeSrDPB+JsDuwBrATck+SqwDaPug5YI/3qcMcc7P79XVYe3uiQzxrsmkiRJSwWTKY3YCTipqu4HSHIi8JIh9S6rqv9tdebQLWm7iy4pOLtNVK0I3DbQ5tvzGcuJ7efs1j/Aj4CPJXkqcGL7xX10uwD/kuSlwKPAxsAGwM/HGevcqroXuDfJ3cD3WvlcYJskawIvBo4bGG+VgfYnV9WjwLVJNpi/w+Tsqvol/P587wQMJlNjHQ/AzVU1p20PnqergWOTnAycPMH4p1XVQ8BDSW5vfY91H5wybMxJnB9JkqRllsmURjwhMxnDQwPbj9DdQwGuqaodxmhz/3zGMjLGSP9U1f9tM2CvBc5M8m7gplHt9gHWB6ZX1e+S3EI34zSZsaBLWB4a2F6JbinsXSOzXxO0n+w5HDF6dmb09/GOZ/R1WK1tvxZ4KfB64B+SbFlVD08i9sFrOZZhY050fiRJkpZZPjOlERcAuydZPckawBvplvGtNYm2NwDrJ9kBIMnKSbZcmMEleSZwU1V9kW6WZBvg3lHxrQ3c3hKPXYCnL+i4VXUPcHOSPVscSbLtBM1GxzWWVyRZtz17tDvd+R40X8eTZAXgaVV1LvARuhd0rDmJOAYNuw8uHKtyz/MjSZK0TDCZEgBVdQUwE7iM7nmpI6pqNnBxexHBoeO0/S2wB3BIkquAOXRLvxamvYB5bWnh5sA32xK5wfiOBWZ0LzhgH+D6hTT2PsC72rFdA4z7Ygu6pXYPJ7lqrBdQNBcBR9OdrxNGPS8F8388KwLHJJkLXAl8rqrumqDN44xxH1w5QbP5PT+SJEnLhIz/bL6kRSHJfsCMqnr/VMcyFboXUIzOHbUg/KtckqRFI8nsqpoxbJ8zU5IkSZLUgy+g0GKT5GPAnqOKj6uqT09FPItDklcBh4wqvrmq3ki3nG5Rj/8Onvj/97q4qt63qMeWJEla1rnMT9Ji5zK/hc+/yiVJWjRc5idJkiRJC5nJlCRJkiT1YDIlSZIkST34AgpJi9306TDLR6YkSdJSzpkpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknrw1eiSFrvZsyGZ6iiWPVVTHYEkScsXZ6YkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEpaSJKcl2RG2z49yTrt896e/f39wPa0JPPGqPePSXadz75nJtmjT1xD+rolyXoLoy9JkqSlicmUNI4kK/VpV1W7VdVdwDpAr2QK+PuJq0BVfaKqftBzDEmSJPVkMqXlXpJ9k1yd5KokR7dZm88mORc4JMkaSY5McnmSK5O8obVbLcl/trbfBlYb6HNktuZgYNMkc5IcmmTDJBe07/OSvGSMmA4GVmv1jm3FKyb5epJrkpyVZLVW9/ezTEkOTnJti+kzExz6rkkuTHJjkte19tNa2RXt8+JWvnObeTs+yfVJjk2SUTGvluSMJO8Z45j2TzIrySy4Y4LQJEmSlny9/tVdWlYk2RL4GLBjVd2ZZF3gs8BmwK5V9UiSfwH+q6remWQd4LIkPwD+HPhNVW2TZBvgiiFDfBTYqqq2a+P9NXBmVX06yYrA6sPiqqqPJnn/QLtpwLOBvavqPUm+A7wZOGbgWNYF3ghsXlXVYh3PNOBlwKbAuUmeBdwOvKKqHkzybOBbwIxW/3nAlsDPgIuBHYGL2r41gf8EvllV3xzjmA4HDu9inVETxCZJkrTEc2ZKy7s/Bo6vqjsBqupXrfy4qnqkbb8S+GiSOcB5wKrAJsBLaclMVV0NXD2J8S4H3pHkIGDrqrp3PmK9uarmtO3ZdMnQoHuAB4EjkrwJ+M0E/X2nqh6tqh8DNwGbAysDX08yFzgO2GKg/mVV9b9V9SgwZ9T43wWOGiuRkiRJWhaZTGl5F2DYLMn9o+q8uaq2a59Nquq6tm++Zliq6gK6JOynwNFJ9p2P5g8NbD/CqJnlqnoYeAFwArA7cMZE4Qz5/iHgF8C2dDNST5rk+BcDrxm99E+SJGlZZjKl5d05wFuSPAV+v1RutDOBD4wkCkme18ovAPZpZVsB2wxpey+w1siXJE8Hbq+qrwP/ATx/nNh+l2TlyR5IkjWBtavqdOCDwHYTNNkzyQpJNgWeCdwArA3c1maf3gasOMnhPwH8EjhssvFKkiQt7UymtFyrqmuATwPnJ7mK7nmp0f6Jbvnb1e315P/Uyr8KrJnkauAjwGVD+v8lcHF72cShwM7AnCRX0j3z9IVxwju8jXnsOHUGrQWc2uI5n26WaTw3tHrfBw6oqgfpkqG3J7mE7rmx+8dpP9oHgVWT/J/5aCNJkrTUSpXPgUtavLoXUMya6jCWOf51LknSwpdkdlXNGLbPmSlJkiRJ6sFXo0tTLMmlwCqjit9WVXMXQt8fA/YcVXxcVX16QfuWJEla3rnMT9Ji5zK/RcO/ziVJWvhc5idJkiRJC5nJlCRJkiT1YDIlSZIkST34AgpJi9306TDLR6YkSdJSzpkpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknrw1eiSFrvZsyGZ6iiWTVVTHYEkScsPZ6YkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZErLlCTTksybgnG3S7LbwPfXJ/noFMSxc5JTpzIGSZKk5cVKUx2Alh1JVqqqh6c6jimyHTADOB2gqk4BTpnKgJaEGCRJkpZlzkzp99qszvVJvpHk6iTHJ1m97bslyXpte0aS89r2QUkOT3IW8M0k+yX5bpIzktyQ5JMD/f9Vknnt88FWtkaS05Jc1cr3auXTk5yfZHaSM5NsOE7c01v7HwHvGyhfNclRSeYmuTLJLq18vyQnJ/lekpuTvL/FdmWSS5Ks2+pt2o5jdpILk2zeyvdssV6V5IIkTwL+EdgryZwke7Uxvtzqb5DkpFb/qiQvHnIMX00yK8k1ST41UH5LkkOSXNY+z2rlM5N8rcV1Y5LXDelzMIY/SXJpO8YfJNlg4PodmeS8JDclOXCg/b7tPrgqydGtbP0kJyS5vH12bOUva8c+p42x1pB49m/HOAvuGOtySpIkLTWcmdJozwHeVVUXJzkSeC/wmQnaTAd2qqoHkuwHvADYCvgNcHmS04AC3gG8EAhwaZLzgWcCP6uq1wIkWTvJysCXgDdU1R0twfo08M4xxj8K+EBVnZ/k0IHy9wFU1dYtEToryWZt31bA84BVgf8G/raqnpfkc8C+wOeBw4EDqurHSV4IHAb8MfAJ4FVV9dMk61TVb5N8AphRVe9vx7HfQBxfBM6vqjcmWRFYc8gxfKyqftX2n5Nkm6q6uu27p6pekGQkrpHEaRrwMmBT4NyRRGsMFwEvqqpK8m7gI8Bft32bA7sAawE3JPkqsBnwMWDHqrpzJMEEvgB8rqouSrIJcCbwXODDwPvafbMm8ODoAKrq8HZOSWbUOLFKkiQtFUymNNqtVXVx2z4GOJCJk6lTquqBge9nV9UvAZKcCOxEl0ydVFX3D5S/BDgD+EySQ4BTq+rCJFvRJTtnJwFYEbht2MBJ1gbWqarzW9HRwGva9k50SRlVdX2Sn9AlCQDnVtW9wL1J7ga+18rnAtu0hODFwHEtBoBV2s+LgZlJvgOcOMG5gS4B27fF8Qhw95A6b0myP92fyQ2BLYCRZOpbAz8/N9DmO1X1KPDjJDfRJUVjeSrw7TbD9yTg5oF9p1XVQ8BDSW4HNmgxH19Vd7a4f9Xq7gpsMXBOntxmoS4GPpvkWODEqvrfcWKRJElaJphMabTRMwYj3x/msWWhq46qc/8k+ghDVNWNSaYDuwH/mm654EnANVW1wyTizZDxBveN5aGB7UcHvj9K9+diBeCuqtpuSMwHtJmq1wJzkjyhzvxI8gy6mZ3tq+rXSWby+HNck9ge9n3Ql4DPVtUpSXYGDhrYN3guHqE7/rHO6wrADqOSZ4CD2wzkbsAlSXatquvHiUeSJGmp5zNTGm2TJCNJzN50y8MAbqFbzgfw5gn6eEWSdZOsBuxON2txAbB7ktWTrAG8EbgwyUbAb6rqGLoZsOcDNwDrj8SRZOUkWw4bqKruAu5OslMr2mdg9wUj39vyvk1a3xOqqnuAm5Ps2donybZte9OqurSqPgHcCTwNuJdumdww5wB/0dqumOTJo/Y/mS4hvbs9y/SaUfv3Gvj5o4HyPZOskGRTuuWS4x3b2sBP2/bbx6k3GPNbkjylxT2yzO8s4P0jlUYSyXZO5lbVIcAsxp8lkyRJWiaYTGm064C3J7kaWBf4aiv/FPCFJBfSzV6M5yK65XZzgBOqalZVXQHMBC4DLgWOqKorga2By5LMoXtG55+r6rfAHsAhSa5q/TzhpQ0D3gF8Jd0LKAZnTA4DVkwyF/g2sF9bzjZZ+wDvajFcA7yhlR+a7qUW8+gStquAc+mWv81pz3gN+ktglxbHbOBxiWFVXQVc2cY4ki75HLRKkktbPx8aKL8BOB/4Pt2zXU94TmnAQXRLFi+kSwDHVVXX0D2ndn47/s+2XQcCM9qLKa4FDmjlH0x7KQfdNfj+RGNIkiQt7VLlc+DqJJlG99zSVgvQx34MvIhBCybJLXTn885R5TPprtXxUxHXgupeQDFrqsNYJvlXuiRJC1eS2VU1Y9g+Z6YkSZIkqQdnprTUSPIVYMdRxV+oqqOmIh7158zUouNf6ZIkLVzjzUz5Nj8tNarqfRPXkiRJkhYPl/lJkiRJUg8mU5IkSZLUg8mUJEmSJPVgMiVJkiRJPfgCCkmL3fTpMMuX+UmSpKWcM1OSJEmS1IPJlCRJkiT1YDIlSZIkST2YTEmSJElSDyZTkiRJktSDb/OTtNjNng3JVEexbKqa6ggkSVp+ODMlSZIkST2YTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg8mUJEmSJPVgMiVJkiRJPZhMSZIkSVIPJlOSJEmS1MO4yVSSdZK8dzIdJbmvbxBJbkmyXt/2y6okRyTZYpz9+yXZaBGOv0iuS5KZSfZYgPY7J3nxwupvYUjywSSrT2UMkiRJWrwmmplaB5hUMqWxJVmpT7uqendVXTtOlf2A+Uqm+sYylYbEvDPw4iFVp9IHgcWWTCVZcXGNJUmSpOEmSqYOBjZNMifJoQBJ/ibJ5UmuTvKpYY2G1UkyLcn1Sb7Ryo8f9S/5H0hyRZK5STZvbdZNcnKrf0mSbVr5QUmOTHJekpuSHDgw9luTXNZi/vexfulM8q4knxv4/p4knx2rj6vwj18AACAASURBVPaZmWRei/FD4524FuPhSc4CvtnaHzpwXv681VshyWFJrklyapLTR2ZZ2vHNGDZ2qzMDOLbFuVqS6UnOTzI7yZlJNhzo51+SnA/8ZZL1k5zQYrk8yY6t3lOSnJXkyiT/DmSMY7tvYHuPJDPb9swkX0zyw3ZdRo4jSb6c5NokpwF/ONB+UjEP1J8GHAB8qB33S9qul44et9WfzP366nbvXZXknIHr9+GBOvPaPbxGktNa3XlJ9mr330bAuUnObfX3btdqXpJDBs9dkkPa8f4gyQsG7uPXtzpj3Ss7Jzk3yf8F5g6LZcixndfGuyzJjSPnqx3Lhe24r0ib6WtjnJ/kO63+wUn2ae3nJtm01Rt6D40nyf5JZiWZBXdMVF2SJGnJV1VjfoBpwLyB768EDqf7JXsF4FTgpW3ffePVaX0VsGOrdyTw4bZ9C/CBtv1e4Ii2/SXgk237j4E5bfsg4IfAKsB6wC+BlYHnAt8DVm71DgP2HePY1gD+Z6DuD4Gtx+oDmA6cPdB+nQnO3UHAbGC19n1/4ONtexVgFvAMYA/g9Hau/gj4NbBHq3ceXcI0dOyR/W175XYM67fvewFHDtQ7bKD9/wV2atubANe17S8Cn2jbr23Xa70hx3bfwPYewMy2PRM4rh3LFsB/t/I3AWcDK9IlHXe1dpOOeci5/fDA97HGHfN+HWi7PnAr8Iz2fd0xxphHdw+/Gfj6QPnaA/fwem17I+D/tb5XAv4L2L3tK+A1bfsk4Kx2Hrblsft7rHtlZ+D+gViHxjLq+M4D/q1t7wb8oG2vDqzatp8NzGrbO7frs2Eb+6fAp9q+vwQ+P949NNkPTC8oP4vgI0mSFq6R35OGfeZ3ydcr2+fK9n3N9ovYBZOo8/+AW6vq4lZ+DHAg8Jn2/cT2czbdL98AO9H9wkhV/VebOVm77Tutqh4CHkpyO7AB8HK6xOPyJACrAbcPO5Cquj/JfwGvS3IdXfI0N8n7x+jje8Azk3wJOI3ul+CJnFJVDwycl20GZk3WbudlJ+C4qnoU+PnIzMYoN01i7OcAWwFnt7hXBG4b2P/tge1dgS1aPYAnJ1mLLul9E0BVnZbk15M4xtFObsdybZINWtlLgW9V1SPAz9p5n9+Y+4w7mfv1RcAFVXUzQFX9aoJx5gKfabNNp1bVhUPqbA+cV1V3ACQ5lu4cnAz8FjhjoK+Hqup3SebSJWsjcQ+7V34LXDYS6yRjgcf/2RoZY2Xgy0m2Ax4BNhuof3lV3dZi/x8eu9/mAru07aH3UFXdO0YMkiRJy5z5TaYC/GtV/fv81mnLs2pU3cHvD7WfjwzENWyZ2UibhwbKRtoE+EZV/d048Q06Avh74HrgqIExh/aRZFvgVcD7gLcA75yg//sHm9PNvp05qs/XThRkVf16EmMHuKaqdphELCsAOwwkeiOxwBOv0dCQBrZXHbVv8LoMXr9h/c5PzBMZNu5k79dhsT3M45fBrgpQVTcmmU43y/OvSc6qqn8c0udYftf+hQPg0ZG4q+rRPPZs2Fj3ys4MnJNJxgLD/2x9CPgF3YzYCsCDQ+o/Lsa2PdJ+6D0kSZK0PJnomal7gbUGvp8JvDPJmgBJNk7yh6PajFdnkyQjvzjvDVw0wfgXAPu0fnYG7qyqe8apfw6wx8h46Z65evpYlavqUuBpwJ8B3xqvj3RvtVuhqk4A/gF4ftv//jabNZEzgb9IsnJrt1mSNejOwZvTPTu1Ad0yq8cZa2wef31uANYfOb9JVk6y5RixnAX8PuY2OwGPP9+vAf5gjPa/SPLcJCsAb5zwyLt+/7Q9C7Qhj81uzE/Mg0bfl2OZzP36I+BlSZ7R6qzbym/hsWv8fLpldqR7e+JvquoYulnVYdfi0tbneume2dsbOH8S8Q7GPexeeZxxYpmMtYHb2mze2+hmBefHWPeQJEnScmPcmamq+mWSi5PMA75fVX+T5LnAj9osxn3AWxlYSldVZ41R5xHgOuDt6V5u8GPgqxPEdxBwVJKrgd8Ab58g3muTfBw4q/2i/zu6mZyfjNPsO8B2VfXrCfp4oMUykoCOzFxtDlzMxI6gW2J1RboTcwewO3AC3fLEecCNdL+I3z2q7cZjjD0T+FqSB4Ad6J5D+mJbCrkS8HngmiGxHAh8pZ3XleiSnQOATwHfSnIF3S///2+MY/ko3fNHt7a415zg2E+ie+ZtbjvG8wGq6rdtKdtkYh70PeD4JG8APjBWpXHuxcH79Y4k+wMntvN7O/AKuuuyb5I5wOUtbuieqzs0yaN098ZftPLDge8nua2qdknyd8C5dLNMp1fVdyc4pkFj3SujjRXLZBwGnJBkzxbn/MwCwtj3kCRJ0nIjj604WsQDdcv8Tq2qrRbLgJOU5FTgc1V1zgK0f1NV/XYBYlizqu5L8hTgMrqXdPy8b3/Ski6ZUd17NbSwLaa/0iVJWm4kmV1VM4btW+r+n0MLS5J16BKXq/omUgBV9bqFEM6pLZ4nAf9kIiVJkiQt+RZbMlVVt9C9uW2xS3Ip3WueB72tqjYbVn9xq6qdpzoGSZIkSfNnuZiZqqoXTnUMkiRJkpYtE73NT5IkSZI0hMmUJEmSJPVgMiVJkiRJPSwXz0xJWrJMnw6zfDO6JElayjkzJUmSJEk9mExJkiRJUg8mU5IkSZLUg8mUJEmSJPVgMiVJkiRJPZhMSZIkSVIPvhpd0mI3ezYkUx2FxlM11RFIkrTkc2ZKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKb0BEl+OM6+nZOcuoD9n5dkxpDy05Os07bvaz+nJZk3n/3PTLLHgsS4JI83auwNk5w1H/UPTHJdkmOTrJLkB0nmJNkryRFJthin7euTfHSC/vdL8uX5OQZJkqSl1UpTHYCWPFX14ikad7epGHcp92rgzPmo/17gNVV1c5IXAStX1XZt37fHa1hVpwCn9AtTkiRp2ePMlJ4gyX3pHJpkXpK5SfYaqPLkJCcluTbJ15IMvY+SrNhmbUb6+NCo/Ssk+UaSf27fb0my3jhxrdhiujzJ1Un+vJUnyZdbPKcBfzhG+02TnJFkdpILk2zeymcm+WqSc5PclORlSY5sMzgzR52Xf0tyRZJzkqw/ZIyXJ7myHe+Rbfbn5UlOGqjziiQntu1XJvlR6/O4JGu28ulJzm+xnplkwzFOy6uB7w+J46/aeZ+X5IOt7GvAM4FTkvwtcAywXZuZ2nRwxjDJq1tMVyU5p5X9ftYpyZ8kubQd6w+SbDDWdRuIaf8ks5LMgjsmqi5JkrTEM5nSWN4EbAdsC+wKHDrwC/0LgL8GtgY2bXWH2Q7YuKq2qqqtgaMG9q0EHAvcWFUfn2RM7wLurqrtge2B9yR5BvBG4DktnvcAY82sHQ58oKqmAx8GDhvY9wfAHwMfAr4HfA7YEtg6ycjMzRrAFVX1fOB84JODnSdZFZgJ7NWOdyXgL4D/Ap47kHy9AziqJY4fB3Ztfc4C/irJysCXgD1arEcCnx59MElWBJ5TVdeOKp/exngh8KJ2np5XVQcAPwN2qapDgHcDF1bVdlX1PwPt1we+Dry5qrYF9hxyLi8CXlRVzwP+E/jIkDqPU1WHV9WMqpoBT8hDJUmSljou89NYdgK+VVWPAL9Icj5dAnMPcFlV3QSQ5Fut7vFD+rgJeGaSLwGnAYPP9vw78J2qekKSMI5XAtvkseeT1gaeDbx0INafJfmv0Q3bjM+LgeOSjBSvMlDle1VVSeYCv6iqua3dNcA0YA7wKI8thTsGOHHUMM8Bbq6qG9v3bwDvq6rPJzkaeGuSo4AdgH3pZpW2AC5uMT0J+FHrZyvg7Fa+InDbkPPxQuDSIeU7ASdV1f3tGE4EXgJcOaTuMC8CLqiqmwGq6ldD6jwV+HZLsJ8E3DzJviVJkpYZJlMaS8bZVxN87wqrfp1kW+BVwPuAtwDvbLt/COyS5N+q6sH5iOkDVfW4Z4SS7DZWDANWAO4aeD5otIfaz0cHtke+j/XnZPSY452zo+hmvB4Ejquqh9NlSmdX1d6P6yTZGrimqnYYpz+A1wBnDCkfL47JCBOfzy8Bn62qU5LsDBy0gGNKkiQtdVzmp7FcAOzVnlNan27257K27wVJntGeldqLbsnXE7RlbCtU1QnAPwDPH9j9H8DpdDNFk03qzwT+oi2DI8lmSdZosf5pi3VDYJfRDavqHuDmJHu2tmmJ3vxYARiZFfsznnjc1wPTkjyrfX8b3XJAqupndEvsPk63FBDgEmDHkfpJVk+yGXADsH6SHVr5ykm2HBLPy4FzhpRfAOze+luDbhnkhfNxnD8CXtaWUJJk3SF11gZ+2rbfPh99S5IkLTOcmdIwBZxEtxztqvb9I1X18/bShh8BB9M9o3RBqzvMxnTPBo0k7X/3uEGqPptkbeDoJPtMIq4j6JbcXdFmde4Adm/j/zEwF7iRlsAMsQ/w1SQfB1ame9bnqkmMO+J+YMsks4G76RLJweN5MMk7eCxBvBz42kCVY4H1R55xqqo7kuwHfCvJyJLDj1fVjW0p4xfb+VkJ+DxwzUhHLcF9sCWJj1NVV7QXZ4wkv0dU1WSX+I3EtT9wYrt2twOvGFXtoHacP6VLCp8x2f4lSZKWFamaaDWPlidJnkL3koWnT3UsS5ok91XVmgvQ/svAlVX1HwshlrcCT62qgxe0r6mQzKjufRtaUvmfBkmSOklmdy/QeiJnpvR7STYCzgM+M8WhLHPabNb9dG9BXGBVdczC6EeSJEn9mUzp99pzPZv1aZvkUh7/djyAt428FW9ZsCCzUu0V55IkSVqGmExpoaiqF051DJIkSdLi5Nv8JEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB58AYWkxW76dJjl/2ZKkiQt5ZyZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHnybn6TFbvZsSKY6Ci0uVVMdgSRJi4YzU5IkSZLUg8mUJEmSJPVgMiVJkiRJPZhMSZIkSVIPJlOSJEmS1IPJlCRJkiT1YDIlSZIkST2YTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJy5Ekxyd55iLqe+skMxdF35IkSUsikylpEpKsNNUxzI90VhhVtiWwYlXdtCjGrKq5wFOTbLIo+pckSVrSmExpuZBkWpLrk3wjydVthmb1tu+WJOu17RlJzmvbByU5PMlZwDeT7Jfku0nOSHJDkk8O9P9XSea1zwdb2RpJTktyVSvfq5VPT3J+ktlJzkyy4ThxPyvJD1ofVyTZNMmaSc5p3+cmecPAMV6X5DDgCuBpo7rbB/juQN/3Jfl06/uSJBu08vWTnJDk8vbZsZXPTbJOS9R+mWTfVn50kl1bt98D/nSMY9k/yawks+COyVw2SZKkJZrJlJYnzwEOr6ptgHuA906izXTgDVX1Z+37C+iSku2APVvyNR14B/BC4EXAe5I8D3g18LOq2raqtgLOSLIy8CVgj6qaDhwJfHqc8Y8FvlJV2wIvBm4DHgTeWFXPB3YB/i1JBo7xm1X1vKr6yai+dgRmD3xfA7ik9X0B8J5W/gXgc1W1PfBm4IhWfnHrY0vgJuAlrfxFwCVte9ZA+eNU1eFVNaOqZsD64xyyJEnS0mGpWrokLaBbq+ritn0McCDwmQnanFJVDwx8P7uqfgmQ5ERgJ6CAk6rq/oHylwBnAJ9JcghwalVdmGQrYCvg7Jb/rEiXID1BkrWAjavqJICqerCVrwz8S5KXAo8CGwMbtGY/qapLhvUHbMjjp4R+C5zatmcDr2jbuwJbPJaf8eQWy4XAS4GfAF8F9k+yMfCrqrqv1b0d2GiM8SVJkpYpJlNantQY3x/msVnaVUfVuX8SfYQhqurGNmu1G/CvbbngScA1VbXDJOId2i/dzNj6wPSq+l2SWwbiHh3voAd4/PH9rqpGjucRHvv7YAVgh1FJJEkuAN4HbAJ8DHgjsAddkjVi1TaOJEnSMs9lflqebJJkJInZG7iobd9Ct5wPumVt43lFknWTrAbsTrf07QJg9ySrJ1mDLsm4MMlGwG+q6hi6GbDnAzcA64/EkWTl9mKIJ6iqe4D/TbJ7q7tKe85rbeD2lkjtAjx9ksd/HfCsSdQ7C3j/yJck27V4bgXWA57dXmJxEfBhHp9MbQbMm2Q8kiRJSzWTKS1PrgPenuRqYF26pWoAnwK+kORCuhma8VwEHA3MAU6oqllVdQUwE7gMuBQ4oqquBLYGLksyh24m55+r6rd0szmHJLmq9fPiccZ7G3Bgi/mHwB/RPUc1o3uRA/sA10/y+E8Ddp5EvQNb/1cnuRY4YGDfpcCNbftCuiWGFw3s36WNI0mStMzLY6t8pGVXkml0zy1ttQB97AfMqKr3T1R3SdRm084FdqyqiZLGPv2vApwP7FRVD49fd0Z176rQ8sD/zEiSlmZJZncv0HoiZ6ak5UR7BuqTdLNJi8ImwEcnSqQkSZKWFc5MSUuAJF+he+34oC9U1VFTEc+i5szU8sX/zEiSlmbjzUz5Nj9pCVBV75vqGCRJkjR/XOYnSZIkST2YTEmSJElSDyZTkiRJktSDz0xJWuymT4dZvn9CkiQt5ZyZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHX40uabGbPRuSqY5CU6FqqiOQJGnhcWZKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKa03EpyRJItpjqOBZVkzyTXJHk0yYxR+/4uyX8nuSHJqwbKX93K/jvJR8fod2aSm5NcleTGJN9MsvHA/luSXDiqzZwk8xb2MUqSJC2JTKa03Kqqd1fVtVMdx2QleVKSNYbsmge8CbhgVP0tgD8FtgReDRyWZMUkKwJfAV4DbAHsPU5S+TdVtS3wHOBK4NwkTxrYv1aSp7Xxntv/6CRJkpY+JlNa6iRZI8lpbcZkXpK9Wvknklz+/9u783DLivLu+9+fQGwmGYT4ihGbEHBgsLEbh4gIyIsaE4WAghoV9ZHgxEN8iWKMBMWoqInROIJRHEhAEBE1AjKJogzd0BMgotBRlPcRZBBEUOB+/lh1ZHM4U6/TfYbu7+e69tVr16qqda/ahz777qpVtLLj0nlikksH2s5NsrQdXzA0k5PkziT/3Pq8OMmjWvm27f1lSd6d5M5W/ugkFw7NxCR51mq83ycm+RfgGmD74eer6uqqumaEpi8CTqqqe6rqeuDHwFPb68dVdV1V/Q44qdUdVXU+DPz/dEnYkC8DB7bjlwL/NcZ9HJJkYZKFcNNYl5MkSZoVTKY0Gz0P+EVVPbmqdgTObOUfq6pdW9n6wF9W1dXAHyX501bnQLoEYLgNgYvbLMyFwOta+UeAj1TVrsAvBuq/DDirquYBTwYWr8L7G0oYX53ke8BngKuBnavqipXo5jHAzwbe39DKRiufiMuBJwy8P5VuVgzgr4Cvj9awqo6rqgVVtQC2nODlJEmSZi6TKc1Gy4C9kxyb5FlVdXsr3zPJJUmWAXvRLW+DLnl6STs+EDh5hD5/B3yjHS8C5rbjZwCntOP/HKh/GfDqJEcDO1XVHZO7pYe4EXgt8L+q6plV9Zke18gIZTVGeZ8+bwFuTXIQXcJ318TDkyRJmt1MpjTrVNWPgPl0SdX72vK+OcAngAOqaifgeGBOa3Iy8JIk23fN69oRuv19VQ0lFPcB644Tw4XA7sDPgS8meeVk72uYA1rfX23397gefdwAPHbg/Z/Qza6NVj4Ru9AlTYNOpnsGa9QlfpIkSWsikynNOkm2Au6qqi8BHwKewgOJ081JNqJLRgCoqp/QJUjvZORZqbFcDOzfjg8aiOFxwC+r6njgP1oMq0xVnV1VBwK7AbcDX0tyTpK5K9HNGcBBSR6eZBtgO+BSulm17ZJs0zaTOKjVHVV7/uww4NE8sKxyyFeBDwBnrURskiRJs57JlGajnYBLkywG3gG8p6puo5uNWgacTpcwDDoZ+BtGfl5qLIcDb2mbWDyaLrEB2ANYnOQKumTrI6N10LZgH9ro4tAkh7bjBUk+M9bFq+pXVfWR9mzWP9AlhcP73y/JDXRLEr+Z5KzW9kq6+72KLgF6Y1XdV1X3Am+iS36uBr7c6o7kg0mWAD8CdgX2bJtWDMZ4R1UdO7xckiRpTZcHVjZJGi7JBsBvq6rac0Evraoxd77T+JIFBQunOwxNA3/lSJJmmySLug20HmrM50IkMR/4WJIAtwGvmeZ4JEmSNEOYTEljqKrv0m19Pq4klwAPH1b8iqpatsoDkyRJ0rQzmZJWkap62nTHIEmSpKnjBhSSJEmS1IPJlCRJkiT1YDIlSZIkST34zJSkKTd/Pix0Z3RJkjTLOTMlSZIkST2YTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg9ujS5pyi1aBMl0RyGNr2q6I5AkzWTOTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg8mUJEmSJPVgMiVJkiRJPZhMSZIkSVIPJlOSJEmS1IPJlGaMJHOTvKxn2xVJtlgFMUyqnySHJ9lgVdVbXZLMT7IsyY+TfDRJRqhzdJKfJ1mc5NokpyV50sD5C5L8dLBtktOT3DlV9yFJkjSdTKY0k8wFeiVTM8jhwESSpInWm5Qkm41y6pPAIcB27fW8Uep9uKrmVdV2wMnAeUm2HDh/G/DMdq1NgUevksAlSZJmAZOptVybSViU5MokhwyUPy/J5UmWJDm3lW2U5HNtRmNpkv1b+Utb2fIkxw70cefA8QFJTmjHJ7TZkO8nuS7JAa3a+4FntZmQvxsl3nWSfGgghjcPnH5zi3lZkie0+hsm+WySy5JckeRFE+iHJOsnOTPJ60aJY8Mk32zjszzJgUkOA7YCzk9yfqv3ySQL2/i+q5WNVG+0sXpx639JkgtH/BAfGluS7JXkP4GFI5x/NPCIqvpBVRXwBWDf8fqtqpOBs3lwwnsScFA7/mvgtDHiOqSNxUK4aSK3IkmSNKOtO90BaNq9pqpuSbI+cFmSr9Al2ccDu1fV9Uk2b3XfCdxeVTtBN+uRZCvgWGA+cCtwdpJ9q+r0ca77aGA34AnAGcCpwJHAEVX1l2O0OwTYBtilqu4diA3g5qp6SpI3AEcA/wt4B3BeVb2mzZxcmuQc4JVj9LMRXZLwhar6wihxPA/4RVW9oI3FJlV1e5K3AHtW1c2t3jva+K4DnJtk56r66Aj1RnMU8Nyq+nmLf1TtszgYeAVwFfDZdjzcY4AbBt7f0Mom4nK6z2zIucDx7f4Oovt83jlSw6o6Djiui3VBTfB6kiRJM5YzUzosyRLgYuCxdEu+ng5cWFXXA1TVLa3u3sDHhxpW1a3ArsAFVXVTVd0LnAjsPoHrnl5V91fVVcCjViLevYFPtWsNxgYPzIosolsyCLAPcGSSxcAFwBxg63H6+RrwuTESKYBlwN5Jjk3yrKq6fZR6L0lyOXAFsAPwpFHqjeYi4IQ2Q7bOaJWSPBX4Kd29Pauq9q+qb1bVfSNVH6FsosnN8Lb3Ad8DDgTWr6oVE+xHkiRp1jOZWosl2YMuqXhGVT2Z7gv/HLovzCN9uR6pfKQv5kMG684Zdu6eCfYxkRiG93kfD8y6Bti/Pfczr6q2rqqrx+nnIuD5I23KMKSqfkQ3G7cMeF+Sox4SaLIN3QzZc6pqZ+CbPHQc/tDlwPEf6lTVocA/0iW6i5M8cpT2S4HX0iVrX0vyuiSPGKXuDcCfDLz/E+AXo9Qdbhfg6mFlJwH/Dnx5gn1IkiStEUym1m6bALdW1V3tGaOnt/IfAM9uyQADS+DOBt401LhtbnBJq7tFW+r1UuA7rcr/SfLEJA8D9ptAPHcAG49T52zg0CTrDottNGfRPUuVVn+XCfRzFPAr4BOjddqW1N1VVV8CPgQ8ZYR7eATwG+D2JI8Cnj/QxfB7HXGskmxbVZdU1VHAzXRJ1UNU1d1V9fmq2p1uqd+2wBVJvjhC3RuBO5I8vY3LK+lm48bUnpHbB/ivYae+C7xvhHJJkqQ1msnU2u1MYN0kS4Fj6Jb6UVU30T37clpbAnhyq/8eYLOhDRHonvm5EXg7cD6wBLi8qoa+mB8JfAM4D7hxAvEsBe5tmy2MuAEF8Bm65WxLWwzj7f53DLBeq7+8vZ9IP4cDc5J8YJR+d6J7/mox3XNZ72nlxwHfSnJ+VS2hm+27ku75pYsG2v+hXns/2lh9sG2SsRy4kG6Mx1RV11bVkcDj6Z5FG8nr6cbgx8BPgG+NUu/v2oYg1wJ/A+zVfj4Gr1dV9aEJPP8lSZK0Rkm3mZckTZ1uA4qHbDQozTj+ipQkJVlUVQtGOufMlCRJkiT14NbompGSPJduy/VB11fVRJ69WpVxPJJu++/hnlNVv5rKWFo8M2JcJEmS5DI/SdPAZX6aLfwVKUlymZ8kSZIkrWImU5IkSZLUg8mUJEmSJPVgMiVJkiRJPbibn6QpN38+LHT/CUmSNMs5MyVJkiRJPZhMSZIkSVIPJlOSJEmS1IPJlCRJkiT1YDIlSZIkST24m5+kKbdoESTTHYXUT9V0RyBJmimcmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKE5bk+z3b7ZvkSROod3SSI9rxCUkO6HO9lYjr4CRbreZrvDDJkavzGpIkSZoeJlOasKr6855N9wXGTaamwcHAs4o/sgAAIABJREFUak2mquqMqnr/6rzGdEuy7nTHIEmSNB1MpjRhSe5sf+6R5IIkpyb5YZITk6Sde3+Sq5IsTfKhJH8OvBD4YJLFSbZN8roklyVZkuQrSTYY57orkrw3yQ+SLEzylCRnJflJkkMH6v1963dpkne1srlJrk5yfJIrk5ydZP0267UAOLHFtf4o1z6q9bk8yXED93nYwH2eNEbsByf5WDs+IclHk3w/yXWDM29J3ppkWRuT97eyeUkubtf4apLNWvkFST6c5MJ2b7smOS3JtUneM9Dn3yS5tN3fp5OsM9Znm+Rfklye5NwkW7bybZOcmWRRku8mecLAvfxrkvOBY5NslORz7R6WJtl/hGsc0j6/hXDTaKFIkiTNGiZT6msX4HC6Gac/BZ6ZZHNgP2CHqtoZeE9VfR84A/j7qppXVT8BTquqXavqycDVwGsncL2fVdUzgO8CJwAHAE8H3g2QZB9gO+CpwDxgfpLdW9vtgI9X1Q7AbcD+VXUqsBB4eYvrt6Nc92Mt1h2B9YG/bOVHAru0+zx0lLYjeTSwW+tnKGl6Pt3s3dPamHyg1f0C8LZ2jWXAPw3087uq2h34FPA14I3AjsDBSR6Z5InAgcAzq2oecB/w8jHi2hC4vKqeAnxn4FrHAW+uqvnAEcAnBtpsD+xdVf8f8E7g9qraqcV73vALVNVxVbWgqhbAluMOlCRJ0kzn8hz1dWlV3QCQZDEwF7gYuBv4TJJvAt8Ype2ObQZlU2Aj4KwJXO+M9ucyYKOqugO4I8ndSTYF9mmvK1q9jeiSqJ8C11fV4la+qMU6UXsmeSuwAbA5cCXwdWAp3azW6cDpK9Hf6VV1P3BVkke1sr2Bz1XVXQBVdUuSTYBNq+o7rc7ngVMG+hkcjyur6kaAJNcBj6VL2OYDl7XJtPWBX44R1/3Aye34S8BpSTYC/hw4pfUB8PCBNqdU1X0D93DQ0ImqunXMUZAkSVoDmEypr3sGju8D1q2qe5M8FXgO3RfrNwF7jdD2BGDfqlqS5GBgj5W43v3Drn0/3c9xgPdV1acHGyWZO0KsIy7pGy7JHLqZmAVV9bMkRwNz2ukXALvTLWF8Z5IdqurelbgPWsxDf9ZEYhqhn7HG4/NV9faV7HdI0c1c39Zmtkbym4HjPvcgSZI0q7nMT6tMm8nYpKr+m24J4NCX8DuAjQeqbgzcmGQ9xl56tjLOAl7TYiDJY5L88Ththsc13FDidHPr94DW98OAx1bV+cBbeWCGra+zW+wbtP43r6rbgVuTPKvVeQXd8ruJOhc4YGgMkmye5HFj1H8Y7f6AlwHfq6pfA9cneXHrI0mePMY9vGnozdDzXZIkSWsyZ6a0Km0MfK3N6AT4u1Z+EnB8ksPovrC/E7gE+B+6ZWpjJTQTUlVnt+eEftCWpN0J/A3dTNRoTgA+leS3wDOGPzdVVbclOb7FuAK4rJ1aB/hSW4oX4MNVddskYj8zyTxgYZLfAf8N/APwqhbfBsB1wKtXos+rkvwjcHZL/n5P91zV/4zS5DfADkkWAbfTPW8FXbL7ydbXenSf5ZIR2r8H+HiS5XRj/i7gtInGK0mSNBulypU50touyZ1VNZnZtZW83oLq9v+QZh9/bUrS2iXJom4DrYdymZ8kSZIk9eAyPwlI8lVgm2HFb6uqiew0SJJXA/97WPFFVfXGVRHfqpLkEh68Ix/AK6ZyVkqSJGlN4TI/SVPOZX6azfy1KUlrF5f5SZIkSdIqZjIlSZIkST2YTEmSJElSD25AIWnKzZ8PC31kSpIkzXLOTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg1ujS5pyixZBMt1RSP1VTXcEkqSZwJkpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSplGSw5NsMAPi2CrJqdMdhyRJ0mxiMiWNIsk6U3CZw4EJJ1OrK6aq+kVVHbA6+pYkSVpTmUxprZRkbpIfJvl8kqVJTk2yQZIVSY5K8j3gxUm2TXJmkkVJvpvkCa39i5MsT7IkyYWtbJ0kH0xyWevzb1v5HkkuaNf4YZIT0zkM2Ao4P8n5Y8R6Z5J3J7kEeEaS+Um+02I6K8mjW73DklzVrn3SGP09O8ni9roiycZtPJa38wcnOa3d97VJPjDQ9nlJLm/3fW4r2zDJZ9t9X5HkRaNc95AkC5MshJtW7gOTJEmagdad7gCkafR44LVVdVGSzwJvaOV3V9VuAC1hOLSqrk3yNOATwF7AUcBzq+rnSTZt7V4L3F5VuyZ5OHBRkrPbuV2AHYBfABcBz6yqjyZ5C7BnVd08RpwbAsur6qgk6wHfAV5UVTclORD4Z+A1wJHANlV1z0BMIzkCeGO7742Au0eoM6/FfA9wTZJ/b/WOB3avquuTbN7qvgM4r6pe0657aZJzquo3gx1W1XHAcd24Lqgx4pMkSZoVTKa0NvtZVV3Ujr8EHNaOTwZoicafA6ckGWrz8PbnRcAJSb4MnNbK9gF2TjK0XG4TYDvgd8ClVXVD63cxMBf43gTjvA/4Sjt+PLAj8O0W0zrAje3cUuDEJKcDp4/R30XAvyY5ETitqm4YuL8h51bV7S3eq4DHAZsBF1bV9QBVdcvAfb8wyRHt/Rxga+DqCd6fJEnSrGQypbXZ8NmRofdDMyoPA26rqnkPaVh1aJupegGwOMk8IMCbq+qswbpJ9qCb4RlyHyv3397dVXXfUHfAlVX1jBHqvQDYHXgh8M4kO1TVvSPE/v4k3wT+Arg4yd48dHZqpHjDQ8dsKKb9q+qalbgnSZKkWc9nprQ22zrJUFLyUobNFFXVr4Hrk7wYoD3n9OR2vG1VXVJVRwE3A48FzgJe35bikWT7JBuOE8MdwMYrEfM1wJZDcSdZL8kOSR4GPLaqzgfeCmwKbDRSBy32ZVV1LLAQeMIEr/0D4NlJtmn9DC3zOwt4c9r0VpJdVuJ+JEmSZi2TKa3NrgZelWQpsDnwyRHqvBx4bZIlwJXA0OYKH0yyrG3acCGwBPgMcBVweSv/NOPPQB0HfGusDSgGVdXvgAOAY1tMi+mWIq4DfCnJMuAK4MNVddso3Rw+tHkG8FvgWxO89k3AIcBpre3J7dQxwHrA0nbfx0ykP0mSpNkuVT4HrrVPkrnAN6pqx2kOZa3UbUCxcLrDkHrzV6ckrT2SLKqqBSOdc2ZKkiRJknpwAwqtlapqBd2ueDNG+/9IPXxY8SuqalnP/l4N/O9hxRdV1Rv79CdJkqQHc5mfpCnnMj/Ndv7qlKS1h8v8JEmSJGkVM5mSJEmSpB5MpiRJkiSpB5MpSZIkSerB3fwkTbn582Gh+09IkqRZzpkpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwd38JE25RYsgme4opNmrarojkCSBM1OSJEmS1IvJlCRJkiT1YDIlSZIkST2YTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg8mUJEmSJPVgMjUDJJmb5GU9265IssUqiGFS/SQ5PMkGq6redEpycJKPra5+khya5JWT7X8mSvIP0x2DJEnSVDGZmhnmAr2SqRnkcGAiSdJE662xqupTVfWF6Y5jNTGZkiRJa421NplKcnqSRUmuTHLIQPnzklyeZEmSc1vZRkk+l2RZkqVJ9m/lL21ly5McO9DHnQPHByQ5oR2fkOSjSb6f5LokB7Rq7weelWRxkr8bJd51knxoIIY3D5x+c4t5WZIntPobJvlsksuSXJHkRRPohyTrJzkzyetGiWPDJN9s47M8yYFJDgO2As5Pcn6r98kkC9v4vquVjVRvtLF6cet/SZILR/wQ+cPsz2kt5muTfGDg3Iifz7D2u7bPY0mSS5Ns3E5tNUqf+yT5QRvvU5JsNE4/Q+1e0NptkeToJEe08guSHNva/CjJs1r5Bkm+3D6jk5NckmTBCPEf1T7j5UmOS5JW/mdJzmnxXJ5k21b+1jYmS5K8v5XNS3Jxu9ZXk2w2ENuCdrxFkhVjjXnrb/32c3ziaJ+ZJEnSGqOq1soXsHn7c31gOfBIYEvgZ8A2w+ocC/zbQNvN6JKCn7Y26wLnAfu283cO1D0AOKEdnwCcQpfEPgn4cSvfA/jGOPG+HvgKsO6w2FYAb27HbwA+047fC/xNO94U+BGw4Tj9zAXOAV45Rhz7A8cPvN9koP0WI4zvOsAFwM6j1BttrJYBjxmKf4x4DgauAzYB5gD/Azx2rM9noO0ftba7tvePaHVH63ML4EJgw1b/bcBR4/TzMWA/4LvAZu380cAR7fgC4F/a8V8A57TjI4BPt+MdgXuBBaP9HLfjLwJ/1Y4vAfZrx3PoZgOfD3wf2GDYZ7QUeHY7fjftZ73FtqAdbwGsGGvMh3+eI8R6CLCwe21dUL58+er5kiRNHWBh1cjfb9bamSngsCRLgIvpvihvBzwduLCqrgeoqlta3b2Bjw81rKpbgV2BC6rqpqq6FzgR2H0C1z29qu6vqquAR61EvHsDn2rXGowN4LT25yK6hAhgH+DIJIvpvhTPAbYep5+vAZ+rsZegLQP2brMpz6qq20ep95IklwNXADvQJY8r4yLghDZDts44dc+tqtur6m7gKuBxTOzzeTxwY1VdBlBVvx4al1H6fHq7j4vauL6qlY/Vz550SdcL2s/NSEb6/HYDTmr9LadLeEayZ5u1WgbsBezQZsUeU1Vfbe3vrqq76D77z7VjquqWJJvQJavfaf19foRxGslI4zOmqjquqhZU1YIux5UkSZrd1spkKskedF8sn1FVT6b7wj8HCFAjNRmhPGNcYrDunGHn7plgHxOJYXif99HNiAzV37+q5rXX1lV19Tj9XAQ8f2ip2Eiq6kfAfLqk6n1JjnpIoMk2dDMrz6mqnYFv8tBx+EOXA8d/qFNVhwL/SJfoLk7yyNFi4sFjOjQGExnbiYzp8D6/PTCmT6qq147Tz3XAxsD2E4h/+Oc3dvDJHOATwAFVtRNwPA/8HI/YZIw4R3IvD/wdMdbP8WDckiRJa421MpmiW550a1Xd1Z4xenor/wHw7JYMkGTzVn428Kahxu2Zkkta3S2SrAO8FBj61/3/k+SJSR5Gt8RrPHfQfeEey9nAoUnWHRbbaM6ie5Zq6BmaXSbQz1HAr+i+oI8oyVbAXVX1JeBDwFNGuIdHAL8Bbk/yKLrlZYxQD0YZqyTbVtUlVXUUcDNdUrUyxvp8hvyQ7tmoXds1Nx4al1FcDDwzyZ+1+hsk2X6cfv4H+GvgC0l2WIn4vwe8pPX3JGCnEeoMJTg3t2e3DoBuZgy4Icm+rf3D0+2geDbwmnZMks3bzOKtQ89qAa/ggXFaQZc4M9T3BPw+yXoTrCtJkjSrra3J1JnAukmWAsfQfUmmqm6ie67jtLYE8ORW/z3AZu0h/yXAnlV1I/B24HxgCXB5VX2t1T8S+Abdczo3TiCepcC9bVOAETegAD5D9wzQ0hbDeLv/HQOs1+ovb+8n0s/hwJzBTReG2Qm4tC1zewfd2AAcB3wryflVtYRutu9K4LN0M14Mr9fejzZWH2wbJSyne05pyTj3+yDjfD5DdX4HHAj8exuLbzP6DNrQz8fBwH+1n52LgSeM109VXQO8HDhlaCOICfgEsGW7ztvofkYetKSyqm6jm41aBpwOXDZw+hV0S1mX0j0n9f9U1ZnAGcDC9vkd0eq+im68lwLz6J6bgi5Zfn2S79M9MzURx9H9bLkBhSRJWuOle6ZK0kzSZtPWq6q7WwJ2LrB9S9xmvWRBdXtRSOrDX92SNHWSLOqe+X4on3OQZqYN6LaQX4/uWafXrymJlCRJ0prCZGqGSfJcuq3YB11fVRN59mpVxvFIutmQ4Z5TVb+aylhaPDNiXKZKVd0BjPgvIJIkSZoZXOYnacq5zE+aHH91S9LUGWuZ39q6AYUkSZIkTYrJlCRJkiT1YDIlSZIkST2YTEmSJElSD+7mJ2nKzZ8PC91/QpIkzXLOTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg1ujS5pyixZBMt1RSJI0cVXTHYFmImemJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKa6UkRyc5YhX0s2mSNwy83yrJqZPtd9g1/jvJpquyz8lIcnCSrQber0iyxXTGJEmSNB1MpqRxJFl3jNObAn9IpqrqF1V1wKq8flX9RVXdtir7nKSDga3GqyRJkrSmM5nSWiPJO5Jck+Qc4PGt7IIkC9rxFklWtOODk5yS5OvA2Uk2SnJuksuTLEvyotbt+4FtkyxO8sEkc5Msb33MSfK5Vv+KJHsO9H1akjOTXJvkA+PEvaLFNjfJ1UmOT3JlkrOTrD9GuwuSfDjJha3dru261yZ5z0C9tyRZ3l6Ht7IRr5XkAGABcGK756Hrv3lgbJ4wSjyHJFmYZCHcNPaHJUmSNAuYTGmtkGQ+cBCwC/DXwK4TaPYM4FVVtRdwN7BfVT0F2BP4lyQBjgR+UlXzqurvh7V/I0BV7QS8FPh8kjnt3DzgQGAn4MAkj53grWwHfLyqdgBuA/Yfp/7vqmp34FPA11pMOwIHJ3lkG5dXA08Dng68Lskuo12rqk4FFgIvb/f821b35jY2nwRGXD5ZVcdV1YKqWgBbTvB2JUmSZi6TKa0tngV8taruqqpfA2dMoM23q+qWdhzgvUmWAucAjwEeNU773YAvAlTVD4H/AbZv586tqtur6m7gKuBxE7yP66tqcTteBMwdp/7QfS4DrqyqG6vqHuA64LEtxq9W1W+q6k7gNLqxWtlrnbYSMUmSJK0RxnoWRFrT1Ahl9/LAPyrMGXbuNwPHL6ebTplfVb9vywGH1x8uY5y7Z+D4Pib+3+LwdqMu8xtW//5hbe9v11yZGMe61j0D9fx7RZIkrRWcmdLa4kJgv/bcz8bAX7XyFcD8djzWxhGbAL9sidSePDCTdAew8RjXfDlAku2BrYFret/B6nEhsG+SDZJsCOwHfHecNmPdsyRJ0lrDZEprhaq6HDgZWAx8hQcShg8Br0/yfWCs7b1PBBZ0myfwcuCHrd9fARe1zRs+OKzNJ4B1kixr1z64LbGbMdq4nABcClwCfKaqrhin2QnAp4ZtQCFJkrTWSdVIK58kafVJFlS3j4UkSbODX5nXXkkWdRtoPZQzU5IkSZLUgw+KSzNEkkuAhw8rfkVVLRun3ceBZw4r/khVfW5VxidJkqQHM5mSZoiqelrPdm9c1bFIkiRpfC7zkyRJkqQeTKYkSZIkqQeTKUmSJEnqwWemJE25+fNhoTujS5KkWc6ZKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6cGt0SVNu0SJIpjsKSZLWfFXTHcGazZkpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSZoAkFyRZ0I7/O8mm7fWG6Y5topIcnGSr6Y5DkiRpqphMSatJknX7tKuqv6iq24BNgVmTTAEHAyZTkiRprWEyJU1CklcmWZpkSZIvJjkhyb8mOR84NsmGST6b5LIkVyR5UWu3fpKTWtuTgfUH+lyRZAvg/cC2SRYn+eAo198oyblJLk+ybKD/uUl+mOQzSZYnOTHJ3kkuSnJtkqe2epsnOb3FcXGSnVv50UmOGLjO8tbn3CRXJzk+yZVJzm73cgCwADixxbv+CLEekmRhkoVw06r6CCRJkqZNr385lwRJdgDeATyzqm5Osjnwr8D2wN5VdV+S9wLnVdVrkmwKXJrkHOBvgbuqaueWwFw+wiWOBHasqnljhHE3sF9V/bolYBcnOaOd+zPgxcAhwGXAy4DdgBcC/wDsC7wLuKKq9k2yF/AFYKzrAWwHvLSqXpfky8D+VfWlJG8CjqiqhSM1qqrjgOO6sVtQ41xDkiRpxjOZkvrbCzi1qm4GqKpbkgCcUlX3tTr7AC8cmOWZA2wN7A58tLVbmmRpzxgCvDfJ7sD9wGOAR7Vz11fVMoAkVwLnVlUlWQbMbXV2A/ZvcZyX5JFJNhnnmtdX1eJ2vGigL0mSpLWKyZTUX4CRZlh+M6zO/lV1zYMadknXqpideTmwJTC/qn6fZAVdwgZwz0C9+wfe388D/+1nhD4LuJcHLwOeM3A82O99DCxRlCRJWpv4zJTU37nAS5I8Errnj0aocxbw5rTsKckurfxCukSIJDsCO4/Q9g5g43Fi2AT4ZUuk9gQet5L3MBjHHsDNVfVrYAXwlFb+FGCbCfQ1kXglSZLWGM5MST1V1ZVJ/hn4TpL7gCtGqHYM8G/A0pZQrQD+Evgk8Lm2vG8xcOkI/f+qbRixHPhWVf39CP2fCHy929SBxcAPV/I2jh6I4y7gVa38K8Arkyyme97qRxPo6wTgU0l+Czyjqn67krFIkiTNKqnyOXBJU6vbgGLEfSokSdIq5Ff9yUuyqKoWjHTOZX6SJEmS1IPL/KRZIMlOwBeHFd9TVU+bjngkSZJkMiXNCm2L8/H+/0+SJEmaQi7zkyRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknpwAwpJU27+fFjo/2ZKkiTNcs5MSZIkSVIPJlOSJEmS1IPJlCRJkiT1YDIlSZIkST2YTEmSJElSD+7mJ2nKLVoEyXRHIUmSZquq6Y6g48yUJEmSJPVgMiVJkiRJPZhMSZIkSVIPJlOSJEmS1IPJlCRJkiT1YDIlSZIkST2YTEmSJElSDyZTkiRJktSDyZQkSZIk9WAyJUmSJEk9mExJkiRJUg8mU5IkSZLUg8mUpIdIcnqSRUmuTHJIK3ttkh8luSDJ8Uk+1sq3TPKVJJe11zOnN3pJkqSpse50ByBpRnpNVd2SZH3gsiTfBN4JPAW4AzgPWNLqfgT4cFV9L8nWwFnAE4d32JKyQ7p3W6/2G5AkSVrdTKYkjeSwJPu148cCrwC+U1W3ACQ5Bdi+nd8beFKSobaPSLJxVd0x2GFVHQcc17VfUKs5fkmSpNXOZErSgyTZgy5BekZV3ZXkAuAaRphtah7W6v52aiKUJEmaGXxmStJwmwC3tkTqCcDTgQ2AZyfZLMm6wP4D9c8G3jT0Jsm8KY1WkiRpmphMSRruTGDdJEuBY4CLgZ8D7wUuAc4BrgJub/UPAxYkWZrkKuDQqQ9ZkiRp6rnMT9KDVNU9wPOHlydZWFXHtZmpr9LNSFFVNwMHTm2UkiRJ08+ZKUkTdXSSxcBy4Hrg9GmOR5IkaVo5MyVpQqrqiOmOQZIkaSZxZkqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknpwNz9JU27+fFi4cLqjkCRJmhxnpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5MpiRJkiSpB5MpSZIkSerBZEqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknowmZIkSZKkHkymJEmSJKkHkylJkiRJ6sFkSpIkSZJ6MJmSJEmSpB5SVdMdg6S1TJI7gGumO4411BbAzdMdxBrKsV19HNvVx7FdfRzb1Wemje3jqmrLkU6sO9WRSBJwTVUtmO4g1kRJFjq2q4dju/o4tquPY7v6OLarz2waW5f5SZIkSVIPJlOSJEmS1IPJlKTpcNx0B7AGc2xXH8d29XFsVx/HdvVxbFefWTO2bkAhSZIkST04MyVJkiRJPZhMSZIkSVIPJlOSVqkkz0tyTZIfJzlyhPMPT3JyO39JkrkD597eyq9J8typjHs26Du2Sf7fJIuSLGt/7jXVsc90k/m5bee3TnJnkiOmKubZYpJ/J+yc5AdJrmw/v3OmMvaZbhJ/J6yX5PNtTK9O8vapjn2mm8DY7p7k8iT3Jjlg2LlXJbm2vV41dVHPDn3HNsm8gb8PliY5cGojH0VV+fLly9cqeQHrAD8B/hT4I2AJ8KRhdd4AfKodHwSc3I6f1Oo/HNim9bPOdN/TTHlNcmx3AbZqxzsCP5/u+5lJr8mM7cD5rwCnAEdM9/3MpNckf27XBZYCT27vH+nfCatsbF8GnNSONwBWAHOn+55mymuCYzsX2Bn4AnDAQPnmwHXtz83a8WbTfU8z5TXJsd0e2K4dbwXcCGw63ffkzJSkVempwI+r6rqq+h1wEvCiYXVeBHy+HZ8KPCdJWvlJVXVPVV0P/Lj1p07vsa2qK6rqF638SmBOkodPSdSzw2R+bkmyL90XpiunKN7ZZDJjuw+wtKqWAFTVr6rqvimKezaYzNgWsGGSdYH1gd8Bv56asGeFcce2qlZU1VLg/mFtnwt8u6puqapbgW8Dz5uKoGeJ3mNbVT+qqmvb8S+AXwJbTk3YozOZkrQqPQb42cD7G1rZiHWq6l7gdrp/cZ5I27XZZMZ20P7AFVV1z2qKczbqPbZJNgTeBrxrCuKcjSbzc7s9UEnOakt+3joF8c4mkxnbU4Hf0P3L/k+BD1XVLas74FlkMr+P/F02tlUyPkmeSjez9ZNVFFdv6053AJLWKBmhbPj/f2G0OhNpuzabzNh2J5MdgGPp/sVfD5jM2L4L+HBV3dkmqvRgkxnbdYHdgF2Bu4BzkyyqqnNXbYiz1mTG9qnAfXRLpTYDvpvknKq6btWGOGtN5vcm90lfAAAGEklEQVSRv8vGNunxSfJo4IvAq6pq+MzglHNmStKqdAPw2IH3fwL8YrQ6bYnJJsAtE2y7NpvM2JLkT4CvAq+sqmn/l7wZZjJj+zTgA0lWAIcD/5DkTas74Flksn8nfKeqbq6qu4D/Bp6y2iOePSYzti8Dzqyq31fVL4GLgAWrPeLZYzK/j/xdNrZJjU+SRwDfBP6xqi5exbH1YjIlaVW6DNguyTZJ/ojugeczhtU5Axja3egA4LzqniY9Azio7T61DbAdcOkUxT0b9B7bJJvS/fJ5e1VdNGURzx69x7aqnlVVc6tqLvBvwHur6mNTFfgsMJm/E84Cdk6yQUsEng1cNUVxzwaTGdufAnulsyHwdOCHUxT3bDCRsR3NWcA+STZLshndSoCzVlOcs1HvsW31vwp8oapOWY0xrhSTKUmrTFuT/ya6XxxXA1+uqiuTvDvJC1u1/6B71uTHwFuAI1vbK4Ev031ZOhN4ow+bP2AyY9va/RnwziSL2+uPp/gWZqxJjq3GMMm/E24F/pXuy9di4PKq+uZU38NMNcmf248DGwHL6cb3c+2BfzGxsU2ya5IbgBcDn05yZWt7C3AM3bheBrzb59EeMJmxBV4C7A4cPPC7bN403MaDpG0vKEmSJElaCc5MSZIkSVIPJlOSJEmS1IPJlCRJkiT1YDIlSZIkST2YTEmSJElSDyZTkiSJJPe1rYaXJ/l6+/+TjdfmznHOb5rkDQPvt0py6iqIdW6S5ZPtZyWvOS/JX0zlNSXNfCZTkiQJ4LdVNa+qdgRuAd64CvrcFPhDMlVVv6iqA1ZBv1Oq/U+D5wEmU5IexGRKkiQN9wPgMUNvkvx9ksuSLE3yruGVk2yU5NwklydZluRF7dT7gW3bjNcHB2eUklySZIeBPi5IMj/Jhkk+2653xUBfI0pycJLT22za9UnelOQtre3FSTYf6P/fkny/zb49tZVv3tovbfV3buVHJzkuydnAF4B3Awe2ezkwyVNbX1e0Px8/EM9pSc5Mcm2SDwzE+rw2RkuSnNvKVup+Jc0s6053AJIkaeZIsg7wHOA/2vt9gO2ApwIBzkiye1VdONDsbmC/qvp1ki2Ai5OcARwJ7FhV81pfcwfanAS8BPinJI8GtqqqRUneC5xXVa9pSw0vTXJOVf1mjLB3BHYB5gA/Bt5WVbsk+TDwSuDfWr0Nq+rPk+wOfLa1exdwRVXtm2QvusRpXqs/H9itqn6b5GBgQVW9qd3LI4Ddq+reJHsD7wX2b+3mtXjuAa5J8u9tjI5vba4fSvKAd/S4X0kzhMmUJEkCWD/JYmAusAj4divfp72uaO83okuuBpOpAO9tScr9dLNajxrnel9u1/gnuqTqlIHrvTDJEe39HGBr4Oox+jq/qu4A7khyO/D1Vr4M2Hmg3n8BVNWFSR7RkpfdaElQVZ2X5JFJNmn1z6iq345yzU2AzyfZDihgvYFz51bV7QBJrgIeB2wGXFhV17dr3TKJ+5U0Q5hMSZIkaM9MtUTiG3TPTH2ULlF6X1V9eoy2Lwe2BOZX1e+TrKBLCkZVVT9P8qu2rO5A4G/bqQD7V9U1KxH7PQPH9w+8v58Hf9ep4WG06z0kvPbnWLNDx9Alcfu1GbcLRonnvhZDRrg+9LtfSTOEz0xJkqQ/aDMqhwFHJFkPOAt4TZKNAJI8JskfD2u2CfDLlkjtSTcTA3AHsPEYlzsJeCuwSVUta2VnAW9Okna9XVbFfTUHtj53A25v93ohXTJIkj2Am6vq1yO0HX4vmwA/b8cHT+DaPwCenWSbdq2hZX6r834lrWYmU5Ik6UGq6gpgCXBQVZ0N/CfwgyTLgFN5aIJ0IrAgyUK6xOSHrZ9fARe1DR8+OMKlTgUOolvyN+QYuiVzS9tmFcesujvj1iTfBz4FvLaVHd1iX0q3YcarRml7PvCkoQ0ogA8A70tyEbDOeBeuqpuAQ4DTkiwBTm6nVuf9SlrNUjXSjLMkSdKaI8kFwBFVtXC6Y5G05nBmSpIkSZJ6cGZKkiRJknpwZkqSJEmSejCZkiRJkqQeTKYkSZIkqQeTKUmSJEnqwWRKkiRJknr4v7mnmD/sJzBaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# View a list of the features and their importance scores\n",
    "importances = rf.feature_importances_\n",
    "indices = np.argsort(importances)[::-1][:15]\n",
    "a = dff.columns[:]\n",
    "features= a.drop('default',1)\n",
    "#plot it\n",
    "plt.figure(figsize=(10,10))\n",
    "plt.title('Feature Importances')\n",
    "plt.barh(range(len(indices)), importances[indices], color='b', align='center')\n",
    "plt.yticks(range(len(indices)), features[indices])\n",
    "plt.xlabel('Relative Importance')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainResult = rf.score(X_train, y_train)\n",
    "testResult = rf.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100.0"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Train Accuracy:\"\n",
    "(trainResult*100.0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71.33333333333334"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"Test Accuracy:\"\n",
    "(testResult*100.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# K-fold cross-validation\n",
    "\n",
    "k-fold cross validation( without stratification)\n",
    "\n",
    "Usually k is set as 10-20 in practical settings, depends on data set size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_folds = 10\n",
    "seed = 77"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=num_folds, random_state=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.68518519, 0.72222222, 0.7962963 , 0.81481481, 0.81132075,\n",
       "       0.79245283, 0.94339623, 1.        , 1.        , 0.98113208])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = cross_val_score(rfm, X_train_upsampled, y_train_upsampled, cv=kfold)\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8546820405310973"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(abs(results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_folds = 10\n",
    "seed = 77"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=num_folds, random_state=seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.68518519, 0.72222222, 0.7962963 , 0.81481481, 0.81132075,\n",
       "       0.79245283, 0.94339623, 1.        , 1.        , 0.98113208])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = cross_val_score(rfm, X_train_upsampled, y_train_upsampled, cv=kfold)\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8546820405310973"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(abs(results))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.110959538023544"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Leave One Out Cross-Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import LeaveOneOut\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A single Validation set is used to reduce overfitting\n",
    "# Many validation sets is used to reduce sampling noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "       0., 0., 0., 0., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 0., 0., 1., 0., 1.,\n",
       "       1., 0., 0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 1., 0., 1., 0., 1.,\n",
       "       0., 0., 1., 1., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 0., 1., 0., 1., 0., 1., 1., 1., 0., 0., 1., 1., 0.,\n",
       "       0., 1., 0., 1., 1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0.,\n",
       "       0., 1., 1., 0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 0., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1.,\n",
       "       1., 1., 1., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 0., 1., 0., 1.,\n",
       "       1., 1., 1., 1., 1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 0., 1., 1., 0., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 1., 1., 1., 1., 1., 1.,\n",
       "       0., 0., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 0., 1., 1., 0., 1., 0., 0., 0., 0., 1., 1.,\n",
       "       1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 0., 1., 1., 1.,\n",
       "       1., 0., 1., 1., 0., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 0., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 1., 1., 1., 1., 1.,\n",
       "       1., 1., 1., 1., 1., 1., 1.])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You Will do it on X_train, y_train to save time. It will take too much time, hence not recommended for bigger data\n",
    "# LOOCV will be used when you cannot afford to leave out large sums of data\n",
    "# KFold is used on large dataset where you can offord to leave out large sums of data\n",
    "\n",
    "# LOOCV is like a KFold where K = N (number of data points)\n",
    "scores = cross_val_score(rfm, X_train_upsampled, y_train_upsampled, cv=LeaveOneOut())\n",
    "scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because we have 350 samples, the leave one out cross-validation yields scores for 350 trials, and the score indicates either \n",
    "defaulter (1.0) or non-defaulter (0.0) prediction. Taking the mean of these gives an estimate of the error rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8576779026217228"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.3493801911616131"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores.std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stratified cross-validation (Bonus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "k-fold cross validation with stratification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "k = 10\n",
    "from sklearn.model_selection  import StratifiedKFold, cross_val_score\n",
    "stratified_kfold = StratifiedKFold(n_splits = k, random_state = 55)\n",
    "results = cross_val_score(rfm, X_train_upsampled, y_train_upsampled, cv = stratified_kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Score\n",
      "Avearge:  0.8531446540880504\n",
      "Standard deviation:  0.1432258706744277\n"
     ]
    }
   ],
   "source": [
    "print('Accuracy Score')\n",
    "print('Avearge: ', results.mean())\n",
    "print('Standard deviation: ', results.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bootstrapping (Bonus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a dataset of size n, a bootstrap sample is created by sampling n instances uniformly from the data \n",
    "(with/without replacement)\n",
    "\n",
    "Create a model with each bootstrap sample and validate it with the test set\n",
    "\n",
    "Final result is calculated by averaging the accuracy of models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of iterations for bootstrapping\n",
    "bootstrap_iteration = 10\n",
    "accuracy = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.utils import resample\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "for i in range(bootstrap_iteration):\n",
    "    X_, y_ = resample(X_train_upsampled, y_train_upsampled)\n",
    "    rfm.fit(X_, y_)\n",
    "    y_pred = rfm.predict(X_test)\n",
    "    \n",
    "    acc = accuracy_score(y_pred, y_test)\n",
    "    accuracy.append(acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Score\n",
      "Avearge:  0.708\n",
      "Standard deviation:  0.011850925889754116\n"
     ]
    }
   ],
   "source": [
    "accuracy = np.array(accuracy)\n",
    "print('Accuracy Score')\n",
    "print('Avearge: ', accuracy.mean())\n",
    "print('Standard deviation: ', accuracy.std())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inference:\n",
    "\n",
    "Here crossfold validation with stratification gives better result than Bootstrapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}