ml_lesson_8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Decision Trees Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "We will be using the wine quality data set for these exercises. This data set contains various chemical properties of wine, such as acidity, sugar, pH, and alcohol. It also contains a quality metric (3-9, with highest being better) and a color (red or white). The name of the file is `Wine_Quality_Data.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import os\n",
    "data_path = ['../data/']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1\n",
    "\n",
    "* Import the data and examine the features.\n",
    "* We will be using all of them to predict `color` (white or red), but the colors feature will need to be integer encoded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "filepath = os.sep.join(data_path + ['Wine_Quality_Data.csv'])\n",
    "data = pd.read_csv(filepath, sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the color feature to an integer. This is a quick way to do it using Pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = data['color'].replace('white',1).replace('red',0).astype(np.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2\n",
    "\n",
    "* Use `StratifiedShuffleSplit` to split data into train and test sets that are stratified by wine quality. If possible, preserve the indices of the split for question 5 below.\n",
    "* Check the percent composition of each quality level for both the train and test data sets.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "feature_cols = [ cols for cols in data.columns if cols != 'color']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "\n",
    "sss = StratifiedShuffleSplit(n_splits = 1, test_size = 1000, random_state=42)\n",
    "train_idx , test_idx = next(sss.split(data[feature_cols],data['color']))\n",
    "\n",
    "X_train = data.loc[train_idx,feature_cols]\n",
    "Y_train = data.loc[train_idx,'color']\n",
    "\n",
    "X_test = data.loc[test_idx,feature_cols]\n",
    "Y_test = data.loc[test_idx,'color']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "white    0.753866\n",
       "red      0.246134\n",
       "Name: color, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "white    0.754\n",
       "red      0.246\n",
       "Name: color, dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Y_test.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed_acidity</th>\n",
       "      <th>volatile_acidity</th>\n",
       "      <th>citric_acid</th>\n",
       "      <th>residual_sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free_sulfur_dioxide</th>\n",
       "      <th>total_sulfur_dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4885</th>\n",
       "      <td>6.8</td>\n",
       "      <td>0.230</td>\n",
       "      <td>0.31</td>\n",
       "      <td>2.8</td>\n",
       "      <td>0.047</td>\n",
       "      <td>40.0</td>\n",
       "      <td>122.0</td>\n",
       "      <td>0.99126</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.64</td>\n",
       "      <td>10.9</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>6.4</td>\n",
       "      <td>0.170</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.8</td>\n",
       "      <td>0.034</td>\n",
       "      <td>33.0</td>\n",
       "      <td>125.0</td>\n",
       "      <td>0.99152</td>\n",
       "      <td>3.03</td>\n",
       "      <td>0.49</td>\n",
       "      <td>10.4</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>224</th>\n",
       "      <td>8.4</td>\n",
       "      <td>0.635</td>\n",
       "      <td>0.36</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.089</td>\n",
       "      <td>15.0</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0.99745</td>\n",
       "      <td>3.31</td>\n",
       "      <td>0.57</td>\n",
       "      <td>10.4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2655</th>\n",
       "      <td>7.5</td>\n",
       "      <td>0.210</td>\n",
       "      <td>0.34</td>\n",
       "      <td>1.2</td>\n",
       "      <td>0.060</td>\n",
       "      <td>26.0</td>\n",
       "      <td>111.0</td>\n",
       "      <td>0.99310</td>\n",
       "      <td>3.51</td>\n",
       "      <td>0.47</td>\n",
       "      <td>10.7</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4989</th>\n",
       "      <td>6.9</td>\n",
       "      <td>0.290</td>\n",
       "      <td>0.30</td>\n",
       "      <td>8.2</td>\n",
       "      <td>0.026</td>\n",
       "      <td>35.0</td>\n",
       "      <td>112.0</td>\n",
       "      <td>0.99144</td>\n",
       "      <td>3.00</td>\n",
       "      <td>0.37</td>\n",
       "      <td>12.3</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      fixed_acidity  volatile_acidity  citric_acid  residual_sugar  chlorides  \\\n",
       "4885            6.8             0.230         0.31             2.8      0.047   \n",
       "5568            6.4             0.170         0.30             2.8      0.034   \n",
       "224             8.4             0.635         0.36             2.0      0.089   \n",
       "2655            7.5             0.210         0.34             1.2      0.060   \n",
       "4989            6.9             0.290         0.30             8.2      0.026   \n",
       "\n",
       "      free_sulfur_dioxide  total_sulfur_dioxide  density    pH  sulphates  \\\n",
       "4885                 40.0                 122.0  0.99126  3.06       0.64   \n",
       "5568                 33.0                 125.0  0.99152  3.03       0.49   \n",
       "224                  15.0                  55.0  0.99745  3.31       0.57   \n",
       "2655                 26.0                 111.0  0.99310  3.51       0.47   \n",
       "4989                 35.0                 112.0  0.99144  3.00       0.37   \n",
       "\n",
       "      alcohol  quality  \n",
       "4885     10.9        7  \n",
       "5568     10.4        6  \n",
       "224      10.4        4  \n",
       "2655     10.7        6  \n",
       "4989     12.3        6  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3\n",
    "\n",
    "* Fit a decision tree classifier with no set limits on maximum depth, features, or leaves.\n",
    "* Determine how many nodes are present and what the depth of this (very large) tree is.\n",
    "* Using this tree, measure the prediction error in the train and test data sets. What do you think is going on here based on the differences in prediction error?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=42,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "tree_cls = DecisionTreeClassifier(random_state = 42)\n",
    "tree_cls = tree_cls.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_cls.tree_.max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "159"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_cls.tree_.node_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_fscore_support as score\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import label_binarize\n",
    "\n",
    "def measure_report(y_true,y_pred,lab):\n",
    "    precesion, recall, fscore,_ = score(y_true,y_pred,average='weighted')\n",
    "    accuracy = accuracy_score(y_true,y_pred)\n",
    "    return pd.Series({'precesion':precesion,\n",
    "                     'recall':recall,\n",
    "                      'accuracy':accuracy,\n",
    "                     'f1':fscore},name = lab)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.985000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.985109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precesion</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.985527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.985000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           train      test\n",
       "accuracy     1.0  0.985000\n",
       "f1           1.0  0.985109\n",
       "precesion    1.0  0.985527\n",
       "recall       1.0  0.985000"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "y_train_pred = tree_cls.predict(X_train)\n",
    "y_test_pred = tree_cls.predict(X_test)\n",
    "\n",
    "train_test_full_score = pd.concat([measure_report(Y_train,y_train_pred,'train'),\n",
    "                                   measure_report(Y_test,y_test_pred,'test')],axis=1)\n",
    "\n",
    "train_test_full_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4\n",
    "\n",
    "* Using grid search with cross validation, find a decision tree that performs well on the test data set. Use a different variable name for this decision tree model than in question 3 so that both can be used in question 6.\n",
    "* Determine the number of nodes and the depth of this tree.\n",
    "* Measure the errors on the training and test sets as before and compare them to those from the tree in question 3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "params_grid = {'max_depth':range(1,tree_cls.tree_.max_depth+1,2),\n",
    "              'max_features':range(1,len(tree_cls.feature_importances_)+1)\n",
    "              }\n",
    "\n",
    "grid_cv = GridSearchCV(\n",
    "                      DecisionTreeClassifier(random_state=42),\n",
    "                      param_grid = params_grid,\n",
    "                      scoring = 'accuracy',\n",
    "                      n_jobs=-1)\n",
    "\n",
    "grid_cv = grid_cv.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "129"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_cv.best_estimator_.tree_.node_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_cv.best_estimator_.tree_.max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_cv.best_estimator_.max_features_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  4.07844502e-02,   1.73914515e-02,   2.55050733e-03,\n",
       "         1.33314151e-02,   2.04879656e-01,   2.23371209e-03,\n",
       "         6.71137448e-01,   1.61568078e-02,   1.36748284e-02,\n",
       "         1.09450126e-02,   6.48578078e-03,   4.28929831e-04])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_cls.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>0.997999</td>\n",
       "      <td>0.988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>f1</th>\n",
       "      <td>0.997997</td>\n",
       "      <td>0.988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precesion</th>\n",
       "      <td>0.998002</td>\n",
       "      <td>0.988</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>0.997999</td>\n",
       "      <td>0.988</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              train   test\n",
       "accuracy   0.997999  0.988\n",
       "f1         0.997997  0.988\n",
       "precesion  0.998002  0.988\n",
       "recall     0.997999  0.988"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "y_train_pred_grd = grid_cv.predict(X_train)\n",
    "y_test_pred_grd = grid_cv.predict(X_test)\n",
    "\n",
    "train_test_full_score_grd = pd.concat([measure_report(Y_train,y_train_pred_grd,'train'),\n",
    "                                   measure_report(Y_test,y_test_pred_grd,'test')],axis=1)\n",
    "\n",
    "train_test_full_score_grd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Test data accuracy has been improved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5\n",
    "\n",
    "* Re-split the data into `X` and `y` parts, this time with `residual_sugar` being the predicted (`y`) data. *Note:* if the indices were preserved from the `StratifiedShuffleSplit` output in question 2, they can be used again to split the data.\n",
    "* Using grid search with cross validation, find a decision tree **regression** model that performs well on the test data set.\n",
    "* Measure the errors on the training and test sets using mean squared error.\n",
    "* Make a plot of actual *vs* predicted residual sugar.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed_acidity</th>\n",
       "      <th>volatile_acidity</th>\n",
       "      <th>citric_acid</th>\n",
       "      <th>residual_sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free_sulfur_dioxide</th>\n",
       "      <th>total_sulfur_dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "      <th>quality</th>\n",
       "      <th>color</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.4</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.076</td>\n",
       "      <td>11.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.9978</td>\n",
       "      <td>3.51</td>\n",
       "      <td>0.56</td>\n",
       "      <td>9.4</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>7.8</td>\n",
       "      <td>0.88</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.6</td>\n",
       "      <td>0.098</td>\n",
       "      <td>25.0</td>\n",
       "      <td>67.0</td>\n",
       "      <td>0.9968</td>\n",
       "      <td>3.20</td>\n",
       "      <td>0.68</td>\n",
       "      <td>9.8</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7.8</td>\n",
       "      <td>0.76</td>\n",
       "      <td>0.04</td>\n",
       "      <td>2.3</td>\n",
       "      <td>0.092</td>\n",
       "      <td>15.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0.9970</td>\n",
       "      <td>3.26</td>\n",
       "      <td>0.65</td>\n",
       "      <td>9.8</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>11.2</td>\n",
       "      <td>0.28</td>\n",
       "      <td>0.56</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.075</td>\n",
       "      <td>17.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.9980</td>\n",
       "      <td>3.16</td>\n",
       "      <td>0.58</td>\n",
       "      <td>9.8</td>\n",
       "      <td>6</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>7.4</td>\n",
       "      <td>0.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.076</td>\n",
       "      <td>11.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.9978</td>\n",
       "      <td>3.51</td>\n",
       "      <td>0.56</td>\n",
       "      <td>9.4</td>\n",
       "      <td>5</td>\n",
       "      <td>red</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fixed_acidity  volatile_acidity  citric_acid  residual_sugar  chlorides  \\\n",
       "0            7.4              0.70         0.00             1.9      0.076   \n",
       "1            7.8              0.88         0.00             2.6      0.098   \n",
       "2            7.8              0.76         0.04             2.3      0.092   \n",
       "3           11.2              0.28         0.56             1.9      0.075   \n",
       "4            7.4              0.70         0.00             1.9      0.076   \n",
       "\n",
       "   free_sulfur_dioxide  total_sulfur_dioxide  density    pH  sulphates  \\\n",
       "0                 11.0                  34.0   0.9978  3.51       0.56   \n",
       "1                 25.0                  67.0   0.9968  3.20       0.68   \n",
       "2                 15.0                  54.0   0.9970  3.26       0.65   \n",
       "3                 17.0                  60.0   0.9980  3.16       0.58   \n",
       "4                 11.0                  34.0   0.9978  3.51       0.56   \n",
       "\n",
       "   alcohol  quality color  \n",
       "0      9.4        5   red  \n",
       "1      9.8        5   red  \n",
       "2      9.8        5   red  \n",
       "3      9.8        6   red  \n",
       "4      9.4        5   red  "
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_cols = [ cols for cols in data.columns if cols != 'residual_sugar']\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "lb = LabelEncoder()\n",
    "\n",
    "\n",
    "X_train = data.loc[train_idx,feature_cols]\n",
    "X_train['color'] = lb.fit_transform(X_train['color'])\n",
    "\n",
    "Y_train = data.loc[train_idx,'residual_sugar']\n",
    "\n",
    "X_test = data.loc[test_idx,feature_cols]\n",
    "X_test['color'] = lb.transform(X_test['color'])\n",
    "Y_test = data.loc[test_idx,'residual_sugar']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.tree  import DecisionTreeRegressor\n",
    "\n",
    "dr = DecisionTreeRegressor().fit(X_train,Y_train)\n",
    "\n",
    "params_grid = {'max_depth':range(1,dr.tree_.max_depth+1,2),\n",
    "              'max_features':range(1,len(dr.feature_importances_)+1)\n",
    "              }\n",
    "\n",
    "grid_cvr = GridSearchCV(\n",
    "                      DecisionTreeRegressor(random_state=42),\n",
    "                      param_grid = params_grid,\n",
    "                      scoring = 'neg_mean_squared_error',\n",
    "                      n_jobs=-1)\n",
    "\n",
    "grid_cvr = grid_cvr.fit(X_train,Y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_cvr.best_estimator_.tree_.max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7381"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_cvr.best_estimator_.tree_.node_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "\n",
    "Y_train_pred = grid_cvr.predict(X_train)\n",
    "Y_test_pred = grid_cvr.predict(X_test)\n",
    "\n",
    "tree_report = pd.DataFrame()\n",
    "tree_report = pd.concat([tree_report,pd.Series({'train_error':mean_squared_error(Y_train,Y_train_pred),\n",
    "                                               'test_error':mean_squared_error(Y_test,Y_test_pred)}\n",
    "                                               ,name=\"MSE\")])\n",
    "tree_report = pd.concat([tree_report,pd.Series({'train_r2_score':r2_score(Y_train,Y_train_pred),\n",
    "                                               'test_r2_score':r2_score(Y_test,Y_test_pred)\n",
    "                                               },name=\"R2_Score\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>test_error</th>\n",
       "      <td>3.420587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_error</th>\n",
       "      <td>0.001456</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>test_r2_score</th>\n",
       "      <td>0.854796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>train_r2_score</th>\n",
       "      <td>0.999935</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       0\n",
       "test_error      3.420587\n",
       "train_error     0.001456\n",
       "test_r2_score   0.854796\n",
       "train_r2_score  0.999935"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tree_report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### There is a huge difference between train and test report.It lokks like overfitting problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}