Adaboost.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ae9a903c",
   "metadata": {},
   "source": [
    "# Ensemble Learning - Adaboost (Adaptive Boosting)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69c8746f",
   "metadata": {},
   "source": [
    "Hi Guys, Welcome to [Tirendaz Academy](https://youtube.com/c/tirendazacademy)😀 In this notebook, I'm going to talk about adaboost. Happy learning 🐱‍🏍 "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2242bc3",
   "metadata": {},
   "source": [
    "# What is Adabost?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d83ae2d",
   "metadata": {},
   "source": [
    "AdaBoost is a boosting method that uses the complete training dataset to train the weak learners. AdaBoost is a sequential process, where each subsequent model tries to correct the errors of the previous model. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8bdc49e",
   "metadata": {},
   "source": [
    "# Loading the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "23f82768",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9296c06c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"wdbc.data\", header = None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7a346311",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>842302</td>\n",
       "      <td>M</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>842517</td>\n",
       "      <td>M</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>84300903</td>\n",
       "      <td>M</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>84348301</td>\n",
       "      <td>M</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84358402</td>\n",
       "      <td>M</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         0  1      2      3       4       5        6        7       8   \\\n",
       "0    842302  M  17.99  10.38  122.80  1001.0  0.11840  0.27760  0.3001   \n",
       "1    842517  M  20.57  17.77  132.90  1326.0  0.08474  0.07864  0.0869   \n",
       "2  84300903  M  19.69  21.25  130.00  1203.0  0.10960  0.15990  0.1974   \n",
       "3  84348301  M  11.42  20.38   77.58   386.1  0.14250  0.28390  0.2414   \n",
       "4  84358402  M  20.29  14.34  135.10  1297.0  0.10030  0.13280  0.1980   \n",
       "\n",
       "        9   ...     22     23      24      25      26      27      28      29  \\\n",
       "0  0.14710  ...  25.38  17.33  184.60  2019.0  0.1622  0.6656  0.7119  0.2654   \n",
       "1  0.07017  ...  24.99  23.41  158.80  1956.0  0.1238  0.1866  0.2416  0.1860   \n",
       "2  0.12790  ...  23.57  25.53  152.50  1709.0  0.1444  0.4245  0.4504  0.2430   \n",
       "3  0.10520  ...  14.91  26.50   98.87   567.7  0.2098  0.8663  0.6869  0.2575   \n",
       "4  0.10430  ...  22.54  16.67  152.20  1575.0  0.1374  0.2050  0.4000  0.1625   \n",
       "\n",
       "       30       31  \n",
       "0  0.4601  0.11890  \n",
       "1  0.2750  0.08902  \n",
       "2  0.3613  0.08758  \n",
       "3  0.6638  0.17300  \n",
       "4  0.2364  0.07678  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "70dda344",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(569, 32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8244dbb",
   "metadata": {},
   "source": [
    "# Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4960b21a",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.loc[:,2:].values\n",
    "y = df.loc[:,1].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a7946a87",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import LabelEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "269ea6b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "le = LabelEncoder()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "73a082c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = le.fit_transform(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "722ca0b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1953969",
   "metadata": {},
   "source": [
    "# Building Desicion Tree Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a75205d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X,y,stratify=y,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a8bc9b96",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "04134fc6",
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = DecisionTreeClassifier(max_depth=1, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6a8f0ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = tree.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c5579b5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = tree.predict(X_train)\n",
    "y_test_pred = tree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "5f93234a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9fa28dcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "tree_train = accuracy_score(y_train, y_train_pred)\n",
    "tree_test = accuracy_score(y_test, y_test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "9bde8a7f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decision tree train/test accuracies: 0.930/0.888\n"
     ]
    }
   ],
   "source": [
    "print(f\"Decision tree train/test accuracies: {tree_train:.3f}/{tree_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c0c3055",
   "metadata": {},
   "source": [
    "# Building Adaboost Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f31e5efe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a258dc4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ada = AdaBoostClassifier(base_estimator=tree,\n",
    "                        n_estimators=500,\n",
    "                        learning_rate=0.5,\n",
    "                        random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ae4a3963",
   "metadata": {},
   "outputs": [],
   "source": [
    "ada = ada.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b843ac39",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_pred = ada.predict(X_train)\n",
    "y_test_pred = ada.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "e2980374",
   "metadata": {},
   "outputs": [],
   "source": [
    "ada_train = accuracy_score(y_train, y_train_pred)\n",
    "ada_test = accuracy_score(y_test, y_test_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "38d2a387",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Adaboost train/test accuracies: 1.000/0.979\n"
     ]
    }
   ],
   "source": [
    "print(f\"Adaboost train/test accuracies: {ada_train:.3f}/{ada_test:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7879527d",
   "metadata": {},
   "source": [
    "Don't forget to follow us on [YouTube](http://youtube.com/tirendazacademy) | [Medium](http://tirendazacademy.medium.com) | [Twitter](http://twitter.com/tirendazacademy) | [GitHub](http://github.com/tirendazacademy) | [Linkedin](https://www.linkedin.com/in/tirendaz-academy) | [Kaggle](https://www.kaggle.com/tirendazacademy) 😎"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}