05-deep-learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial 3:  Deep Learning with PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'torch'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-dd4d0b069f28>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0;32mimport\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtime\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mVariable\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'torch'"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import sys\n",
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import torch\n",
    "import time\n",
    "from torch.autograd import Variable\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Package Versions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print('__Python VERSION:', sys.version)\n",
    "print('__PyTorch VERSION:', torch.__version__)\n",
    "print('__CUDNN VERSION:', torch.backends.cudnn.version())\n",
    "print('__Number CUDA Devices:', torch.cuda.device_count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PyTorch \n",
    "\n",
    "\n",
    "What is PyTorch?\n",
    "\n",
    "It’s a Python based scientific computing package targeted at two sets of\n",
    "audiences:\n",
    "\n",
    "-  A replacement for numpy to use the power of GPUs\n",
    "-  a deep learning research platform that provides maximum flexibility\n",
    "   and speed\n",
    "\n",
    "\n",
    "\n",
    "### Tensors\n",
    "\n",
    "Tensors are similar to numpy’s ndarrays, with the addition being that\n",
    "Tensors can also be used on a GPU to accelerate computing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Construct a 5x3 matrix, uninitialized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = torch.Tensor(5, 3)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Get its size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y = torch.rand(5, 3)\n",
    "print(x + y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(torch.add(x, y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Addition: giving an output tensor\n",
    "result = torch.Tensor(5, 3)\n",
    "torch.add(x, y, out=result)\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Addition: in-place\n",
    "y.add_(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy Bridge\n",
    "\n",
    "\n",
    "Converting a torch Tensor to a numpy array and vice versa is a breeze.\n",
    "\n",
    "The torch Tensor and numpy array will share their underlying memory\n",
    "locations, and changing one will change the other.\n",
    "\n",
    "Converting torch Tensor to numpy Array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = torch.ones(5)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b = a.numpy()\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# See how the numpy array changed in value.\n",
    "a.add_(1)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Converting numpy array to torch Tensor\n",
    "\n",
    "See how changing the np array changed the torch Tensor automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = np.ones(5)\n",
    "b = torch.from_numpy(a)\n",
    "np.add(a, 1, out=a)\n",
    "print(a)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the Tensors on the CPU except a CharTensor support converting to\n",
    "NumPy and back.\n",
    "\n",
    "CUDA Tensors\n",
    "\n",
    "\n",
    "Tensors can be moved onto GPU using the ``.cuda`` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let us run this cell only if CUDA is available\n",
    "if torch.cuda.is_available():\n",
    "    x = x.cuda()\n",
    "    y = y.cuda()\n",
    "    x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "torch.cuda.is_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Autograd: automatic differentiation\n",
    "===================================\n",
    "\n",
    "Central to all neural networks in PyTorch is the ``autograd`` package.\n",
    "Let’s first briefly visit this, and we will then go to training our\n",
    "first neural network.\n",
    "\n",
    "\n",
    "The ``autograd`` package provides automatic differentiation for all operations\n",
    "on Tensors. It is a define-by-run framework, which means that your backprop is\n",
    "defined by how your code is run, and that every single iteration can be\n",
    "different.\n",
    "\n",
    "Let us see this in more simple terms with some examples.\n",
    "\n",
    "Variable\n",
    "--------\n",
    "\n",
    "``autograd.Variable`` is the central class of the package. It wraps a\n",
    "Tensor, and supports nearly all of operations defined on it. Once you\n",
    "finish your computation you can call ``.backward()`` and have all the\n",
    "gradients computed automatically.\n",
    "\n",
    "You can access the raw tensor through the ``.data`` attribute, while the\n",
    "gradient w.r.t. this variable is accumulated into ``.grad``.\n",
    "\n",
    "There’s one more class which is very important for autograd\n",
    "implementation - a ``Function``.\n",
    "\n",
    "``Variable`` and ``Function`` are interconnected and build up an acyclic\n",
    "graph, that encodes a complete history of computation. Each variable has\n",
    "a ``.grad_fn`` attribute that references a ``Function`` that has created\n",
    "the ``Variable`` (except for Variables created by the user - their\n",
    "``grad_fn is None``).\n",
    "\n",
    "If you want to compute the derivatives, you can call ``.backward()`` on\n",
    "a ``Variable``. If ``Variable`` is a scalar (i.e. it holds a one element\n",
    "data), you don’t need to specify any arguments to ``backward()``,\n",
    "however if it has more elements, you need to specify a ``grad_output``\n",
    "argument that is a tensor of matching shape."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# Create a variable:\n",
    "x = Variable(torch.ones(2, 2), requires_grad=True)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# Do an operation of variable:\n",
    "y = x + 2\n",
    "print(y)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# ``y`` was created as a result of an operation, so it has a ``grad_fn``.\n",
    "print(y.grad_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "###############################################################\n",
    "# Do more operations on y\n",
    "z = y * y * 3\n",
    "out = z.mean()\n",
    "\n",
    "print(z, out)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# Gradients\n",
    "# ---------\n",
    "# let's backprop now\n",
    "# ``out.backward()`` is equivalent to doing ``out.backward(torch.Tensor([1.0]))``\n",
    "\n",
    "out.backward()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "###############################################################\n",
    "# print gradients d(out)/dx\n",
    "#\n",
    "\n",
    "print(x.grad)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have got a matrix of ``4.5``. Let’s call the ``out`` *Variable* $o$.\n",
    "We have that: $o = \\frac{1}{4}\\sum_i z_i$, \n",
    "$z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$\n",
    "\n",
    "Therefore,\n",
    "$$\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$$ hence\n",
    "$$\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You can do many crazy things with autograd!\n",
    "x = torch.randn(3)\n",
    "x = Variable(x, requires_grad=True)\n",
    "\n",
    "y = x * 2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y * 2\n",
    "\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gradients = torch.FloatTensor([0.1, 1.0, 0.0001])\n",
    "y.backward(gradients)\n",
    "\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3.1 Calculate the gradient of the following function wrt variables $w$ and $b$:\n",
    "$$ f(w, b) = \\frac{1}{1+e^{-(w^Tx+b)}}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "w = Variable(torch.randn(10), requires_grad=True)\n",
    "b = Variable(torch.randn(1), requires_grad=True)\n",
    "X = torch.randn(100,10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3.2  implement a logistic regression using PyTorch\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "cancer = load_breast_cancer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, stratify=cancer.target, random_state=42)\n",
    "y_train = y_train.astype('float')\n",
    "y_test = y_test.astype('float')\n",
    "X_train = sklearn.preprocessing.scale(X_train)\n",
    "X_test = sklearn.preprocessing.scale(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(X_train.shape)\n",
    "print(X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Logistic regression implemented in PyTorch\n",
    "# Complete the implementation of the following class\n",
    "class MyLogisticRegression:\n",
    "        \n",
    "    def __init__(self, num_iter=100, lr=0.001) :\n",
    "        self.num_iter = num_iter\n",
    "        self.lr = lr\n",
    "    \n",
    "    def fit(self, X, y) :\n",
    "        X = np.concatenate((X, np.ones((X.shape[0],1))), axis=1)\n",
    "        X = Variable(Tensor(X))\n",
    "        y = Variable(Tensor(y))\n",
    "              \n",
    "        # gradient descent\n",
    "        for iter in range(self.num_iter):\n",
    "            pass\n",
    "\n",
    "\n",
    "            \n",
    "    # calculate cross entropy\n",
    "    @staticmethod\n",
    "    def cross_entropy(y, prob):\n",
    "        # calculate cross entropy between label y and predicted prob \n",
    "        pass\n",
    "    \n",
    "    @staticmethod\n",
    "    def sigmoid(x):  \n",
    "        # given x as np array, return sigmoid transformation of x\n",
    "        pass\n",
    "\n",
    "    # return prediction accuracy given X and labey y\n",
    "    def score(self, X, y) :\n",
    "        pass\n",
    "    \n",
    "    # return predicted labels (1/0) given X\n",
    "    def predict(self, X):\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# After implemention, create an instance \n",
    "lr = MyLogisticRegression(1000,0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# fit the model with training data\n",
    "lr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# check the score of training data\n",
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# check the score of test data\n",
    "lr.score(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Tensor = torch.Tensor\n",
    "# class MyLogisticRegression:\n",
    "    \n",
    "#     def __init__(self, num_iter=100, lr=0.001) :\n",
    "#         self.num_iter = num_iter\n",
    "#         self.lr = lr\n",
    "    \n",
    "#     def fit(self, X, y) :\n",
    "#         X = np.concatenate((X, np.ones((X.shape[0],1))), axis=1)\n",
    "#         X = Variable(Tensor(X))\n",
    "#         y = Variable(Tensor(y))\n",
    "        \n",
    "#         w = Variable(Tensor(0.1*np.random.randn(X.size()[1])), requires_grad=True)\n",
    "        \n",
    "#         # gradient descent\n",
    "#         for iter in range(self.num_iter):\n",
    "#             if hasattr(w.grad, \"data\"):\n",
    "#                 w.grad.data.zero_()  # zero grad\n",
    "#             prob = Variable.sigmoid(X.matmul(w))\n",
    "#             loss = self.cross_entropy(y, prob)\n",
    "#             if iter%100==0:\n",
    "#                 print('{}: {}'.format(iter, loss.data.cpu().numpy()))\n",
    "#             loss.backward()            \n",
    "#             w.data.sub_(self.lr*w.grad.data)\n",
    "            \n",
    "#         self.w = w.data.cpu().numpy()    \n",
    "            \n",
    "#     # calculate cross entropy\n",
    "#     @staticmethod\n",
    "#     def cross_entropy(y, prob):\n",
    "#         return -(y*Variable.log(prob+0.00001) + (1-y)*Variable.log(1-prob+0.00001)).mean()\n",
    "    \n",
    "#     @staticmethod\n",
    "#     def sigmoid(x):\n",
    "#         return 1 / (1 + np.exp(-x))\n",
    "\n",
    "#     def score(self, X, y) :\n",
    "#         pred_y = self.predict(X)\n",
    "#         return 1.0*np.sum(pred_y==y)/len(y)\n",
    "    \n",
    "#     def predict(self, X):\n",
    "#         return 1*(self.sigmoid(np.concatenate((X, np.ones((X.shape[0],1))), axis=1).dot(self.w)) >= 0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lr.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implement logistic regression using torch.nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "       # an affine operation: y = Wx + b\n",
    "        self.fc1 = nn.Linear(30, 1)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = F.sigmoid(self.fc1(x))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = Net()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = list(net.parameters())\n",
    "print(len(params))\n",
    "print(params[0].size())  # fc1's .weight\n",
    "print(params[1].size())  # bias"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# check the parameters\n",
    "print(net.fc1.weight)\n",
    "print(net.fc1.bias)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input = Variable(Tensor(X_train))\n",
    "out = net(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.BCELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "# create your optimizer\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.01)\n",
    "\n",
    "input = Variable(Tensor(X_train))\n",
    "target = Variable(Tensor(y_train))\n",
    "\n",
    "# in your training loop:\n",
    "for i in range(1000):\n",
    "    optimizer.zero_grad()   # zero the gradient buffers\n",
    "    output = net(input)\n",
    "    loss = criterion(output, target)\n",
    "    loss.backward()\n",
    "    optimizer.step()    # Does the update\n",
    "    if i%100 == 0:\n",
    "        print('iter: {} loss: {}'.format(i, loss[0].data.numpy()[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# training accuracy\n",
    "sklearn.metrics.accuracy_score(1*(net(Variable(Tensor(X_train))).data.cpu().numpy()>0.5), y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# test accuracy\n",
    "sklearn.metrics.accuracy_score(1*(net(Variable(Tensor(X_test))).data.cpu().numpy()>0.5), y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pred_y = net(Variable(Tensor(X_test))).data.cpu().numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_auc(target_y, pred_y) :\n",
    "    FPR, TPR, thresholds = sklearn.metrics.roc_curve(target_y,pred_y)\n",
    "    roc_auc = sklearn.metrics.auc(FPR, TPR)\n",
    "\n",
    "    plt.title('LOG_LOSS=' + str(sklearn.metrics.log_loss(target_y, pred_y)))\n",
    "    plt.plot(FPR, TPR, 'b', label='AUC = %0.6f' % roc_auc)\n",
    "    plt.plot([0,1],[0,1],'--k')\n",
    "    plt.legend()\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Posive Rate')\n",
    "    plt.grid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "plot_auc(y_test, pred_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CIFAR10 Image Classifciation\n",
    "It has the classes: ‘airplane’, ‘automobile’, ‘bird’, ‘cat’, ‘deer’,\n",
    "‘dog’, ‘frog’, ‘horse’, ‘ship’, ‘truck’. \n",
    "\n",
    "The images in CIFAR-10 are of\n",
    "size 3x32x32, i.e. 3-channel color images of 32x32 pixels in size.\n",
    "\n",
    "Training an image classifier\n",
    "\n",
    "\n",
    "We will do the following steps in order:\n",
    "\n",
    "1. Load and normalizing the CIFAR10 training and test datasets using\n",
    "   ``torchvision``\n",
    "2. Define a Convolution Neural Network\n",
    "3. Define a loss function\n",
    "4. Train the network on the training data\n",
    "5. Test the network on the test data\n",
    "\n",
    "1. Loading and normalizing CIFAR10\n",
    "\n",
    "Using ``torchvision``, it’s extremely easy to load CIFAR10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torchvision\n",
    "import torchvision.transforms as transforms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preprocessing\n",
    "The output of torchvision datasets are PILImage images of range [0, 1].\n",
    "\n",
    "We transform them to Tensors of normalized range [-1, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "transform = transforms.Compose(\n",
    "    [transforms.ToTensor(),\n",
    "     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n",
    "\n",
    "trainset = torchvision.datasets.CIFAR10(root='./data', train=True,\n",
    "                                        download=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,\n",
    "                                          shuffle=True, num_workers=2)\n",
    "\n",
    "testset = torchvision.datasets.CIFAR10(root='./data', train=False,\n",
    "                                       download=True, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=4,\n",
    "                                         shuffle=False, num_workers=2)\n",
    "\n",
    "classes = ('plane', 'car', 'bird', 'cat',\n",
    "           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let us show some of the training images, for fun."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# functions to show an image\n",
    "def imshow(img):\n",
    "    img = img / 2 + 0.5     # unnormalize\n",
    "    npimg = img.numpy()\n",
    "    plt.imshow(np.transpose(npimg, (1, 2, 0)))\n",
    "\n",
    "import builtins\n",
    "\n",
    "# get some random training images\n",
    "dataiter = builtins.iter(trainloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# show images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "# print labels\n",
    "print(' '.join('%5s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################################################\n",
    "# 2. Define a Convolution Neural Network\n",
    "# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
    "# Copy the neural network from the Neural Networks section before and modify it to\n",
    "# take 3-channel images (instead of 1-channel images as it was defined).\n",
    "\n",
    "\n",
    "\n",
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 5)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 5 * 5, 120)\n",
    "        self.fc2 = nn.Linear(120, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 5 * 5)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = Net()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################################################\n",
    "# 3. Define a Loss function and optimizer\n",
    "# Let's use a Classification Cross-Entropy loss and SGD with momentum\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################################################\n",
    "# 4. Train the network\n",
    "# ^^^^^^^^^^^^^^^^^^^^\n",
    "#\n",
    "# This is when things start to get interesting.\n",
    "# We simply have to loop over our data iterator, and feed the inputs to the\n",
    "# network and optimize\n",
    "\n",
    "for epoch in range(2):  # loop over the dataset multiple times\n",
    "\n",
    "    running_loss = 0.0\n",
    "    for i, data in enumerate(trainloader, 0):\n",
    "        # get the inputs\n",
    "        inputs, labels = data\n",
    "\n",
    "        # wrap them in Variable\n",
    "        inputs, labels = Variable(inputs), Variable(labels)\n",
    "\n",
    "        # zero the parameter gradients\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        # forward + backward + optimize\n",
    "        outputs = net(inputs)\n",
    "        loss = criterion(outputs, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "\n",
    "        # print statistics\n",
    "        running_loss += loss.data[0]\n",
    "        if i % 2000 == 1999:    # print every 2000 mini-batches\n",
    "            print('[%d, %5d] loss: %.3f' %\n",
    "                  (epoch + 1, i + 1, running_loss / 2000))\n",
    "            running_loss = 0.0\n",
    "\n",
    "print('Finished Training')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#######################################################################\n",
    "# 5. Test the network on the test data\n",
    "# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n",
    "#\n",
    "# We have trained the network for 2 passes over the training dataset.\n",
    "# But we need to check if the network has learnt anything at all.\n",
    "#\n",
    "# We will check this by predicting the class label that the neural network\n",
    "# outputs, and checking it against the ground-truth. If the prediction is\n",
    "# correct, we add the sample to the list of correct predictions.\n",
    "#\n",
    "# Okay, first step. Let us display an image from the test set to get familiar.\n",
    "\n",
    "dataiter = builtins.iter(testloader)\n",
    "images, labels = dataiter.next()\n",
    "\n",
    "# print images\n",
    "imshow(torchvision.utils.make_grid(images))\n",
    "print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "########################################################################\n",
    "# Okay, now let us see what the neural network thinks these examples above are:\n",
    "\n",
    "outputs = net(Variable(images))\n",
    "\n",
    "########################################################################\n",
    "# The outputs are energies for the 10 classes.\n",
    "# Higher the energy for a class, the more the network\n",
    "# thinks that the image is of the particular class.\n",
    "# So, let's get the index of the highest energy:\n",
    "_, predicted = torch.max(outputs.data, 1)\n",
    "\n",
    "print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]\n",
    "                              for j in range(4)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let us look at how the network performs on the whole dataset.\n",
    "\n",
    "correct = 0\n",
    "total = 0\n",
    "for data in testloader:\n",
    "    images, labels = data\n",
    "    outputs = net(Variable(images))\n",
    "    _, predicted = torch.max(outputs.data, 1)\n",
    "    total += labels.size(0)\n",
    "    correct += (predicted == labels).sum()\n",
    "\n",
    "print('Accuracy of the network on the 10000 test images: %d %%' % (\n",
    "    100 * correct / total))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3.3:  Change the above model so that it can be run on GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}