Part 2 - Model Construction and Initialization.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import numpy as np\n",
    "import torch\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "import imageio\n",
    "from skimage import io, transform\n",
    "import torchvision\n",
    "from torchvision import transforms\n",
    "\n",
    "from scripts.dataloader import Dataset, Transforms, Resize, ToTorch, Sampler, collate, DataLoader\n",
    "from functools import partial\n",
    "\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 0 - Imports from the Part 1\n",
    "\n",
    "Now that we have a dataloader, we can start to build model that uses that data.  Before we begin, let us create a dataset and dataloader from which we'll extract the minibatches required to train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_path = r'data/processed_dataframe.csv'\n",
    "img_col = 'filename'\n",
    "cont_cols = ['followers', 'following', 'engagement_factor_std', 'month', 'year', 'day_name', 'hour']\n",
    "cat_cols = []\n",
    "target_col = 'engagement_factor_moving_avg'\n",
    "image_path = Path(r'data/Images')\n",
    "tfms = Transforms([Resize(256), ToTorch()])\n",
    "\n",
    "ds_train = Dataset(df_path, \n",
    "                   img_col = img_col,\n",
    "                   cont_cols = cont_cols, \n",
    "                   cat_cols = cat_cols, \n",
    "                   target_col = target_col, \n",
    "                   image_path = image_path, \n",
    "                   transforms = tfms)\n",
    "\n",
    "dl_train = DataLoader(dataset = ds_train,\n",
    "                      sampler = Sampler(ds_train, bs = 16),\n",
    "                      collate_func = collate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch 0, with target shape torch.Size([16])\n",
      "Minibatch 1, with target shape torch.Size([16])\n",
      "Minibatch 2, with target shape torch.Size([16])\n",
      "Minibatch 3, with target shape torch.Size([16])\n",
      "Minibatch 4, with target shape torch.Size([16])\n",
      "Minibatch 5, with target shape torch.Size([16])\n",
      "Minibatch 6, with target shape torch.Size([16])\n"
     ]
    }
   ],
   "source": [
    "for i, (xb,yb) in enumerate(dl_train):\n",
    "    print (f\"Minibatch {i}, with target shape {yb.shape}\")\n",
    "    if i>5: break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "iterator = iter(dl_train)\n",
    "xb, yb = next(iterator)\n",
    "x_image, x_tab = xb\n",
    "x_image = x_image.float()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 3, 256, 256]), torch.Size([16, 7]), torch.Size([16]))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_image.shape, x_tab.shape, yb.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a single batch to work with as we test our network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1 - Investigating the components of an nn.Module\n",
    "\n",
    "Pytorch models are created using nn.Module as a base class.  This imparts many features onto the model and its subcomponents, but at the core it allows a few things to happen:\n",
    "- You can initialize the module with __init__ to store all the key variables as well as a call to the nn.Module.__init__ using super. That will set up many of the core instance variables of the module (including the parameters that will be tracked and updated during training)\n",
    "- You can define how an input is processed by the model and what it outputs (should output a torch tensor).  This is the \"forward\" class method and we will use it extensively when constructing our models\n",
    "- You can register forwards or backwards hooks (extra code that will be run during the forward or backwards phase).  These (and callbacks in general) will be the subject of a future instalment\n",
    "- Other useful class methods such as zero_grad, or the ability to set the state as training or validation (eval)\n",
    "\n",
    "Note: rather than have a backwards class, pytorch uses an autograd system to update the gradients for all the relevant parameters, and an optimizer class to update the parameters themselves.  We will go over these features in a future installment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, one of the basic modules is the nn.Linear class, which performs the operation wx + b, where w is the weight parameter and b is the bias.  Not that for historical reasons the operation is performed as x.(w.T) + b, so the shape of the weights will be the opposite of what you expect\n",
    "\n",
    "```python\n",
    "class Linear(Module):\n",
    "    r\"\"\"Applies a linear transformation to the incoming data: :math:y = xA^T + b\"\"\"\n",
    "    __constants__ = ['bias', 'in_features', 'out_features']\n",
    "\n",
    "    def __init__(self, in_features, out_features, bias=True):\n",
    "        super(Linear, self).__init__()\n",
    "        self.in_features = in_features\n",
    "        self.out_features = out_features\n",
    "        self.weight = Parameter(torch.Tensor(out_features, in_features))\n",
    "        if bias:\n",
    "            self.bias = Parameter(torch.Tensor(out_features))\n",
    "        else:\n",
    "            self.register_parameter('bias', None)\n",
    "        self.reset_parameters()\n",
    "\n",
    "    def reset_parameters(self):\n",
    "        init.kaiming_uniform_(self.weight, a=math.sqrt(5))\n",
    "        if self.bias is not None:\n",
    "            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)\n",
    "            bound = 1 / math.sqrt(fan_in)\n",
    "            init.uniform_(self.bias, -bound, bound)\n",
    "\n",
    "    def forward(self, input):\n",
    "        return F.linear(input, self.weight, self.bias)\n",
    "\n",
    "    def extra_repr(self):\n",
    "        return 'in_features={}, out_features={}, bias={}'.format(\n",
    "            self.in_features, self.out_features, self.bias is not None\n",
    "        )\n",
    "```\n",
    "\n",
    "The F.linear is defined as:\n",
    "\n",
    "\n",
    "```python \n",
    "def linear(input, weight, bias=None):\n",
    "    if input.dim() == 2 and bias is not None:\n",
    "        # fused op is marginally faster\n",
    "        ret = torch.addmm(bias, input, weight.t())\n",
    "    else:\n",
    "        output = input.matmul(weight.t())\n",
    "        if bias is not None:\n",
    "            output += bias\n",
    "        ret = output\n",
    "    return ret\n",
    "\n",
    "```\n",
    "\n",
    "Which is essentially x.(w.T) + b for our purposes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that initializing the module creates the appropriate weight and bias tensors, then initializes them using kaiming_uniform.  If you want to initialize your parameters in a different way, you can either modify them after the layer has been created (which we will discuss later), or create another Linear class that overwrites the reset_parameters class method.  \n",
    "\n",
    "We can explore the input and output of the linear module using a test_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 20]), torch.Size([16, 5]))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_batch = torch.rand(16, 20)\n",
    "test_linear_layer = nn.Linear(20, 5)\n",
    "results = test_linear_layer(test_batch)\n",
    "test_batch.shape, results.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that we input a tensor with batch size of 16 and with 20 elements, and that after going through the linear layer we are left with 5 elements from our of our 16 input samples, as expected"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2 - nn.module Building Blocks\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some useful modules \n",
    "\n",
    "The torch.nn.Module class provides  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "- nn.Linear() - The basic building block of a fully-connected network.  It will take an input with shape (batch_size, in_features) and will produce an output with shape (batch_size, out_features)\n",
    "    * in_features - the number of features coming into the module (independent of batch size) \n",
    "    * out_features - the number of desired features leaving the module\n",
    "    * Note: nn.Linear will have up to two parameters:\n",
    "        - A weight matrix of shape (out_features, in_features).\n",
    "        - A bias vector of shape (out_features)\n",
    "\n",
    "\n",
    "- nn.Conv2d() - The basic building block of a convolutional neural net.  It takes in an input with shape (batch_size, in_channels, height, width), and will produce an output with shape (batch_size, out_channels, height, width)\n",
    "    * in_channels - The number of channels coming into the layer.  \n",
    "    * out_channels - The number of channels leaving the layer\n",
    "    * kernel_size,\n",
    "    * stride=1,\n",
    "    * padding=0,\n",
    "    * dilation=1,\n",
    "    * groups=1,\n",
    "    * bias=True,\n",
    "    * padding_mode='zeros',\n",
    "    * Note: nn.Conv2d has up to two types of parameters:\n",
    "        - The kernel with shape (out_channels, in_channels, kernel_size, kernel_size).  Another way to think about this is that there are \"out_channel\" kernels of shape (in_channels, kernel_size, kernel_size), which each produce a single channel of the output\n",
    "        - A bias vector of shape (out_channels).  Each channel will receive a single bias, which will be applied to each pixel of the channel.\n",
    "    \n",
    "    \n",
    "- nn.ReLU()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear node\n",
      "torch.Size([10, 5])\n",
      "torch.Size([10])\n",
      "\n",
      "Conv2d\n",
      "torch.Size([20, 10, 3, 3])\n",
      "torch.Size([20])\n"
     ]
    }
   ],
   "source": [
    "def param_shapes(module):\n",
    "    for p in module.parameters():\n",
    "        print (p.shape)\n",
    "\n",
    "print ('Linear node')\n",
    "param_shapes(nn.Linear(5, 10))\n",
    "print('')\n",
    "\n",
    "print ('Conv2d')\n",
    "param_shapes(nn.Conv2d(10,20,3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create our own modules by inheriting from the nn.Modules class.  The key requirements when inheriting from the nn.Modules class is that you call the superclass's initialization, and that you define a forward() class method.  Generally, this forward class method accepts a tensor as an input and returns a tensor as an output.  Thanks to the autograd feature of pytorch, you can do whatever you want within those two requirements as tensors will track their own grads throughout the process.  \n",
    "\n",
    "A useful class to make is a Lambda module, which accepts a function during class initialization and applies that function to the input tensor.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Lambda(nn.Module):\n",
    "    def __init__(self, func):\n",
    "        super(Lambda, self).__init__()\n",
    "        self.func = func\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.func(x)\n",
    "    \n",
    "def flatten(x):\n",
    "    return x.view(x.shape[0], -1)\n",
    "\n",
    "def simulate_fc_output(x, n):\n",
    "    return torch.rand((x.shape[0], n))\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3 - Creating a Custom Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building with the end in mind\n",
    "\n",
    "We don't have all the components that we need for the final model, but we can construct our mixed model using some simulated components.  Those components are:\n",
    "- The tabular model\n",
    "- The CNN/image model\n",
    "- The mixed model (that combines the two outputs)\n",
    "\n",
    "Once we have all of these models, we can combine them into a MixedInputModel class, where we call each model sequentially and by passing in the appropriate inputs for each submodel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class MixedInputModel(nn.Module):\n",
    "    def __init__(self, cnn_model,  tabular_model, mixed_model):\n",
    "        super(MixedInputModel, self).__init__()\n",
    "        \n",
    "        self.cnn_model = cnn_model\n",
    "        self.tabular_model = tabular_model\n",
    "        self.mixed_model = mixed_model\n",
    "        \n",
    "    \n",
    "    def forward(self, x):\n",
    "        #unpack the x_batch tuple into the image and tabular components\n",
    "        x_image, x_tab = x\n",
    "        x_image = x_image.float()\n",
    "        x_tab = x_tab.float()\n",
    "        \n",
    "        #run each component seperately through their respective models\n",
    "        cnn_output = self.cnn_model(x_image)\n",
    "        tabular_output = self.tabular_model(x_tab)\n",
    "        \n",
    "        #concatenate the outputs from both networks and pass it through the mixed model output\n",
    "        concat_outputs = torch.cat((cnn_output, tabular_output), dim = 1)\n",
    "        mixed_model_output = self.mixed_model(concat_outputs) \n",
    "        \n",
    "        return(mixed_model_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn_model = Lambda(partial(simulate_fc_output, n = 10))\n",
    "tabular_model = Lambda(partial(simulate_fc_output, n = 5))\n",
    "mixed_model = Lambda(partial(simulate_fc_output, n = 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([16, 10]),\n",
       " torch.Size([16, 5]),\n",
       " torch.Size([16, 15]),\n",
       " torch.Size([16, 1]))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn_output = cnn_model(x_image)\n",
    "tabular_output = tabular_model(x_tab)\n",
    "concat_outputs = torch.cat((cnn_output, tabular_output), dim = 1)\n",
    "mixed_model_output = mixed_model(concat_outputs)\n",
    "\n",
    "(cnn_output.shape, tabular_output.shape, concat_outputs.shape, mixed_model_output.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "final_model = MixedInputModel(cnn_model = cnn_model,\n",
    "                              tabular_model = tabular_model,\n",
    "                              mixed_model = mixed_model\n",
    "                             )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([16, 1])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "final_model(xb).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all there is to mixing the two inputs!  Unfortunately, we won't get great results from this model since the submodels just create constant outputs.  We now need to create the three input models to replace our placeholder models: the cnn_model, the tabular_model, and the mixed_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple Tabular Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BasicTabularModel(nn.Module):\n",
    "    h1 = 10\n",
    "    h2 = 5    \n",
    "    \n",
    "    def __init__(self, in_features, out_features):\n",
    "        super(BasicTabularModel, self).__init__()\n",
    "        self.layer_1 = nn.Linear(in_features, self.h1)\n",
    "        self.layer_2 = nn.Linear(self.h1, self.h2)\n",
    "        self.layer_3 = nn.Linear(self.h2, out_features)\n",
    "        \n",
    "        self.relu = nn.ReLU()\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.layer_1(x)\n",
    "        x = self.relu(x)\n",
    "        \n",
    "        x = self.layer_2(x)\n",
    "        x = self.relu(x)\n",
    "        \n",
    "        x = self.layer_3(x)\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs, in_features = x_tab.shape\n",
    "basic_model = BasicTabularModel(in_features, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-11109.4121],\n",
       "        [-46031.8281],\n",
       "        [-13607.8604],\n",
       "        [-11813.8057],\n",
       "        [ -4535.3691],\n",
       "        [ -1341.2977],\n",
       "        [ -3102.7595],\n",
       "        [-11813.0830],\n",
       "        [ -2798.6069],\n",
       "        [ -3874.8674],\n",
       "        [ -9163.5762],\n",
       "        [ -4943.7153],\n",
       "        [ -7840.6665],\n",
       "        [-46031.1914],\n",
       "        [ -9162.2314],\n",
       "        [ -3874.8804]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "basic_model(x_tab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can define a complete model during the class initialization and then use that during the forward pass.  This approach will make things easier when we put everything together later on in the notebook.  We can also make the model more flexible by accepting the tabular models hidden and final layer sizes as inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class TabularModel(nn.Module):    \n",
    "    def __init__(self, layer_sizes):\n",
    "        super(TabularModel, self).__init__()\n",
    "        \n",
    "        layers = []      \n",
    "        \n",
    "        for i in range(len(layer_sizes)-1):\n",
    "            layers.append(nn.Linear(layer_sizes[i], layer_sizes[i+1]))\n",
    "            layers.append(nn.ReLU())\n",
    "        \n",
    "        \n",
    "        self.model = nn.Sequential(*layers[:-1]) #ignore the last nn.ReLU\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs, in_features = x_tab.shape\n",
    "tab_model = TabularModel([in_features, 10, 5, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-11450.5615],\n",
       "        [-47384.1445],\n",
       "        [-14024.9053],\n",
       "        [-12180.5898],\n",
       "        [ -4680.1562],\n",
       "        [ -1380.0170],\n",
       "        [ -3189.9885],\n",
       "        [-12180.7090],\n",
       "        [ -2890.3040],\n",
       "        [ -4001.6138],\n",
       "        [ -9402.6016],\n",
       "        [ -5090.2539],\n",
       "        [ -8087.6436],\n",
       "        [-47384.3242],\n",
       "        [ -9402.9619],\n",
       "        [ -4001.6382]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tab_model(x_tab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Starting with a known number of inputs, we have created a tabular model that produces a single output value.  Notably, these values are much larger than the -1 to 1 range that we usually expect.  This is because the weights of the network have not been properly initialized.  We will handle network initialization later.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Simple CNN Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16, 3, 256, 256)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bs, c, w, h = x_image.shape\n",
    "bs, c, w, h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SimpleCNNModel(nn.Module):\n",
    "    c1 = 4\n",
    "    c2 = 8\n",
    "    c3 = 12\n",
    "    \n",
    "    def __init__(self):\n",
    "        super(SimpleCNNModel, self).__init__()\n",
    "        self.model = nn.Sequential(nn.Conv2d(3, self.c1, kernel_size = 3, padding = 1, stride = 2), nn.ReLU(),\n",
    "                                  nn.Conv2d(self.c1, self.c2, kernel_size = 3, padding = 1, stride = 2), nn.ReLU(), \n",
    "                                  nn.Conv2d(self.c2, self.c3, kernel_size = 3, padding = 1, stride = 2), nn.ReLU())\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([16, 12, 32, 32])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "simple_cnn_model = SimpleCNNModel()\n",
    "simple_cnn_model(x_image).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see here that we have the desired 12 output channels. The height and width of our output image has changed.  After each convolutional layer, the output height and width will change in the following manner: \n",
    "\n",
    "$$ output\\_size = \\frac{input\\_size + 2*padding - dilation*(kernel\\_size-1) -1}{stride} + 1 $$\n",
    "\n",
    "Although this is not something that we need to concern ourselves with during the input stage, it's necessary to understand how these values change so that when we want to unflatten the image into a fully connected layer, we know how many input layers there are.  Tracking the size also lets us customize the CNN component such that it brings the final image down to an appropriate size.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding in the Fully Connected Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export \n",
    "\n",
    "# class Lambda(nn.Module):\n",
    "#     def __init__(self, func):\n",
    "#         super(Lambda, self).__init__()\n",
    "#         self.func = func\n",
    "        \n",
    "#     def forward(self, x):\n",
    "#         return self.func(x)\n",
    "\n",
    "    \n",
    "# def flatten(x):\n",
    "#     return x.view(x.shape[0], -1)\n",
    "    \n",
    "\n",
    "class CNNModel(nn.Module):\n",
    "    h = [5, 7, 10, 14] #hidden layer channels    \n",
    "    def __init__(self, img_channels, img_size):\n",
    "        super(CNNModel, self).__init__()\n",
    "        self.img_channels, self.size = img_channels, img_size\n",
    "        \n",
    "        current_channels = img_channels\n",
    "        output_size = img_size\n",
    "        \n",
    "        cnn_model_components = []\n",
    "        for new_channels in self.h:\n",
    "            layer, output_size = self.get_cnn_layer(current_channels, new_channels, 3, 1, 2, output_size)\n",
    "            current_channels = new_channels\n",
    "            cnn_model_components.append(layer)\n",
    "        \n",
    "        while output_size >5:\n",
    "            layer, output_size = self.get_cnn_layer(self.h[-1], self.h[-1], 3, 1, 2, input_size = output_size)\n",
    "            \n",
    "        cnn_model_components.append(nn.AdaptiveAvgPool2d(1))\n",
    "        cnn_model_components.append(Lambda(flatten))\n",
    "        \n",
    "        fcc_model_components = nn.Sequential(nn.Linear(self.h[-1], 20), nn.ReLU(),\n",
    "                                            nn.Linear(20, 10), nn.ReLU(),\n",
    "                                            nn.Linear(10, 1))      \n",
    "        \n",
    "        self.model = nn.Sequential(*cnn_model_components, fcc_model_components)\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        \n",
    "    def get_cnn_layer(self, inp_chs, out_chs, kernel_size = 3, padding = 1, stride = 2, input_size = None):\n",
    "        \"\"\"\n",
    "        This function acts as a default for \n",
    "        \n",
    "        \n",
    "        We can keep track of the final size of our network based on the initial size.  The formula for output\n",
    "        size is the floor of O = ((Input_size + 2*padding - dilation*(kernel_size-1) -1)/stride) + 1.  For instance, with\n",
    "        input size = 256, kernel_size = 3, padding = 2 and stride = 2, we get (256+2*2-1*(3-1)-1)/2 = 127.5, whose floor\n",
    "        is 127.  We therefore expect the output height and width to be 127\n",
    "        \n",
    "        \"\"\"\n",
    "        \n",
    "        cnn_layer = nn.Conv2d(inp_chs, out_chs, kernel_size, stride, padding,1)\n",
    "        \n",
    "        if input_size is None: return cnn_layer\n",
    "        else:\n",
    "            new_size = ((input_size + 2*padding - 1*(kernel_size - 1) -1)/stride + 1)//1\n",
    "            return cnn_layer, new_size\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "            \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CNNModel(\n",
       "  (model): Sequential(\n",
       "    (0): Conv2d(3, 5, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "    (1): Conv2d(5, 7, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "    (2): Conv2d(7, 10, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "    (3): Conv2d(10, 14, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "    (4): AdaptiveAvgPool2d(output_size=1)\n",
       "    (5): Lambda()\n",
       "    (6): Sequential(\n",
       "      (0): Linear(in_features=14, out_features=20, bias=True)\n",
       "      (1): ReLU()\n",
       "      (2): Linear(in_features=20, out_features=10, bias=True)\n",
       "      (3): ReLU()\n",
       "      (4): Linear(in_features=10, out_features=1, bias=True)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn_model = CNNModel(3, h)\n",
    "cnn_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-0.1497],\n",
      "        [-0.1461],\n",
      "        [-0.1499],\n",
      "        [-0.1471],\n",
      "        [-0.1471],\n",
      "        [-0.1465],\n",
      "        [-0.1489],\n",
      "        [-0.1483],\n",
      "        [-0.1487],\n",
      "        [-0.1499],\n",
      "        [-0.1481],\n",
      "        [-0.1490],\n",
      "        [-0.1484],\n",
      "        [-0.1472],\n",
      "        [-0.1468],\n",
      "        [-0.1491]], grad_fn=<AddmmBackward>) \n",
      "\n",
      "torch.Size([16, 1])\n"
     ]
    }
   ],
   "source": [
    "predictions = cnn_model(x_image)\n",
    "print (predictions,'\\n')\n",
    "print (predictions.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adapting a Pretrained Model\n",
    "\n",
    "We are now able to output a single prediction from each image of our image batch.  However, although you can get great results from a simple tabular model, it's unlikely that we will be able to get good predictive results from a network this simple for image processing.  Fortunately, there are many existing cnn_models that we can use.  An added bonus is the ability to use pretrained networks, which allows us to benefit from weights that have been trained on huge datasets of images.  Although we may not ultimately use our model for the same application, many of the earlier kernels will provide detection of more universal components.  \n",
    "\n",
    "Now that we know how to build a model from components, we can start to mix in these preconstructed and pretrained CNN models into our final model.  torchvision.models provides many models to choose from.  We will "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ResNet(\n",
       "  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
       "  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (relu): ReLU(inplace=True)\n",
       "  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
       "  (layer1): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (layer2): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (3): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (layer3): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (3): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (4): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (5): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (layer4): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
       "  (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "base_model = torchvision.models.resnet34(pretrained=True)\n",
    "base_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going through the model, we see that there are four types of nn.Modules used.  Some are familiar to us and others are new.\n",
    "\n",
    "Known:\n",
    "- Conv2d\n",
    "- Linear\n",
    "- ReLU\n",
    "\n",
    "New:\n",
    "- BatchNorm2d\n",
    "- BasicBlock"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from our base model that the final output is a fully-connected layer with 1000 outputs.  That is a very useful output for our purposes, since out final prediction requires only a single output.  That means we have a lot of flexibility between the 1000 inputs and the single output to implement our model.  Before we begin, let us test that this model handles our image batch correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([16, 1000])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = base_model(x_image)\n",
    "results.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, our output is of the form (batch_size, 1000).  That means we're good to go in terms of incorporating Resnet34 into our final cnn model.  As before, we'll add in some Linear layers in order to gradually bring the final number of nodes down to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class CustomResnet(nn.Module):\n",
    "    def __init__(self, base_model, connected_layer_sizes):\n",
    "        super(CustomResnet, self).__init__()\n",
    "        #self.base_mode = base_model\n",
    "        #self.model_outputs = model_outputs\n",
    "        self.connected_layer_sizes = connected_layer_sizes\n",
    "        \n",
    "        connected_layers = []\n",
    "        for i in range(len(connected_layer_sizes)-1):\n",
    "            h1 = connected_layer_sizes[i]\n",
    "            h2 = connected_layer_sizes[i+1]\n",
    "            \n",
    "            connected_layers.append(nn.Linear(h1, h2))\n",
    "            connected_layers.append(nn.ReLU())\n",
    "            \n",
    "        connected_model = nn.Sequential(*connected_layers)\n",
    "        \n",
    "        \n",
    "        self.model = nn.Sequential(base_model, connected_model)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        return self.model(x)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.0000],\n",
       "        [0.0174],\n",
       "        [0.0665],\n",
       "        [0.0586],\n",
       "        [0.0000],\n",
       "        [0.0862],\n",
       "        [0.0000],\n",
       "        [0.1083],\n",
       "        [0.1043],\n",
       "        [0.0000],\n",
       "        [0.0604],\n",
       "        [0.0000],\n",
       "        [0.0099],\n",
       "        [0.3023],\n",
       "        [0.2319],\n",
       "        [0.1932]], grad_fn=<ReluBackward0>)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn_model_resnet = CustomResnet(torchvision.models.resnet34(pretrained = True), [1000, 50, 20, 1])\n",
    "cnn_model_resnet(x_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Side Note - Further Modifications\n",
    "\n",
    "We were fortunate that the output of the base model was conveniently sized, but what if this was not the case?  For instance, if the model had a fully connected component that tailored the output to a very specific application.  Fortunately, we can "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Child: 0\n",
      "Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False) \n",
      "\n",
      "Child: 1\n",
      "BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) \n",
      "\n",
      "Child: 2\n",
      "ReLU(inplace=True) \n",
      "\n",
      "Child: 3\n",
      "MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False) \n",
      "\n",
      "Child: 4\n",
      "Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (2): BasicBlock(\n",
      "    (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 5\n",
      "Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (2): BasicBlock(\n",
      "    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (3): BasicBlock(\n",
      "    (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 6\n",
      "Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (2): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (3): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (4): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (5): BasicBlock(\n",
      "    (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 7\n",
      "Sequential(\n",
      "  (0): BasicBlock(\n",
      "    (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (downsample): Sequential(\n",
      "      (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    )\n",
      "  )\n",
      "  (1): BasicBlock(\n",
      "    (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      "  (2): BasicBlock(\n",
      "    (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "    (relu): ReLU(inplace=True)\n",
      "    (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "    (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 8\n",
      "AdaptiveAvgPool2d(output_size=(1, 1)) \n",
      "\n",
      "Child: 9\n",
      "Linear(in_features=512, out_features=1000, bias=True) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, c in enumerate(base_model.children()):\n",
    "    print('Child:', i)\n",
    "    print (c, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can convert these children into a list and select only the children that we want.  Note that each nn.Sequential (or module) will be grouped as a single child.  Each of these children may have children of its own.  For instance, Child 4 has three children, each a BasicBlock.  Each of these basic blocks has 5 children of their own."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Groups of "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([16, 512, 8, 8])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "children = list(base_model.children())\n",
    "new_model = nn.Sequential(*children[:-2])\n",
    "results = new_model(x_image)\n",
    "results.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
       "  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (2): ReLU(inplace=True)\n",
       "  (3): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
       "  (4): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (5): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (3): BasicBlock(\n",
       "      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (6): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (3): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (4): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (5): BasicBlock(\n",
       "      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       "  (7): Sequential(\n",
       "    (0): BasicBlock(\n",
       "      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (downsample): Sequential(\n",
       "        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
       "        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      )\n",
       "    )\n",
       "    (1): BasicBlock(\n",
       "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "    (2): BasicBlock(\n",
       "      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "      (relu): ReLU(inplace=True)\n",
       "      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    )\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By extracting children of this pretrained model, we can use only the components that we need.  Since this is a pretrained model, however, it makes more sense to cut out layers from the end rather than the beginning.  The middle layers have been trained to accept the inputs from their previous layers.  If those are gone you may lose the benefit of training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4 - Putting it all Together\n",
    "\n",
    "Now that we have all the components in place, we will construct our model from the following components:\n",
    "- A tabular model (fully connected) that accepts the tabular inputs and outputs 4 numbers\n",
    "- A CNN model that accepts the image inputs and outputs 10 nodes\n",
    "- Another fully connected model that accepts the concatenated outputs from the tabular and cnn (14 total), and results in a single predictive output\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs, ch_img, h_img, w_img = x_image.shape\n",
    "bs, tab_inputs = x_tab.shape\n",
    "\n",
    "num_cnn_outputs = 10\n",
    "num_tabular_outputs = 4\n",
    "\n",
    "num_mixed_inputs = num_cnn_outputs + num_tabular_outputs\n",
    "\n",
    "\n",
    "input_cnn_model = CustomResnet(torchvision.models.resnet34(pretrained = True), [1000,50,20, num_cnn_outputs])\n",
    "input_tabular_model = TabularModel([tab_inputs, 10, num_tabular_outputs])\n",
    "input_mixed_model = TabularModel([num_mixed_inputs, 7, 1])\n",
    "\n",
    "mixed_model = MixedInputModel(input_cnn_model, input_tabular_model, input_mixed_model)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2210.7942],\n",
       "        [9096.5645],\n",
       "        [2698.2668],\n",
       "        [2344.1638],\n",
       "        [ 908.9642],\n",
       "        [ 277.5974],\n",
       "        [ 628.1006],\n",
       "        [2344.1633],\n",
       "        [ 564.4508],\n",
       "        [ 777.1080],\n",
       "        [1837.3689],\n",
       "        [ 992.0399],\n",
       "        [1563.8647],\n",
       "        [9096.4697],\n",
       "        [1837.2316],\n",
       "        [ 777.0434]], grad_fn=<AddmmBackward>)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mixed_model(xb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we enumerate through the children, we see that each of our models is a separate child"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Child: 0\n",
      "CustomResnet(\n",
      "  (model): Sequential(\n",
      "    (0): ResNet(\n",
      "      (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
      "      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "      (relu): ReLU(inplace=True)\n",
      "      (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
      "      (layer1): Sequential(\n",
      "        (0): BasicBlock(\n",
      "          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (1): BasicBlock(\n",
      "          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (2): BasicBlock(\n",
      "          (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "      )\n",
      "      (layer2): Sequential(\n",
      "        (0): BasicBlock(\n",
      "          (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (downsample): Sequential(\n",
      "            (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "            (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          )\n",
      "        )\n",
      "        (1): BasicBlock(\n",
      "          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (2): BasicBlock(\n",
      "          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (3): BasicBlock(\n",
      "          (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "      )\n",
      "      (layer3): Sequential(\n",
      "        (0): BasicBlock(\n",
      "          (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (downsample): Sequential(\n",
      "            (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "            (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          )\n",
      "        )\n",
      "        (1): BasicBlock(\n",
      "          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (2): BasicBlock(\n",
      "          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (3): BasicBlock(\n",
      "          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (4): BasicBlock(\n",
      "          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (5): BasicBlock(\n",
      "          (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "      )\n",
      "      (layer4): Sequential(\n",
      "        (0): BasicBlock(\n",
      "          (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (downsample): Sequential(\n",
      "            (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)\n",
      "            (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          )\n",
      "        )\n",
      "        (1): BasicBlock(\n",
      "          (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "        (2): BasicBlock(\n",
      "          (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "          (relu): ReLU(inplace=True)\n",
      "          (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
      "          (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
      "        )\n",
      "      )\n",
      "      (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))\n",
      "      (fc): Linear(in_features=512, out_features=1000, bias=True)\n",
      "    )\n",
      "    (1): Sequential(\n",
      "      (0): Linear(in_features=1000, out_features=50, bias=True)\n",
      "      (1): ReLU()\n",
      "      (2): Linear(in_features=50, out_features=20, bias=True)\n",
      "      (3): ReLU()\n",
      "      (4): Linear(in_features=20, out_features=10, bias=True)\n",
      "      (5): ReLU()\n",
      "    )\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 1\n",
      "TabularModel(\n",
      "  (model): Sequential(\n",
      "    (0): Linear(in_features=7, out_features=10, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=10, out_features=4, bias=True)\n",
      "  )\n",
      ") \n",
      "\n",
      "Child: 2\n",
      "TabularModel(\n",
      "  (model): Sequential(\n",
      "    (0): Linear(in_features=14, out_features=7, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=7, out_features=1, bias=True)\n",
      "  )\n",
      ") \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, c in enumerate(mixed_model.children()):\n",
    "    print('Child:', i)\n",
    "    print (c, '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 5 - A Note on Initializing our Network\n",
    "\n",
    "The first few cycles of training are crucial as inproper starting weights can cause the gradients to vanish (go to zero) or explode (go towards infinity).  This occurs when the outputs from each layer tend away from a mean of zero and a std of 1.  There are a few methods that models use to address this.  In many CNN models, there is a BatchNorm layer, which essentially resets the batch to have a mean of 0 and std of 1, then scales it using some learned parametes.\n",
    "\n",
    "Another helpful way to reduce these issues is to ensure that you initialize your parameters correctly based on an understanding of how the mean and standard deviation change after each layer.  We saw earlier that the linear component of our network is initialized using kaiming_uniform and that the CNN component is built using transfer leaarning.  This suggests that we should be alright in terms of getting things trained.\n",
    "\n",
    "If you do want to do your own initialization, you can either create your own versions of the layers (as we discussed earlier with the nn.Linear class and the reset_parameters() class method).  Alternatively, you can re-initialize any of the parameters once your model has been created.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we take our cnn_model, for example, we can look at all of the children it contains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Conv2d(3, 5, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "  (1): Conv2d(5, 7, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "  (2): Conv2d(7, 10, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "  (3): Conv2d(10, 14, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))\n",
       "  (4): AdaptiveAvgPool2d(output_size=1)\n",
       "  (5): Lambda()\n",
       "  (6): Sequential(\n",
       "    (0): Linear(in_features=14, out_features=20, bias=True)\n",
       "    (1): ReLU()\n",
       "    (2): Linear(in_features=20, out_features=10, bias=True)\n",
       "    (3): ReLU()\n",
       "    (4): Linear(in_features=10, out_features=1, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn_model.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last child (6) is the one we're interested in.  We can extract that by converting the children into a list and taking only the last one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "fcc_components = list(cnn_model.model.children())[-1]\n",
    "all_children = list(fcc_components.children())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we wanted to reinitialize all of the weights from this sections, we can access them through the .weight attribute for each child."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear(in_features=14, out_features=20, bias=True): Has a weight parameter\n",
      "ReLU(): Does not have a weight parameter\n",
      "Linear(in_features=20, out_features=10, bias=True): Has a weight parameter\n",
      "ReLU(): Does not have a weight parameter\n",
      "Linear(in_features=10, out_features=1, bias=True): Has a weight parameter\n"
     ]
    }
   ],
   "source": [
    "for c in all_children:\n",
    "    print(c, end = ': ')\n",
    "    if hasattr(c, 'weight'): print(\"Has a weight parameter\")\n",
    "    else: print(\"Does not have a weight parameter\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_weights = []\n",
    "for c in all_children:\n",
    "    if hasattr(c, 'weight'): all_weights.append(c.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a quick look at the original weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[-0.0086,  0.2395,  0.1832,  0.0250,  0.1219, -0.0510,  0.0065,  0.2465,\n",
       "          -0.0095,  0.0606, -0.2030,  0.0678, -0.1657, -0.1405],\n",
       "         [ 0.1449, -0.1771,  0.0197, -0.0646,  0.0891,  0.0353,  0.0960, -0.0454,\n",
       "           0.0648,  0.1256, -0.0635,  0.1252,  0.2633,  0.1042],\n",
       "         [ 0.1193,  0.1443, -0.2558, -0.2162,  0.1685,  0.2045, -0.0755,  0.0240,\n",
       "           0.2509,  0.0289, -0.2601,  0.1930,  0.0518,  0.1046],\n",
       "         [-0.1411, -0.2553,  0.2507,  0.2606, -0.1996,  0.0155, -0.1708,  0.2156,\n",
       "          -0.1029, -0.1964,  0.2318,  0.0528,  0.0619,  0.0811],\n",
       "         [ 0.0150,  0.0394,  0.0984, -0.1106,  0.2386, -0.0659, -0.1350,  0.1376,\n",
       "          -0.0171,  0.0051, -0.2357, -0.1509, -0.0033, -0.0018],\n",
       "         [ 0.2660, -0.1156,  0.1181,  0.1123,  0.0391, -0.2664, -0.0678, -0.0529,\n",
       "           0.0100,  0.2422, -0.2064,  0.0430, -0.2255,  0.2311],\n",
       "         [ 0.2470, -0.1270,  0.1958, -0.0086, -0.1246, -0.1845, -0.0673,  0.2231,\n",
       "           0.2261,  0.1784, -0.1505,  0.2380,  0.1632, -0.0874],\n",
       "         [ 0.1907, -0.0330, -0.0708,  0.0121,  0.2106,  0.1238,  0.0955,  0.1957,\n",
       "           0.1482, -0.0352,  0.1219,  0.1809, -0.2198, -0.1672],\n",
       "         [-0.2177, -0.0562,  0.0655, -0.2152,  0.0903,  0.0091,  0.1136,  0.2129,\n",
       "           0.2038,  0.2326,  0.0073, -0.0394, -0.0419,  0.1203],\n",
       "         [ 0.2590, -0.0881, -0.0918, -0.0608,  0.2581, -0.0240,  0.2379, -0.2314,\n",
       "          -0.2146,  0.1175, -0.0511, -0.0581, -0.0419,  0.2360],\n",
       "         [-0.0577,  0.1082,  0.2142,  0.1267, -0.1599,  0.0598,  0.1221, -0.0690,\n",
       "           0.1441,  0.1958,  0.0134, -0.0134,  0.1579, -0.0640],\n",
       "         [ 0.1647,  0.2119,  0.1593, -0.2409, -0.2405, -0.0657,  0.2258, -0.2377,\n",
       "          -0.1643,  0.0798,  0.1566,  0.0906,  0.2610,  0.2143],\n",
       "         [-0.0632,  0.2356,  0.0682,  0.0602,  0.1807,  0.0819,  0.0647, -0.2575,\n",
       "          -0.2581,  0.0966,  0.1290,  0.1044, -0.1114,  0.0626],\n",
       "         [ 0.1098,  0.0687, -0.0725, -0.2504,  0.0411,  0.0685, -0.1389, -0.2429,\n",
       "           0.1414,  0.1502, -0.1908,  0.0755,  0.0318, -0.2652],\n",
       "         [ 0.0674,  0.1589,  0.1586,  0.0562,  0.0949, -0.0394,  0.0836, -0.1072,\n",
       "           0.2307,  0.2240,  0.2056, -0.0660, -0.0691,  0.0129],\n",
       "         [-0.0861, -0.1921, -0.1653,  0.1308, -0.1471,  0.0156,  0.2601, -0.1084,\n",
       "          -0.1927,  0.0175,  0.1458,  0.0202,  0.1633, -0.1242],\n",
       "         [ 0.1340,  0.1070,  0.0147, -0.1961, -0.0886, -0.2244,  0.0755,  0.2269,\n",
       "           0.1998, -0.0848, -0.0918,  0.0768,  0.1506, -0.0523],\n",
       "         [-0.0716,  0.2667,  0.0920, -0.1690, -0.1766, -0.0116,  0.1522, -0.0645,\n",
       "          -0.0372, -0.1652, -0.2574, -0.2299,  0.1688, -0.0816],\n",
       "         [-0.2166, -0.0166, -0.1979,  0.0951, -0.0975, -0.1863, -0.2321, -0.2475,\n",
       "          -0.0619,  0.0763,  0.2267, -0.0685, -0.1092, -0.0468],\n",
       "         [ 0.0659, -0.0984,  0.0137,  0.2340,  0.1430, -0.0100, -0.2567, -0.0862,\n",
       "          -0.1668, -0.0720,  0.2271,  0.0577,  0.0696,  0.1692]],\n",
       "        requires_grad=True), Parameter containing:\n",
       " tensor([[ 0.0740,  0.1206, -0.0206,  0.1068,  0.1268, -0.2074, -0.0209, -0.0347,\n",
       "           0.1181, -0.0487,  0.1696, -0.0484,  0.2073, -0.0385,  0.0231, -0.1442,\n",
       "          -0.1815,  0.0908, -0.0945,  0.0664],\n",
       "         [ 0.1026,  0.0540,  0.1149,  0.0441,  0.0566, -0.2235,  0.0552, -0.2027,\n",
       "          -0.1419,  0.1689, -0.1044,  0.0242,  0.1235, -0.0003,  0.0377, -0.0117,\n",
       "          -0.0469,  0.1944, -0.0591,  0.0018],\n",
       "         [ 0.0571,  0.0773,  0.1378, -0.2079,  0.1884, -0.1795,  0.0602, -0.1770,\n",
       "           0.0143, -0.0120, -0.1669, -0.0768,  0.2069,  0.1384,  0.2101, -0.2170,\n",
       "          -0.0030,  0.0590,  0.0331, -0.2198],\n",
       "         [ 0.2227, -0.0174,  0.1663, -0.1252,  0.1508, -0.0159,  0.0622,  0.1900,\n",
       "          -0.1650, -0.1535,  0.0191,  0.1934,  0.0810,  0.1002,  0.1987,  0.2196,\n",
       "          -0.1186, -0.1711,  0.1056,  0.0517],\n",
       "         [-0.0639,  0.2223,  0.1734, -0.0873, -0.1830,  0.1247, -0.1272, -0.2066,\n",
       "           0.2112, -0.1965, -0.0951, -0.1277, -0.0134, -0.0187, -0.0866, -0.1666,\n",
       "          -0.0012,  0.1659,  0.0629, -0.0223],\n",
       "         [ 0.0267, -0.1698, -0.1988, -0.1850, -0.0071,  0.0835,  0.1030,  0.0380,\n",
       "          -0.0090,  0.1229, -0.1326,  0.0222,  0.1566,  0.0944,  0.0940, -0.0097,\n",
       "          -0.1317, -0.0247, -0.0802, -0.1164],\n",
       "         [-0.1137,  0.1184,  0.1501, -0.1999, -0.1862, -0.2072, -0.0796, -0.1321,\n",
       "           0.2070, -0.0474, -0.1258,  0.0504,  0.1464,  0.0425,  0.1545,  0.1760,\n",
       "          -0.0877,  0.0246,  0.0102,  0.1012],\n",
       "         [ 0.1508, -0.1932,  0.1761,  0.1975, -0.0148, -0.1659, -0.0062, -0.1567,\n",
       "          -0.0793,  0.1615, -0.1719,  0.1353, -0.2140, -0.0183,  0.1075, -0.1511,\n",
       "           0.0567, -0.2114, -0.0829, -0.1717],\n",
       "         [-0.0923, -0.1433, -0.0417,  0.1222,  0.0980, -0.0303,  0.1489, -0.0679,\n",
       "           0.0377, -0.1766,  0.1983, -0.1323,  0.0654,  0.2017, -0.0989, -0.1411,\n",
       "          -0.1997,  0.0565, -0.1759,  0.0227],\n",
       "         [-0.2205, -0.1511, -0.1739,  0.1820,  0.0839, -0.1210, -0.1247, -0.0978,\n",
       "           0.1474,  0.2049, -0.1534, -0.1195, -0.1454, -0.1841, -0.0397, -0.1892,\n",
       "           0.0395, -0.1813,  0.1565, -0.1661]], requires_grad=True), Parameter containing:\n",
       " tensor([[ 0.0993,  0.0973,  0.1726,  0.2076,  0.2091, -0.1064, -0.1740,  0.2633,\n",
       "           0.0612,  0.0533]], requires_grad=True)]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, we can set reinitialize all of the weights to 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]],\n",
       "        requires_grad=True), Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.]], requires_grad=True), Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], requires_grad=True)]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for w in all_weights:\n",
    "    nn.init.ones_(w)\n",
    "all_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we then go back to the original model, we can see that the parameters have indeed been changed!  The init.ones_ method is likely a poor choice for the weights matrix, but there are many others that exist in torch.init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([-0.1895, -0.0506, -0.0046, -0.0836, -0.2458,  0.1487,  0.1709, -0.2504,\n",
       "         -0.0911, -0.0970, -0.0023,  0.0088,  0.2596, -0.2128,  0.0139, -0.0676,\n",
       "         -0.2070,  0.2148,  0.1451,  0.0008], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([-0.0989, -0.0383,  0.1280, -0.0826, -0.2236, -0.0840, -0.1844,  0.1129,\n",
       "          0.1450, -0.0491], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([-0.1892], requires_grad=True)]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(cnn_model.parameters())[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want, you can also set the parameters yourself or use your own custom function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def set_to_zero(tensor):\n",
    "    tensor.data = torch.zeros(tensor.shape)\n",
    "\n",
    "with torch.no_grad():\n",
    "    set_to_zero(all_weights[-3])\n",
    "    #all_weights[-1] = torch.zeros(w, h)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Parameter containing:\n",
       " tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
       "         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],\n",
       "        requires_grad=True), Parameter containing:\n",
       " tensor([-0.1895, -0.0506, -0.0046, -0.0836, -0.2458,  0.1487,  0.1709, -0.2504,\n",
       "         -0.0911, -0.0970, -0.0023,  0.0088,  0.2596, -0.2128,  0.0139, -0.0676,\n",
       "         -0.2070,  0.2148,  0.1451,  0.0008], requires_grad=True), Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.],\n",
       "         [1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,\n",
       "          1., 1.]], requires_grad=True), Parameter containing:\n",
       " tensor([-0.0989, -0.0383,  0.1280, -0.0826, -0.2236, -0.0840, -0.1844,  0.1129,\n",
       "          0.1450, -0.0491], requires_grad=True), Parameter containing:\n",
       " tensor([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]], requires_grad=True), Parameter containing:\n",
       " tensor([-0.1892], requires_grad=True)]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(cnn_model.parameters())[-6:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 6 - Using the GPU\n",
    "\n",
    "So far, we have been using the CPU to make predictions using our models.  This is alright for single use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.87 s ± 125 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mixed_model(xb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.1538, 0.4972, 1.2127, 0.7262, 0.8618, 1.0807, 0.8220, 1.0631, 0.8150,\n",
       "        0.8680, 1.1262, 0.9052, 1.2725, 0.6386, 0.5025, 0.8337],\n",
       "       device='cuda:0')"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mixed_model.cuda()\n",
    "xb = tuple(map(lambda x: x.cuda(), xb))\n",
    "yb.cuda()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16.9 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit mixed_model(xb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For my machine, we see that the time for CPU inference is 1.87s and the time for GPU inference is 16.3ms.  That means that the GPU is almost 130 times faster for this model, which is why most of deep learning is performed on the GPU!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook Script Export"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted Part 2 - Model Construction and Initialization.ipynb to scripts\\custom_models.py\n"
     ]
    }
   ],
   "source": [
    "!python scripts/notebook2script.py \"Part 2 - Model Construction and Initialization.ipynb\" 'custom_models.py'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}