Introduction_of_Keras.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "coursera": {
      "course_slug": "neural-networks-deep-learning",
      "graded_item_id": "XaIWT",
      "launcher_item_id": "zAgPl"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.2"
    },
    "colab": {
      "name": "Introduction_of_Keras.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hussain0048/Deep-Learning-with-Keras/blob/master/Introduction_of_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2WZFQQORCAcS"
      },
      "source": [
        "# **1-Introduction Keras**[1]\n",
        "## 1.1 - **What is keras**\n",
        "\n",
        "Keras is a open source deep-learning framework that provides a Convenient way to define and train almost any kind of Deep leanring model. It is written in Python and can be run on top of Tensorflow , CNTK, or Theano. you are free to use it in commerical projects since it is distributed under the MIT licens [1].In Keras we can developed DL model with fee line of code. It was created by French AI researcher Francois **Chollet**.\n",
        "\n",
        "Keras doesn't handle low-level computation. Instead, it uses another library to do it, called the \"Backend. So Keras is high-level API wrapper for the low-level API, capable of running on top of TensorFlow, CNTK, or Theano [17].\n",
        "\n",
        "Keras High-Level API handles the way we make models, defining layers, or set up multiple input-output models. In this level, Keras also compiles our model with loss and optimizer functions, training process with fit function. Keras doesn't handle Low-Level API such as making the computational graph, making tensors or other variables because it has been handled by the \"backend\" engine[17]."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G5li5TZBz4NQ"
      },
      "source": [
        "# 1.2- **What makes Keras so popular** \n",
        "once of the most important characteristic of kearas is it user-friendly API. You could develop a start of art DL Model in no time . Therefore it is easy and fast prototyping.In addition, it support many modern DL layers such as Convolutional and recurrent layers.Keras layers can be added sequentially or many different combinations in very easy wasy. Regarding hardware, you can run keras on CPU and GPU and switch between them in very easy way[1] "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LavPp9XnZbKd"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=14zLPIcbm9MiaqLIWO6_v0bNp6-UXrkeN)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z-OS-JtDaAzN"
      },
      "source": [
        "!git clone https://github.com/hussain0048/Deep-Learning-with-Keras.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6B35X-rixxH"
      },
      "source": [
        "# **1.2 What is a Backend? [17]**\n",
        "\n",
        "Backend is a term in Keras that performs all low-level computation such as tensor products, convolutions and many other things with the help of other libraries such as Tensorflow or Theano. So, the \"backend engine\" will perform the computation and development of the models. Tensorflow is the default \"backend engine\" but we can change it in the configuration.\n",
        "\n",
        "Theano, Tensorflow, and CNTK Backend"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZ729BWkUioJ"
      },
      "source": [
        "## **1.2.1-Keras and TensorFlow**[11]\n",
        "\n",
        "Kera is fully integrated with tensorflow2. If as you dive into deep learning you find yourself needing to use low-level features as for have a finer control of how your network applied gradients you could use tensorflow and tweek whatever you need"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lolpKwKiQMUe"
      },
      "source": [
        "# **1.3 Why we used Keras or Advantages of Kera**[11] \n",
        "we can build industry -ready models in no time with much less code\n",
        "This allows for quicly and easily checking if a neural network will get your problem sovled. In Addition you can build any architecture you can imagine, from simple network  to more complext one like auto-encoder, CNN, RRN. It can aslo be delployed across a wide  range of platform like antrod,IOS, web, App, etc "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XUd8wTEDjw-b"
      },
      "source": [
        "**Cross-Platform and Easy Model Deployment [17]**\n",
        "\n",
        "With a variety of supported devices and platforms, you can deploy Keras on any device like\n",
        "\n",
        "- iOS with CoreML\n",
        "- Android with Tensorflow Android,\n",
        "- Web browser with .js support\n",
        "- Cloud engine\n",
        "- Raspberry Pi"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvrr0QHLpdSH"
      },
      "source": [
        "**Multi GPUs Support [17]**\n",
        "\n",
        "You can train Keras with on a single GPU or use multiple GPUs at once. Because Keras has a built-in support for data parallelism so it can process large volumes of data and speed up the time needed to train it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qx0cmyhZpr-n"
      },
      "source": [
        "# **1.4 Disadvantages of Keras [17]**\n",
        "\n",
        "**Cannot handle low-level API**\n",
        "\n",
        "Keras only handles high-level API which runs on top other framework or backend engine such as Tensorflow, Theano, or CNTK. So it's not very useful if you want to make your own abstract layer for your research purposes because Keras already have pre-configured layers."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7VhNHu7cQ5w"
      },
      "source": [
        "# **1.5-Features Engineering**[11]\n",
        "Neural network are good feature extractor, since they learn the best way to make sense of \n",
        "unstructed data. Previously, It was the domain expert that had to set rules based on experimentation\n",
        "and heuristics to extract the relevant features of data. NN can learn the best features and their combination\n",
        ", they can perform feature engineering themselves "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ssoZne6ILooE"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1h04uCSqD2hslXfE_8ufbUTAWpZZ4YJiF)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KWQTSA7wMuH6"
      },
      "source": [
        "But what is unstructer data? Unstructer data is data that is not easily put into a table, for instance, sound ,video, images ,etc\n",
        "it is also the type of data where performing feature engineering can be more challenging , that is why leaving this task to NN is good idea"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W7hjPkwVMwa5"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1zCN1FkBe9FIb6mUMMPG3QPvUxk96Nmk2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "beVu-1UqCAcU"
      },
      "source": [
        "# 1.3-  **Installing Keras** \n",
        "The installation process is very easy. First, we need to install the backedn where  all the calculation take place (we will choose tensorFlow). Then we install keras [1]\n",
        "\n",
        "**Direct install or Virtual Environment**[17]\n",
        "\n",
        "Which one is better? Direct install to the current python or use a virtual environment? I suggest using a virtual environment if you have many projects. Want to know why? This is because different projects may use a different version of a library.\n",
        "\n",
        "For example, I have a project that needs Python 3.5 using OpenCV 3.3 with older Keras-Theano backend but in the other project I have to use Keras with the latest version and a Tensorflow as it backend with Python 3.6.6 support\n",
        "\n",
        "We don't want the library to conflict at each other right? So we use a Virtual Environment to localize the project with a specific type of library or we can use another platform such as Cloud Service to do our computation for us like Amazon Web Service."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZF03EKpuCAcV"
      },
      "source": [
        "pip install tensorflow"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hLq1Pb9z9bk-"
      },
      "source": [
        "pip install keras"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ktand8m9iKt"
      },
      "source": [
        "# it is simple as this. Let us test the implementation \n",
        "python -c 'import keras; print(keras._version_)'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jnY7VuCDI-qN"
      },
      "source": [
        "## 1.4- **Kera Workflow**#\n",
        "\n",
        "In order to build DL project in keras you normally would follow the following workk flow [1]:\n",
        "- 1) Define your training data\n",
        "- 2) Define your network \n",
        "- 3) Configure the learning process by choosing \n",
        "      - 1) optimizer \n",
        "      - 2) Metrics \n",
        "- 4) iterate over the training data and start fitting your model\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a21M_iV6uN_M"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1eVwjExOmSWHL6nFcvsWhCxy6aokQNXQq)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g_aCyhDgoTRK"
      },
      "source": [
        "## 1.5- **Keras Models?** \n",
        "The core data structure of keras is the model class. it is found under keras.model that gives you two ways to define models:\n",
        " - Sequential class\n",
        " - Model class\n",
        "\n",
        "- The sequential class builds the network layers by layers  in a sequential order . you can start with input layers. Add a couple of hidden layers and finally end of your model by adding output layers \n",
        "\n",
        "- The model class allows for more Complext network structures[1]. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sq4DpmTeRNNy"
      },
      "source": [
        "## 1.6- **Model lifecycle**\n",
        "A Keras model follows the following lifecycle [1]\n",
        "\n",
        " -1. Modle creation\n",
        "\n",
        "     - Define a model using the sequential or model class\n",
        "     - Add the layers \n",
        "\n",
        " -2. Configure the model by specifying the loss, optimizer and metric. This is done by calling the compile method\n",
        "\n",
        " -3. Train the model by calling the fit method\n",
        " \n",
        " -4. By then you will have a trained model that you could use for evaluation or prediction on new data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zOsOIr2HuwEu"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1QghXXdHiADg72Szq7843cDaXKkp7fRUW)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zcVaahYFRsEF"
      },
      "source": [
        "## 1.7- **Core layers**\n",
        "keras supports many layers for building our neural network. They accessible from **keras.layers** and the following shows the most basic classes we are going to use [1]:\n",
        "\n",
        " - Dense: is the standard layer of fully connected neuron to the pervious layer. It implemented the operation output =activation (X*W+Bias)\n",
        " - Activation: Applies an activation function to an output \n",
        " - Dropout: applies dropout to the input. Basically , it work randomly deactivation a set of neurons in a given layer according to a predefined probability rate. Drouput is used to prevent overfitting \n",
        " - Conv2D: applies a 2D Convolution to train a set of kernels mainly on image database\n",
        " - Flatten: Flattens the input into ID matrix. Mainly used after feature extraction in Convolution Neural network     \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3C_dTcsnSGFD"
      },
      "source": [
        "## 1.8-**Loss and Optimizers**\n",
        "After defining a model, we need to select a loss function and an optimizer. The optimizer's jobs is to find the best model parameters that minimizes the losss function[1].\n",
        "\n",
        "Avalilable optimizers: SGD, RMSprop, Adagrad, Adadelta, Adam, Adamax, Nadam\n",
        "\n",
        "Available loss Functions: mean absolut error, mean absolute percentage error, mean squared. Logarithmic error , Squared hing, Categorical hinge, logcosh,categorical crossentropy, sparse Categorical crossentropy, binary corssentropy , kullback divergence, poisson, costine proximity \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MWjlZqTBTeD7"
      },
      "source": [
        "## 1.9. **Keras Utils** \n",
        "Keras provides additional utility functions that facilitates building and viewing models. We will mainly use the them to preprocess data and viewing models [1] \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UOLixKnwiYkE"
      },
      "source": [
        "#**2-Data Pre-processing**\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sNBeXQIhvbKB"
      },
      "source": [
        "## **2.1-Dealing with Data**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XwuJEzGMvqDe"
      },
      "source": [
        "**Motivation**\n",
        "\n",
        "Training deep learning models requires data… A lot of data! Unfortunately, in most cases data comes messy, and our models are very sensitive towards this. Therefore, we need to be careful while preparing our data to achieve the best results.Get your laptops ready, we have a lot of preprocessing to do [3]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HKcl915YwRoh"
      },
      "source": [
        "## **2.3 Working with Numerical Data**\n",
        "\n",
        "Numerical values are going to be the most frequent data types you are going to deal with. Even though they are already in a suitable format for calculations, we still need to do some work.\n",
        "\n",
        "The main problem with numerical data is the different scales each feature holds. Consider a housing prices dataset with information about: house size, number of bedrooms, construction year, and price. Suppose our goal is to predict the price given the house size, number of bedrooms and construction year. Each of those features is presented on a different scale. A house size may range let us say between 100 and 500 meters squared, construction year is a 4 digits number that goes back around 200 years ago, and finally the number of bedrooms is a number between 1-4 [3].\n",
        "\n",
        "The issue here is that a model may give more attention to one feature based on its value. This way the house size may get more attention since its values are bigger, and other important features such as number of bedrooms may be neglected since their values are small [3].\n",
        "\n",
        "Well, do not panic! We have two simple solutions for this problem, they are called Normalization and Standardization [3]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGMKB3xlxPkb"
      },
      "source": [
        "**Normalization**\n",
        "\n",
        "Normalization simply scales the values in the range [0-1]. To apply it on a dataset you just have to subtract the minimum value from each feature and divide it with the range (max – min).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E8TZNiORxvHy"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1DsPxs-4x_BTRfXzjE8Pu_O9dsCYfshff)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "63E6i9mcx45k"
      },
      "source": [
        "**Standardization**\n",
        "\n",
        "Standardization on the other hand transforms data to have a zero mean and one unit standard deviation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VAhobx7lyJq1"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1jaqkqDWPNebx1rrp191w8D4NB5IkbwqK)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cha3yOU2y06q"
      },
      "source": [
        "**Implementation**\n",
        "\n",
        "Implementing the above techniques in Keras is easier than you think. We will show you an example using the Boston Housing dataset that can be easily loaded with Keras."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ROO7qWcpy9a0"
      },
      "source": [
        "from keras.datasets import boston_housing\n",
        "# data is returned as a tuple for the training and the testing datasets\n",
        "(X_train, y_train), (X_test, y_test) = boston_housing.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fNvfq35MzLkU"
      },
      "source": [
        "Let us look at the first example in the training dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a-dMYQKpzOqD"
      },
      "source": [
        "print(X_train[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JnphcnsdzYWc"
      },
      "source": [
        "See the different scales? To solve this we will use the popular Scikit-Learn library.\n",
        "\n",
        "Use the MinMaxScaler for data normalization:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7v2Gv7bizfPM"
      },
      "source": [
        "from sklearn.preprocessing import MinMaxScaler\n",
        "scaler = MinMaxScaler()\n",
        "X_normalized = scaler.fit_transform(X_train)\n",
        "print(X_normalized[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kKDG6Zh-zvYj"
      },
      "source": [
        "OR, use the StandardScaler to standardize:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q8RC5Wk1zxE7"
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "scaler = StandardScaler()\n",
        "X_scaled = scaler.fit_transform(X_train)\n",
        "print(X_scaled[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WngdvLztz8Qt"
      },
      "source": [
        "## **2.4 Working with Categorical Data**[3][9]\n",
        "\n",
        "Categorical data need special treatment because they can not be fed to a neural network in their own format (Since neural networks only accept numerical data types).\n",
        "\n",
        "We will introduce two main techniques for handling categorical data: Indexing and OneHotEncoding.\n",
        "\n",
        "**Indexing**\n",
        "\n",
        "Indexing is simply replacing a category name with an index or a number."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7UtrHLda6vd5"
      },
      "source": [
        "import numpy as np\n",
        "data = np.array(['small', 'medium', 'small', 'large', 'xlarge', 'large'])\n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "encoder = LabelEncoder()\n",
        "data_encoded = encoder.fit_transform(data)\n",
        "print(data_encoded)\n",
        "# Output\n",
        "#array([2, 1, 2, 0, 3, 0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JMlKj6wb6_6G"
      },
      "source": [
        "Values ‘small’, ‘medium’, ‘large’, and ‘xlarge’ where replaced by numbers from 0 to 3. To strictly specify the numbers used you may refer to OrdinalEncoder."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uCl3Up6b7INQ"
      },
      "source": [
        "**OneHotEncoding**[3]\n",
        "\n",
        "OneHotEncoding is replacing each element by a list of boolean values with 1 in the present category index and 0 in the others.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WIKagqd67Ut2"
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "data = np.array(['red', 'blue', 'orange', 'white', 'red', 'orange', 'white', 'red'])\n",
        "\n",
        "from sklearn.preprocessing import LabelBinarizer\n",
        "encoder = LabelBinarizer()\n",
        "data_encoded = encoder.fit_transform(data)\n",
        "\n",
        "print(data_encoded)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u8n-JdPH7goO"
      },
      "source": [
        "Instead of replacing each color with a number, it was replaced with a list. The list represents the following: is_blue, is_orange, is_red, and is_white. The value 1 is added to the relative index of each color presence and 0 is added otherwise. Example: The color red is represented by: [0 0 1 0]. This technique is used to break the ordinal relation between number if present."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cHpJ7G167yku"
      },
      "source": [
        "## **2.5 Working with Images [3]**\n",
        "\n",
        "We will discuss only two simple image manipulations in this post. More advanced techniques will be introduced later.\n",
        "\n",
        "In the first phase of this tutorial, we will deal with images as simple array of flat consecutive pixels. But images are normally 2 dimensional with 1 or 3 color channels. Therefore we need to reshape each image as an array before we use it. This is done via the reshape function in Numpy.\n",
        "\n",
        "Suppose we have a list of 500 images each with 28 * 28 pixels and 3 color channels RGB. This list needs to be reshaped into (500, 2352) in order to be fed to the network. 2353 here is the size of each image after resize (28 * 28 * 3).\n",
        "\n",
        "Reshaping this list is very easy using Numpy:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nHlddBWp9T9X"
      },
      "source": [
        "data_reshaped = data.reshape(500, 28*28*3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RSLo_Rtk9hda"
      },
      "source": [
        "Simple! Now since our pixels are numeric values, we need to scale them as well. One simple scaling technique for images is to divide each pixel with 255 (the maximum value for each pixel)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D3Sy-j1s9j3M"
      },
      "source": [
        "images = images / 255."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nj4qo0uT-CaN"
      },
      "source": [
        "#**3-Regression with Keras[2]**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HzXUem-Ot5c5"
      },
      "source": [
        "After two introductory tutorials, its time to build our first neural network! The network we are building solves a simple regression problem. Regression is a process where a model learns to predict a continuous value output for a given input data, e.g. predict price, length, width, etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D9nGFk0AuD2v"
      },
      "source": [
        "## **3.1 Problem Definition**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S4zWbWluuOe5"
      },
      "source": [
        "Our objective is to build prediction model that predicts housing prices from a set of house features. We will use the Boston Housing dataset, which is collected by the U.S Census Service concerning housing in the area of Boston Mass. It was obtained from the StatLib archive, and has been used extensively throughout the literature to benchmark algorithms.\n",
        "\n",
        "The dataset is small in size with only 506 cases. It contains 14 features described as follows:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vvyXeFFnucBg"
      },
      "source": [
        "- CRIM: per capita crime rate by town\n",
        "- ZN: proportion of residential land zoned for lots over 25,000 sq.ft.\n",
        "- INDUS: proportion of non-retail business acres per town.\n",
        "- CHAS: Charles River dummy variable (1 if tract bounds river; 0 otherwise)\n",
        "- NOX: nitric oxides concentration (parts per 10 million)\n",
        "- RM: average number of rooms per dwelling\n",
        "- AGE: proportion of owner-occupied units built prior to 1940\n",
        "- DIS: weighted distances to five Boston employment centres\n",
        "- RAD: index of accessibility to radial highways\n",
        "- TAX: full-value property-tax rate per $10,000\n",
        "- pupil-teacher ratio by town\n",
        "- B: 1000(Bk — 0.63)² where Bk is the proportion of blacks by town\n",
        "- LSTAT: % lower status of the population\n",
        "- MEDV: Median value of owner-occupied homes in $1000’s\n",
        "\n",
        "The goal behind our regression problem is to use the 13 features to predict the value of MEDV (which represents the housing price)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jdpi_K4rvCfp"
      },
      "source": [
        "## **3.2 Loading the Data**\n",
        "\n",
        "Fortunately, Keras has a set of datasets already available. You can access them from keras.dataset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x5YHKhOgvXGZ"
      },
      "source": [
        "from keras.datasets import boston_housing\n",
        "(X_train, y_train), (X_test, y_test) = boston_housing.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fwbel5atvzmt"
      },
      "source": [
        "print(X_train[0], y_train[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "whZL8kvtv7xv"
      },
      "source": [
        "The data is returned as two tuples representing the training and testing splits. The X_train and X_test contain the feature columns, while the y_train and y_test contain the label/output column."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HT086MDYv__8"
      },
      "source": [
        "## **3.3 Preprocessing**\n",
        "\n",
        "As discussed in the previous article, we need to preprocess our data before feeding it to the network. Obviously, our data needs to be rescaled. Time for our buddy (StandarScaler) from the scikit-learn package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Rv_UPBj5wW6Y"
      },
      "source": [
        "from sklearn.preprocessing import StandardScaler\n",
        "scaler = StandardScaler()\n",
        "# first we fit the scaler on the training dataset\n",
        "scaler.fit(X_train)\n",
        "# then we call the transform method to scale both the training and testing data\n",
        "X_train_scaled = scaler.transform(X_train)\n",
        "X_test_scaled = scaler.transform(X_test)\n",
        "# a sample output\n",
        "print(X_train_scaled[0])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qErxmpKTwkDn"
      },
      "source": [
        "Much better! Note that we only rescale the features and not the label column. This dataset is simple and no further preprocessing is needed. Time for the most exciting part…"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FyWsjth6wwip"
      },
      "source": [
        "## **3.4 Building the Model**\n",
        "\n",
        "We will build the model layer by layer in a sequential manner. To do so we have to import 1) the model class 2) and the layer class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4VUuAxNBw-02"
      },
      "source": [
        "from keras import models, layers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t4iXryMaxFrO"
      },
      "source": [
        "Then, we create the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "69k6lIByxJsx"
      },
      "source": [
        "model = models.Sequential()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3TrlFvL8xNtl"
      },
      "source": [
        "And we start adding the layers:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lwqdG98_xRtF"
      },
      "source": [
        "model.add(layers.Dense(8, activation='relu', input_shape=[X_train.shape[1]]))\n",
        "model.add(layers.Dense(16, activation='relu'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EQ-_FAW_xjAC"
      },
      "source": [
        "# output layer\n",
        "model.add(layers.Dense(1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzprTmu5xlDf"
      },
      "source": [
        "Notice that we only specify the input shape for the first layer, all layers later will know automatically their input shape from the previous one.\n",
        "\n",
        "The activation parameter here specifies the function we want to perform on top of the layer to calculate the output = activation(X * W + bias). Relu is a activation function that is used to break the linearity of the model. There are many other activation functions but Relu is one of the most popular in this kind of networks.\n",
        "\n",
        "The output layer is simply a layer with one neuron and linear activation function since we are predicting only one continuous value."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qYUJAUzORv-x"
      },
      "source": [
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "71W2kbQlx6zk"
      },
      "source": [
        "## **3.5-Compiling the Model**\n",
        "\n",
        "After building the network we need to specify two important things: 1) the optimizer and 2) the loss function. The optimizer is responsible for navigating the space to choose the best model parameters, while the loss function is used by the optimizer to know how to move in the search space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xQFbH3D_ySn1"
      },
      "source": [
        "model.compile(optimizer='rmsprop', loss='mse', metrics=['mae'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ixiOJIyzyU1H"
      },
      "source": [
        "Keras supports other optimizers than RMSprop, and you are supposed to do a trial and error process to choose the best one for your problem. But normally RMSprop works fine with its default parameters."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VVMFO38WyYhG"
      },
      "source": [
        "The loss function used is the Mean Squared Error which is the average squared error a point is from the mean value. Keras supports other loss functions as well that are chosen based on the problem type.\n",
        "\n",
        "The metrics shown here has nothing to do with the model training. It is just a user friendly value that is easier to evaluate than the main loss value. Example: an absolute value loss is easier for us to evaluate and make sense of than the squared error."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d55pwxZdyj7W"
      },
      "source": [
        "## **3.6 Model Training**\n",
        "\n",
        "Let the show begin… All is set, we just have to call the fit method to start training…"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-w--wt37y1lc"
      },
      "source": [
        "history = model.fit(X_train_scaled, y_train, validation_split=0.2, epochs=100)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yXKnPTCI-1GO"
      },
      "source": [
        "The fit method takes both the features and the labels, the validation split indicates that the model has to keep 20% of the data as a validation set. The epochs indicate the number of iterations on the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19JW2hHk96ps"
      },
      "source": [
        "Look how the validation loss decreased from 648 to 20. Impressive!\n",
        "\n",
        "Let us plot the training and validation error convergence according to the epoch number:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "utht3eML-V79"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1TtZxfoUxmV4fdVg-pGScnl5qGV6RpToj)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "chgZOyDjy5Nk"
      },
      "source": [
        "We started with an error of 20K per prediction, and went down to around 3K. This is a very acceptable error value for a housing price."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rv_i08Jo_EcY"
      },
      "source": [
        "##**3.7 Evaluation on Test Data**\n",
        "Model evaluation is super easy in Keras. Check the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6lZINwEY_Pa0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "33b8e733-033e-4ecd-e1bb-0cf7ad2bd3a9"
      },
      "source": [
        "model.evaluate(X_test_scaled, y_test)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4/4 [==============================] - 0s 1ms/step - loss: 24.0712 - mae: 3.1230\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[24.071184158325195, 3.1230244636535645]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b5BwREGA_Y6l"
      },
      "source": [
        "The output values represent the loss (Mean Squarred Error) and the metrics (Mean Absolute Error)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EuLkXmAl_b9j"
      },
      "source": [
        "## **3.8 Model Prediction**\n",
        "Using the model for prediction is simpler than you expect. Have a look:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v7sWmfbn_qRb"
      },
      "source": [
        "# we get a sample data (the first 2 inputs from the training data)\n",
        "to_predict = X_train_scaled[:2]\n",
        "# we call the predict method\n",
        "predictions = model.predict(to_predict)\n",
        "# print the predictions\n",
        "print(predictions)\n",
        "# output\n",
        "# array([[13.272537], [39.808475]], dtype=float32)\n",
        "# print the real values\n",
        "print(y_train[:2])\n",
        "# array([15.2, 42.3])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MwlagZ2mOaex"
      },
      "source": [
        "#**4-Classification [4]**\n",
        "\n",
        "**Introduction**\n",
        "\n",
        "Welcome to Part 3 of Deep Learning with Keras. The goal of this series is to get you familiar with the famous deep learning library Keras and how to use it for building various deep learning models. In this part we will focus on classification. Generally speaking, classification is the process of identifying to which predefined set of categories a new observation belongs.\n",
        "\n",
        "Building a classification neural network requires some tweaks to what we have done before. Let us investigate the process next.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gYmWyTKvPWig"
      },
      "source": [
        "## **4.1- Problem Definaiton** \n",
        "\n",
        "Consider a set of images containing handwritten digits from 0 to 9, our goal is to train a model that takes each picture and predict the correct digit it corresponds to. The data we are going to use is the famous MNIST database of handwritten digits. Keras already has this data available and we can load it as in the following:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rVSqlSY2PkWx"
      },
      "source": [
        "from keras.datasets import mnist\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JpbehzAFPuoA"
      },
      "source": [
        "Make sure you have an Internet connection the first time you run the above code since Keras will download the data from the web.\n",
        "\n",
        "The obtained data is a set of 28*28 gray-scale images with 60000 training data and 10000 testing data. Each label is a number from 0-9 that represents the written digit in each image. The figure below shows a snapshot of the data:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YuhBvS8eQIPf"
      },
      "source": [
        "## **4.2-Data Preprocessing**\n",
        "\n",
        "As discussed in a previous article, we need to preprocess our image data. The following code reshapes and scales the images according to the techniques introduced here."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GBnhMvkzQUQx"
      },
      "source": [
        "X_train_final = X_train.reshape(-1, 28*28) / 255.\n",
        "X_test_final = X_test.reshape(-1, 28*28) / 255."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBGx397mQn7F"
      },
      "source": [
        "In addition to modifying images, we need to transform out labels to one-hot-encoded presentations in order to break the existing nominal relation. This is very simple in Keras and could be done as what follows:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nqj9uJLhQyR2"
      },
      "source": [
        "from keras.utils import to_categorical\n",
        "y_train_final = to_categorical(y_train)\n",
        "y_test_final = to_categorical(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZPWhDG11Q-7O"
      },
      "source": [
        "See the difference before and after encoding:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hVHT1Ij-RBJT"
      },
      "source": [
        "y_train[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZYpqMyTvRISJ"
      },
      "source": [
        "y_train_final[0]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ocCnVsuSRSep"
      },
      "source": [
        "## **4.3-Building the Network** \n",
        "\n",
        "Time to start building the network. We will build a simple one with one hidden layer and a special output layer that we will talk about soon. Let us start:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c0Vaw29y1WuI"
      },
      "source": [
        "from keras import models, layers\n",
        "model = models.Sequential()\n",
        "model.add(layers.Dense(512, activation='relu', input_shape=(28*28, )))\n",
        "model.add(layers.Dense(10, activation='softmax'))\n",
        "model.compile('rmsprop', 'categorical_crossentropy', metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "06PLqZ8C1sPM"
      },
      "source": [
        "We have two new things here: 1) activation = ‘softmax’ and loss=’categorical_crossentropy’.\n",
        "\n",
        "Softmax is a special activation function that transforms the output into probability values of each class. Therefore, with Dense(10) we will have 10 neurons each representing the probability of a given digit.\n",
        "\n",
        "Categorical Cross-entropy is simply a loss function that calculates the error between the predicted digit and the actual one. We use categorical cross-entropy when we have 3 or more classes and binary cross-entropy when we have 2 classes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A5OqfErk1uqy"
      },
      "source": [
        "##**4.4 Training the Model**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nNNDIENj17CK",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "6489bc7b-82fd-4211-a4ed-9fe568f892a1"
      },
      "source": [
        "history = model.fit(X_train_final, y_train_final, epochs=10, batch_size=128, validation_split=0.2)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.2849 - acc: 0.9172 - val_loss: 0.1710 - val_acc: 0.9502\n",
            "Epoch 2/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.1172 - acc: 0.9655 - val_loss: 0.1112 - val_acc: 0.9673\n",
            "Epoch 3/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0763 - acc: 0.9778 - val_loss: 0.0929 - val_acc: 0.9731\n",
            "Epoch 4/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0545 - acc: 0.9838 - val_loss: 0.0819 - val_acc: 0.9766\n",
            "Epoch 5/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0406 - acc: 0.9880 - val_loss: 0.0846 - val_acc: 0.9756\n",
            "Epoch 6/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0313 - acc: 0.9906 - val_loss: 0.0814 - val_acc: 0.9766\n",
            "Epoch 7/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0229 - acc: 0.9935 - val_loss: 0.0827 - val_acc: 0.9785\n",
            "Epoch 8/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0175 - acc: 0.9951 - val_loss: 0.0873 - val_acc: 0.9778\n",
            "Epoch 9/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0138 - acc: 0.9962 - val_loss: 0.0910 - val_acc: 0.9783\n",
            "Epoch 10/10\n",
            "375/375 [==============================] - 4s 10ms/step - loss: 0.0102 - acc: 0.9971 - val_loss: 0.0930 - val_acc: 0.9788\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rG0UBuWV2EFY"
      },
      "source": [
        "## **4.5 Model Evaluation**\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "68OMx2Os2Oxv"
      },
      "source": [
        "model.evaluate(X_test_final, y_test_final)\n",
        "# output \n",
        "# [0.07102660850500979, 0.9799]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BU1wNSiuZ4y1"
      },
      "source": [
        "#5- **Convolutional Neural Networks(CNN)[5]**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8pdTt5wsfYOl"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1ZusaD_qjhqU4JElI1C62oNsApOlm6NOm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XF7lC6KAi4Vy"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1NAGzRwbfJqIqadWK4kfISPMma2KKQ_2i)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2H5m-TAWaQDR"
      },
      "source": [
        "##**5.1-Motivation**\n",
        "\n",
        "In the previous articles, we solved problems with numeric and categorical data, and we learned the different transformations needed for each. Regarding images, we investigated a simple hack that resizes each image into an array of pixels and feed it to the input layer. The approach worked well and we reached around 97% accuracy on MNIST dataset.\n",
        "\n",
        "However, dealing with large images with more complex patterns is completely different. Scientists struggled to reach an acceptable performance to even classify a dog vs cat image. Images like that contain many features that are related in a specific way. For example: some set of pixels in a given order define an edge, a circle, a nose, a mouth, etc. Therefore, we need a special kind of layer that detects these relations.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lVQEqlK3jYdZ"
      },
      "source": [
        "## **5.2-Convolution Layer**\n",
        "\n",
        "Here comes the role of the convolution layer. It is a neural network layer that scans an image, and extracts a set of features from it. Normally, we would accumulate those layers to learn more complex features. This way, the first layers learn very basic features such as horizontal edges, vertical edges, lines, etc. The deeper we go the more complex become the features. Layers will then be able to combine low level features into high level ones. For example: edges and curves could be combined to detect shapes of different heads, noses, ears, etc.\n",
        "\n",
        "Convolution layers made a really high impact on the whole machine and deep learning fields. It allowed us to automated very complex tasks with human-level performance or even outperform humans in some cases. So, pay close attention you are going to have a very powerful weapon in your arsenal."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F5IlyAbTbme8"
      },
      "source": [
        "### **5.2.1 Image Kernels**\n",
        "\n",
        "An kernel (or filter) is simply a small matrix applied to an image with the convolution operator.\n",
        "The process is as follows:\n",
        "\n",
        "- a small matrix of shape (k1, k2) slides over the input,\n",
        "- applies a pairwise multiplication on the two matrices,\n",
        "- the sum of the resulting matrix is taken and the result is put into the final matrix output\n",
        "\n",
        "The idea behind convolution is the use of image kernels. A kernel is a small matrix (usually of size 3 by 3) used to apply effect to an image (like sharpening, blurring…).They’re also used in machine learning for ‘feature extraction’, a technique for determining the most important portions of an image [8]\n",
        "\n",
        "See the image for better clarification:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X45Zvw2Xexqz"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1V73BeBE4oCpjOktfJ_UbZzHwImkmL66G)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AOiJggA4fO6v"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1BtWe8HMd2HwYYLBEgJJhYMPvLI5H75ia)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xchxGYqhgQXP"
      },
      "source": [
        "Applying a filter to an image extracts some features from it. The following image shows how a simple kernel detects edges.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H3xOPEBagWt-"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1g8zrLe9XUtmdH-xPuMpsqiFuo207ywsh)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UZVg9q8HhcUq"
      },
      "source": [
        "The question here is how to get those numbers inside the kernel? Well, why don’t we make the neural network learn the best kernels to classify a set of images? This is core concept behind convolutional neural networks. Convolutional layers act as automatic feature extractors that are learned from the data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VyNyafwqhmCT"
      },
      "source": [
        "**5.2.3 Padding and Striding**[8]\n",
        "\n",
        "Padding: If you see the animation above, notice that during the sliding process, the edges essentially get “trimmed off”, converting a 5×5 feature matrix to a 3×3 one.\n",
        "\n",
        "This is how padding works, pad the edges with extra, “fake” pixels (usually of value 0, hence the oft-used term “zero padding”). This way, the kernel when sliding can allow the original edge pixels to be at its center, while extending into the fake pixels beyond the edge, producing an output the same size as the input.\n",
        "\n",
        "Striding: Often when running a convolution layer, you want an output with a lower size than the input. This is commonplace in convolutional neural networks, where the size of the spatial dimensions are reduced when increasing the number of channels. One way of accomplishing this is by using a pooling layer (eg. taking the average/max of every 2×2 grid to reduce each spatial dimensions in half). Yet another way to do is is to use a stride:\n",
        "\n",
        "The idea of the stride is to skip some of the slide locations of the kernel. A stride of 1 means to pick slides a pixel apart, so basically every single slide, acting as a standard convolution. A stride of 2 means picking slides 2 pixels apart, skipping every other slide in the process, downsizing by roughly a factor of 2, a stride of 3 means skipping every 2 slides, downsizing roughly by factor 3, and so on."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gfv7aZvSiLpA"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1BRVPIh_mDU1TCEvCFiE-nyae5i3fdbrz)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "md-IbsqrkVjE"
      },
      "source": [
        "### **5.2.4-The Multi Channel RGB Image version**[8]\n",
        "\n",
        "We deal with RGB images most of the time.In practicality, most input images have 3 channels, and that number only increases the deeper you go into a network\n",
        "\n",
        "Each filter actually happens to be a collection of kernels, with there being one kernel for every single input channel to the layer, and each kernel being unique.Hence for a RGB image we have 3 input channels and 3 kernels which together makes a filter.\n",
        "\n",
        "Each of the kernels of the filter “slides” over their respective input channels, producing a processed version of each. Some kernels may have stronger weights than others, to give more emphasis to certain input channels than others.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wRefJ61Wk9Jl"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1C5B6H3FQ1K4wI58GRlTCENKzPtyrCQoB)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ckC-ZrTVlcNl"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1md5Mtkh0oem4SOea3f12_l4-1GQFpWmV)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fm5rMT7Wl5nm"
      },
      "source": [
        "Each of the per-channel processed versions are then summed together to form one channel. The kernels of a filter each produce one version of each channel, and the filter as a whole produces one overall output channel."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "StGc8nljl-vg"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1xEH9Ibr7G-jp8ztIUuePp82tzuYio9Ba)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMP_OlOVm9fU"
      },
      "source": [
        "Finally, then there’s the bias term. The way the bias term works here is that each output filter has one bias term. The bias gets added to the output channel so far to produce the final output channel."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OK-nuGaanho4"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1-cvuzfYgOwVW0ktg83f0HVkWkwMR0Fex)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Exu0MhKGpSZy"
      },
      "source": [
        "## **5.3 Problem Definition**[5]\n",
        "\n",
        "In this article we will train a convolutional neural network to classify clothes types from the fashion MNIST dataset.\n",
        "\n",
        "Fashion-MNIST is a dataset of Zalando’s article images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28×28 grayscale image, associated with a label from 10 classes."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K1t5oVM3qo-l"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1ckdRNZhpFKtyrH0G1dg79ALS2Tfej5pX)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aMbryWrdDnrY"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1fiH6y06Th69wDX0Gf8oWxQqHJj5OG548)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbaAvEutDzEJ"
      },
      "source": [
        "## **5.4-Loading the Data**\n",
        "Again we will use Keras to download our data.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ATmY90CWEBuj"
      },
      "source": [
        "from keras.datasets import fashion_mnist\n",
        "(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oT-3n2z2ELpR"
      },
      "source": [
        "## **5.5 Preprocessing Data**[5]\n",
        "We need to do three simple modifications to our data:\n",
        "\n",
        "- Transform the y_train and y_test into one hot encoded versions\n",
        "- Reshape our images into (width, height, number of channels). Since we are - - dealing with gray scale images the number of channels will be one\n",
        "- Scale our images by dividing with 255"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jrnLdUWOEhFF"
      },
      "source": [
        "# to categorical\n",
        "from keras.utils import to_categorical\n",
        "y_train_final = to_categorical(y_train)\n",
        "y_test_final = to_categorical(y_test)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x9HjRFElEqQi"
      },
      "source": [
        "# reshape\n",
        "X_train_final = X_train.reshape(-1, 28, 28, 1) / 255.\n",
        "X_test_final = X_test.reshape(-1, 28, 28, 1) / 255."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P7-hu5BcEwwA"
      },
      "source": [
        "## **5.6 Building the Network**\n",
        "\n",
        "Building a convolutional neural network is not different that building a normal one. The one difference here is that we do not need to reshape our images, because convolutional layers work with 2D images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hY63y6a6FAuS"
      },
      "source": [
        "from keras import models, layers\n",
        "model = models.Sequential()\n",
        "model.add(layers.Conv2D(8, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
        "model.add(layers.Flatten())\n",
        "model.add(layers.Dense(128, activation='relu'))\n",
        "model.add(layers.Dense(10, activation='softmax'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VrTsEQxmFF5R"
      },
      "source": [
        "model.compile('rmsprop', 'categorical_crossentropy', metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sH_6a-WbFS-4"
      },
      "source": [
        "The only new thing here is the first layer and the Flattern layer. We use a Conv2D that is the 2D convolution layer for 2D images. The parameters are the following:\n",
        "\n",
        "- The number of kernels/filters to learn. Here we used 32 kernels. Imagine that each one of these kernels will learn a simple feature like vertical edge detection, horizontal edge detection, etc\n",
        "- The size of the kernel. Here we used a 3 by 3 matrix.\n",
        "- The activation function applied to the final output\n",
        "- The input shape where 28 is the image width and height and 1 is the number of channels (1 since it is a gray scale image, for RGB we use 3)\n",
        "\n",
        "Since the output of a convolution is a multidimensional matrix, we need to reshape the output (as we did before with a regular neural network). The flatten layer here does the same, it unfolds the matrix into an array that is then fed to the next layer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zo9XvxQYGGzI"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=15t9QfUIeDfLeclUxVzjkVQaJuQiDsO92)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KkPnSJSbGR7w"
      },
      "source": [
        "Note: We used a softmax output layer of 10 Dense connected neurons since we have 10 labels to learn."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSpLPMukGWlW"
      },
      "source": [
        "## **6.7-Training the Network**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q-cxuNq7Ggdo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 139
        },
        "outputId": "839256f5-7c30-4d21-c96e-6d7dd959ddaa"
      },
      "source": [
        "history = model.fit(X_train_final, y_train_final, validation_split=0.2, epochs=3, shuffle=True)\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "1500/1500 [==============================] - 23s 15ms/step - loss: 0.2225 - acc: 0.9215 - val_loss: 0.2935 - val_acc: 0.8982\n",
            "Epoch 2/3\n",
            "1500/1500 [==============================] - 23s 15ms/step - loss: 0.2008 - acc: 0.9279 - val_loss: 0.3264 - val_acc: 0.8986\n",
            "Epoch 3/3\n",
            "1500/1500 [==============================] - 23s 15ms/step - loss: 0.1842 - acc: 0.9362 - val_loss: 0.3251 - val_acc: 0.9007\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gSZxTAmnGqyK"
      },
      "source": [
        "With a very simple convolutional network we were able to reach 90% accuracy. The network could be improved for sure by adding more advanced layers and maybe some regularization techniques, but we will keep this for later articles."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZloOF89VLi1_",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "ca664058-4dbb-4f47-b9c4-b86ebee83242"
      },
      "source": [
        "import keras\n",
        "from matplotlib import pyplot as plt\n",
        "plt.plot(history.history['acc'])\n",
        "plt.plot(history.history['val_acc'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3hdZZ33//enSdr0kDZpkpbSJm2BtlBOLQREBFph1CJyVkEBxVEYBUYYBx9hnEf9MfrAXA8zjgdGBx0UZjiKgoyC/AApqBwkhQIF6YECbdpC00N6PiX5Pn+slbAbknRvyM5Oms/ruvbVte91r7W/a2V3ffe97rXWrYjAzMwsW4MKHYCZmfUvThxmZpYTJw4zM8uJE4eZmeXEicPMzHLixGFmZjlx4jDrhqSfS/p2lnVfl/RX+Y7JrNCcOMzMLCdOHGYDgKTiQsdgew8nDuv30lNEX5X0gqQtkv5T0lhJD0jaJOlhSRUZ9U+T9JKkJklzJR2UMW+mpGfT5e4ESjt81sckzU+XfULSYVnGeIqk5yRtlLRc0rc6zD8uXV9TOv/CtHyopH+R9IakDZL+mJbNltTQyX74q3T6W5LulvTfkjYCF0o6WtKT6WeskvRDSYMzlj9Y0kOS1kl6S9I/SNpH0lZJlRn1jpDUKKkkm223vY8Th+0tzgY+BEwFTgUeAP4BqCb5nn8ZQNJU4HbginTe/cD/SBqcHkTvBf4LGA38Il0v6bIzgZuAvwEqgf8A7pM0JIv4tgCfAcqBU4AvSTojXe/ENN4fpDHNAOany10PHAkcm8b0v4DWLPfJ6cDd6WfeCrQAfwdUAe8HTgIuSWMoAx4GfgfsCxwAPBIRbwJzgU9mrPcC4I6I2JVlHLaXceKwvcUPIuKtiFgB/AF4OiKei4jtwD3AzLTeOcBvI+Kh9MB3PTCU5MB8DFAC/FtE7IqIu4FnMj7jYuA/IuLpiGiJiJuBHely3YqIuRHxYkS0RsQLJMlrVjr708DDEXF7+rlrI2K+pEHAXwOXR8SK9DOfiIgdWe6TJyPi3vQzt0XEvIh4KiKaI+J1ksTXFsPHgDcj4l8iYntEbIqIp9N5NwPnA0gqAj5FklxtgHLisL3FWxnT2zp5PyKd3hd4o21GRLQCy4Hx6bwVsfuTP9/ImJ4I/H16qqdJUhNQky7XLUnvk/RoeopnA/BFkl/+pOt4tZPFqkhOlXU2LxvLO8QwVdJvJL2Znr76P1nEAPBrYLqkySStug0R8ed3GZPtBZw4bKBZSZIAAJAkkoPmCmAVMD4ta1ObMb0c+E5ElGe8hkXE7Vl87m3AfUBNRIwCfgy0fc5yYP9OllkDbO9i3hZgWMZ2FJGc5srU8dHXPwJeAaZExEiSU3mZMezXWeBpq+0uklbHBbi1MeA5cdhAcxdwiqST0s7dvyc53fQE8CTQDHxZUomks4CjM5b9CfDFtPUgScPTTu+yLD63DFgXEdslHU1yeqrNrcBfSfqkpGJJlZJmpK2hm4B/lbSvpCJJ70/7VBYBpennlwD/COypr6UM2AhslnQg8KWMeb8Bxkm6QtIQSWWS3pcx/xbgQuA0nDgGPCcOG1AiYiHJL+cfkPyiPxU4NSJ2RsRO4CySA+Q6kv6QX2UsWw9cBPwQWA8sSetm4xLgGkmbgG+QJLC29S4DPkqSxNaRdIwfns6+EniRpK9lHfDPwKCI2JCu86ckraUtwG5XWXXiSpKEtYkkCd6ZEcMmktNQpwJvAouBD2bM/xNJp/yzEZF5+s4GIHkgJzPLhqTfA7dFxE8LHYsVlhOHme2RpKOAh0j6aDYVOh4rLJ+qMrNuSbqZ5B6PK5w0DNziMDOzHLnFYWZmORkQDz6rqqqKSZMmFToMM7N+Zd68eWsiouP9QQMjcUyaNIn6+vpCh2Fm1q9I6vTSa5+qMjOznDhxmJlZTpw4zMwsJwOij6Mzu3btoqGhge3btxc6lLwqLS1lwoQJlJR4zB0z6xkDNnE0NDRQVlbGpEmT2P1hqHuPiGDt2rU0NDQwefLkQodjZnuJAXuqavv27VRWVu61SQNAEpWVlXt9q8rMeteATRzAXp002gyEbTSz3jWgE4eZ2d5o+64WHl24mn/6zcvsbM52iPrsDdg+jkJramritttu45JLLslpuY9+9KPcdtttlJeX5ykyM+tvIoLX1mxh7sJGHlvUyFNL17KjuZUhxYM464jxHLzvqB79PCeOAmlqauLf//3f35E4mpubKS7u+s9y//335zs0M+sHtu5s5slX1/LYokbmLmxk2bqtAOxXNZxPHV3L7GnVHLNfJaUlRT3+2U4cBXLVVVfx6quvMmPGDEpKSigtLaWiooJXXnmFRYsWccYZZ7B8+XK2b9/O5ZdfzsUXXwy8/fiUzZs3c/LJJ3PcccfxxBNPMH78eH79618zdOjQAm+ZmeVDRPBq4+b2VsXTr61jZ3MrQ0uKOHb/Sr5w/GRmTx1DbeWwPa/sPcpr4pA0B/geUAT8NCKu6zB/IsmYytUkw2KeHxENafk9JH0wJcAPIuLH6TJzgXHAtnQ1H46I1e8lzv/vf17i5ZUb38sq3mH6viP55qkHdzn/uuuuY8GCBcyfP5+5c+dyyimnsGDBgvbLZm+66SZGjx7Ntm3bOOqoozj77LOprKzcbR2LFy/m9ttv5yc/+Qmf/OQn+eUvf8n555/fo9thZoWzZUczf1qypr1VsaIpOewdMGYEnzlmIrOmVXPUpNF5aVV0J2+JQ1IRcAPJOMYNwDOS7ouIlzOqXQ/cEhE3SzoRuBa4AFgFvD8idkgaASxIl12ZLndeOv7zXuPoo4/e7V6L73//+9xzzz0ALF++nMWLF78jcUyePJkZM2YAcOSRR/L666/3Wrxm1vMigkVvbWbuwtU8tqiRZ15fx66WYPjgIo49oIovzd6fWVOrqRmd/1ZFd/LZ4jgaWBIRSwEk3QGcDmQmjunAV9LpR4F7ASJiZ0adIeT56q/uWga9Zfjw4e3Tc+fO5eGHH+bJJ59k2LBhzJ49u9N7MYYMGdI+XVRUxLZt295Rx8z6tk3bd/GnJWvaT0Gt2pD8X582toy//sBkZk2tpm7SaAYX952LYPOZOMYDyzPeNwDv61DneeAsktNZZwJlkiojYq2kGuC3wAHAVzNaGwA/k9QC/BL4dnQyjKGki4GLAWpra3tok3pOWVkZmzZ1Pgrnhg0bqKioYNiwYbzyyis89dRTvRydmeVLRPCXVZuYu2g1jy1sZN4b62luDUYMKea4A6q4/KRqTphazb7lfbe/stCd41cCP5R0IfA4sAJoAYiI5cBhkvYF7pV0d0S8RXKaaoWkMpLEcQFwS8cVR8SNwI0AdXV1fW583MrKSj7wgQ9wyCGHMHToUMaOHds+b86cOfz4xz/moIMOYtq0aRxzzDEFjNTM3qsN23bxx8VreGxRcgrqrY07ADho3EguOmE/Zk2t5siJFZQU9Z1WRXfymThWADUZ7yekZe3SVsRZAGlfxtkR0dSxjqQFwPHA3RGxIi3fJOk2klNi70gc/cFtt93WafmQIUN44IEHOp3X1o9RVVXFggUL2suvvPLKHo/PzN6d1tbg5VUb2/sqnl3WREtrUFZazAlTqpk1rZpZU6sZO7K00KG+K/lMHM8AUyRNJkkY5wKfzqwgqQpYFxGtwNUkV1ghaQKwNiK2SaoAjgO+K6kYKI+INZJKgI8BD+dxG8zMstK0dSePL17D3IWreXzRGtZsTloVh4wfyZdm7c/sadXMqCmnuJ+0KrqTt8QREc2SLgMeJLkc96aIeEnSNUB9RNwHzAaulRQkp6ouTRc/CPiXtFzA9RHxoqThwINp0igiSRo/ydc2mJl1pbU1eHHFBuYubGTuotU8v7yJ1oDyYSUcP6Wa2VOTvorqsiF7Xlk/k9c+joi4H7i/Q9k3MqbvBu7uZLmHgMM6Kd8CHNnzkZqZ7dnazTv4Q1urYvEa1m3ZiQSHjR/FZSdOYfa0ag6fUE7RoL374aKF7hw3M+uzWlqD5xuakktlF67mhRUbiIDRwwdzwpQqZk8bw/FTqqgcsfe1KrrjxGFmlqFx0w4eX9TI3EWN/GFxI01bdzFIMKOmnCtOmsrsadUcOn4Ug/byVkV3nDjMbEBrbmll/vKm9r6KBSuSxw9VjRjCSQeOZda0ao4/oIqK4YMLHGnf4cTRT4wYMYLNmzcXOgyzvcLqjduZu6iRxxYmrYqN25spGiSOqC3nyg9PZfa0MUwfN3JAtyq648RhZnu9XS2tPPvGeuamDwv8y6qkVTGmbAhzDtmHWVPHcNwBVYwaVlLgSPsHJ44Cueqqq6ipqeHSS5MrkL/1rW9RXFzMo48+yvr169m1axff/va3Of300wscqVn/tGrDNh5bmCSKPy1Zw6YdzRQPEkdOrOBrcw5k1tRqDhpX5uGV3wUnDoAHroI3X+zZde5zKJx8XZezzznnHK644or2xHHXXXfx4IMP8uUvf5mRI0eyZs0ajjnmGE477TR/sc2ysLO5lfo31rUni4VvJc+CGzeqlI8dPo5ZU6s59oAqRpa6VfFeOXEUyMyZM1m9ejUrV66ksbGRiooK9tlnH/7u7/6Oxx9/nEGDBrFixQreeust9tlnn0KHa9YnNazf2j5WxRNL1rBlZwslReKoSaP5hyMPZNbUMUwdO8I/vnqYEwd02zLIp0984hPcfffdvPnmm5xzzjnceuutNDY2Mm/ePEpKSpg0aVKnj1M3G6h2NLfwzGvrmbtwNXMXNbJkdXLByPjyoZwxczyzp43h/ftXMmKID2355L1bQOeccw4XXXQRa9as4bHHHuOuu+5izJgxlJSU8Oijj/LGG28UOkSzglu2diuPLVqdtCpeXcu2XS0MLhrE+/YbzblH1TB72hj2rx7uVkUvcuIooIMPPphNmzYxfvx4xo0bx3nnncepp57KoYceSl1dHQceeGChQzTrddt3tfDU0rU8ll4uu3TNFgBqRw/jE3UTmD2tmmP2q2TYYB++CsV7vsBefPHtTvmqqiqefPLJTuv5Hg7bm722ZguPpaefnlq6lu27WhlSPIhj9qvkgvdPZPa0MUyqHOZWRR/hxGFmvW7bzhaeXLomuQJqUSNvrN0KwOSq4Zx7VG17q6K0pKjAkVpnnDjMLO8iglcbt7QPbPT0a+vY2dxKackgjt2/is8fl4ytPbFyeKFDtSwM6MQREXt907eT4djNesWWHc088era9o7thvXbADhgzAguOGYis6dVc9Sk0W5V9EMDNnGUlpaydu1aKisr99rkERGsXbuW0tL+OTyl9S8RweLVm9tbFc+8tp6dLa0MG1zEsftX8cVZ+zNrajU1o4cVOlR7jwZs4pgwYQINDQ00NjYWOpS8Ki0tZcKECYUOw/ZSm7bv4k9L2q6AWs3KDcl9R9PGlnHhByYxe2o1dZNGM7i4/w+Xam/La+KQNAf4Hskwrz+NiOs6zJ9IMs54NbAOOD8iGtLye4BBQAnwg4j4cbrMkcDPgaEkowteHu/ifExJSQmTJ09+t5tmNiBFBK+8uSkZ2GjRaupfX09zazBiSDHHHVDF355Uzayp1exbPrTQoVoe5S1xSCoCbgA+BDQAz0i6LyJezqh2PXBLRNws6UTgWuACYBXw/ojYIWkEsCBddiXwI+Ai4GmSxDEHeCBf22E20G3Ytos/LVnTfgrqrY07ADho3EguOmE/Zk2t5siJFZQUuVUxUOSzxXE0sCQilgJIugM4HchMHNOBr6TTjwL3AkTEzow6Q0haHkgaB4yMiKfS97cAZ+DEYdZjIoKXVm5svwFv3rL1tLQGZaXFnDAlaVHMmlbN2JHuOxuo8pk4xgPLM943AO/rUOd54CyS01lnAmWSKiNiraQa4LfAAcBXI2KlpLp0PZnrHN/Zh0u6GLgYoLa2tgc2x2zv1bR1J39YvIa5Cxt5fHEjjZuSVsUh40fypVn7M2taNTNryil2q8IofOf4lcAPJV0IPA6sAFoAImI5cJikfYF7Jd2dy4oj4kbgRoC6ujpfk2qWobU1WLByQ9pX0chzy9bTGlA+rITjp1Qze2o1x0+tYkyZWxX2TvlMHCuAmoz3E9KydmmfxVkAaV/G2RHR1LGOpAXA8cCf0vV0uU4z69y6LTv5w+LkEeSPL2pk7ZadSHDY+FFcduIUZk2tZkZNOUUeLtX2IJ+J4xlgiqTJJAf3c4FPZ1aQVAWsi4hW4GqSK6yQNAFYGxHbJFUAxwHfjYhVkjZKOoakc/wzwA/yuA1m/VZLa/BCQxNz08d6vNDQRASMHj6YE6ZUMXvaGI6fUkXliCGFDtX6mbwljoholnQZ8CDJ5bg3RcRLkq4B6iPiPmA2cK2kIDlVdWm6+EHAv6TlAq6PiLanAV7C25fjPoA7xs3ardm8g8fTgY3+sLiR9Vt3IcGMmnKuOGkqs6dVc8j4UW5V2HuigfBIirq6uqivry90GGY9rrmllefbWhULG3lxxQYAqkYM5oSp1Umr4oAqKoYPLnCk1h9JmhcRdR3LC905bmY5Wr1pe/tTZf+4eA0btu1ikOCI2gqu/PBUZk8bw/RxIxnkVoXliROHWR+3q6WV55Y1JcOlLmzk5VUbARhTNoQPTx/L7GljOO6AKkYNKylwpDZQOHGY9UFvbtje/lTZPy5Zw6btzRQPEkdMrOB/zZnG7KljOGhc2V77gE7r25w4zPqAnc2tzHtjPXMXreaxhY288uYmAPYZWcoph45j9rRqjj2gipGlblVY4TlxmBVARPDami08tXQdcxeu5olX17J5RzMlReKoSaO5+uQDmT1tDFPHjnCrwvocJw6zXrBh2y6eX97Ec8uaeG75euYvb6Jp6y4AxpcP5bQZ+zJ7atKqGDHE/y2tb/M31KyHtbQGi1dvSpLEsvU8u6yJJas3AyDB1DFlzDl4H46oreCIiRXsXz3crQrrV5w4zN6jtZt3tLcknlvWxPPLm9iyswVI7tKeWVPOGTP2ZWZtBYdNGEWZ+ymsn3PiMMvBzuZWXnlzY3tr4rnlTbyxdisAxYPEQeNGcvaREziitoKZteXUjh7m1oTtdZw4zLqxasO2t5PEsiZeXLGBHc2tAIwdOYQjais47321zKyt4JB9RzF0cFGBIzbLPycOs9T2XS0sWLFht9NOq9IxtAcXD+LQ8aO44JiJzKyt4IiJ5Ywb5eFRbWBy4rABKSJYtm7rbqecXl65kebW5NlttaOHcfTk0cysKWdmbQUHjRvJ4GIPYmQGThw2QGze0cwLy5t4bvnbp53WbklGKB42uIjDJ5Rz8Qn7MTPtm6jyo8bNuuTEYXud1tZg6ZrNPJvRN7HorU2kjQn2rx7OiQeOaU8SU8eW+THjZjlw4rB+r2nrzrQlkSSK+cub2LS9GYCRpcXMrK1gziH7MLO2ghkTyv0wQLP3yInD+pXmllYWvpXcXPfssvXMX9bE0jVbABgkmLbPSE47fN/21sTkyuF+vLhZD3PisD5t9abtaUsiaU280LCBbbuSm+uqRgxmZm0FH6+bwMya5Oa64X5ch1ne5fV/maQ5wPdIho79aURc12H+RJJxxquBdcD5EdEgaQbwI2Ak0AJ8JyLuTJf5OTAL2JCu5sKImJ/P7bDesaO5hZdXbmxvTTy3rIkVTdsAKCkS0/cdxTlH1XDExApm1pQzoWKob64zK4C8JQ5JRcANwIeABuAZSfdFxMsZ1a4HbomImyWdCFwLXABsBT4TEYsl7QvMk/RgRDSly301Iu7OV+yWfxHBiqZtb7cmlq/npRUb2dmS3Fw3vnwoM2rL+dwHJjGztoKD9x1JaYlvrjPrC/LZ4jgaWBIRSwEk3QGcDmQmjunAV9LpR4F7ASJiUVuFiFgpaTVJq6QJ65e27mzmxYYNPLe8iWffSO6baNy0A4DSkkEcNr4tSST3TYwdWVrgiM2sK/lMHOOB5RnvG4D3dajzPHAWyemsM4EySZURsbatgqSjgcHAqxnLfUfSN4BHgKsiYkfHD5d0MXAxQG1t7XvfGsta21gTmXdgv/LmJlrS62EnVw3n+AOq2pPEtH3KKCnyzXVm/UWhexKvBH4o6ULgcWAFSZ8GAJLGAf8FfDYiWtPiq4E3SZLJjcDXgGs6rjgibkznU1dXF/nbBNu4PWOsifQu7LaxJkYMKWZGTTmXzN6fmbXlzKipYPTwwQWO2Mzei3wmjhVATcb7CWlZu4hYSdLiQNII4Oy2fgxJI4HfAl+PiKcyllmVTu6Q9DOS5GO9pONYE88ta2JJ42YikrEmpowZwZyD92lvTexfPcI315ntZfKZOJ4BpkiaTJIwzgU+nVlBUhWwLm1NXE1yhRWSBgP3kHSc391hmXERsUrJ5TRnAAvyuA0D3trNO5ifMXLd88s3sHlHcnNdxbASZtZWtN83cVjNKI+JbTYA5C1xRESzpMuAB0kux70pIl6SdA1QHxH3AbOBayUFyamqS9PFPwmcAFSmp7Hg7ctub5VUDQiYD3wxX9sw0OxqaeUvqzofa6JokDhoXBlnzhzPERPLmVlTwcRKjzVhNhApYu8//V9XVxf19fWFDqPPeXPD9vYE0XZzXdtYE2PKhrQPRjSztoJDx3usCbOBRtK8iKjrWF7oznHrJdt3tfDSyg08+0YnY00UDeKQ8SM5/5iJ7cli3KhStybMrFNOHHuhiGD5um3tCeK5Zet5edVGdrUkrcua0UM5atLo9tbEQePKGFLs1oSZZceJYy+weUczLzQ07XalU+ZYE4dNGMUXjt+vfVCi6jKPNWFm754TRz+z+1gTSaLoONbEBw8ck7QmaiqYOnYExb65zsx6kBNHH9e0dSfzlze1D0rUcayJGbUVfCS9b2JGTTnlw3xznZnllxNHH5I51kTbfRNLG3cfa+LUw/dtP+W0X5XHmjCz3ufEUUCrN21n/rKm3S6H3bozeeJK5fBkrImzj5jAzNpyDptQzgiPNWFmfYCPRL0kc6yJtkTRsD4Za6J4kDh435F8sq6GmbXlHFFb4bEmzKzPcuLIg4hgZdvNdWnfxIKVG9mZ3ly376hSZtZWcOGxyWPED953lMeaMLN+w4mjB2SONdGWLFanY00MKR7EYRNGceGxkzgifTrsPqM81oSZ9V9OHDmKCF5fu/Xt1sTy9fxl1dtjTUyqHMYH2saaqKngwHEea8LM9i5OHHuwp7EmDq8ZxZdm7d9+OWzlCN9cZ2Z7NyeOblx627Pc/+Kq3caa+Mj0t8eaOGCMx5ows4HHiaMb79+vkmljy5hZW87hNeUea8LMDCeObp1/zMRCh2Bm1udk1Wsr6VeSTpHkXl4zswEu20Tw7yTDvi6WdJ2kaXmMyczM+rCsEkdEPBwR5wFHAK8DD0t6QtLnJHV54l/SHEkLJS2RdFUn8ydKekTSC5LmSpqQls+Q9KSkl9J552QsM1nS0+k670zHJzczs16S9aknSZXAhcAXgOeA75Ekkoe6qF8E3ACcDEwHPiVpeodq1wO3RMRhwDXAtWn5VuAzEXEwMAf4N0nl6bx/Br4bEQcA64HPZ7sNZmb23mXbx3EP8AdgGHBqRJwWEXdGxN8CI7pY7GhgSUQsjYidwB3A6R3qTAd+n04/2jY/IhZFxOJ0eiWwGqhW8vCmE4G702VuBs7IZhvMzKxnZNvi+H5ETI+IayNiVeaMzgYyT40Hlme8b0jLMj0PnJVOnwmUpS2bdpKOBgYDrwKVQFNENHezzrblLpZUL6m+sbGx+60zM7OsZZs4pmecKkJShaRLeuDzrwRmSXoOmAWsAFoyPmcc8F/A5yKiNZcVR8SNEVEXEXXV1dU9EKqZmUH2ieOiiGhqexMR64GL9rDMCqAm4/2EtKxdRKyMiLMiYibw9bSsCUDSSOC3wNcj4ql0kbVAuaTirtZpZmb5lW3iKFLG4BBpx/eermZ6BpiSXgU1GDgXuC+zgqSqjHtDrgZuSssHA/eQdJy39WcQEUHSF/LxtOizwK+z3AYzM+sB2SaO3wF3SjpJ0knA7WlZl9J+iMuAB4G/AHdFxEuSrpF0WlptNrBQ0iJgLPCdtPyTwAnAhZLmp68Z6byvAV+RtISkz+M/s9wGMzPrAUp+xO+hUtIq+BvgpLToIeCnEdHS9VJ9R11dXdTX1xc6DDOzfkXSvM4ugMrqWVVpx/SP0peZmQ1gWSUOSVNIbs6bDrQPXxcR++UpLjMz66Oy7eP4GUlroxn4IHAL8N/5CsrMzPqubBPH0Ih4hKRP5I2I+BZwSv7CMjOzvirb8Th2pB3kiyVdRnLvRFePGjEzs71Yti2Oy0meU/Vl4EjgfJJ7KMzMbIDZY4sjvdnvnIi4EtgMfC7vUZmZWZ+1xxZHeq/Gcb0Qi5mZ9QPZ9nE8J+k+4BfAlrbCiPhVXqIyM7M+K9vEUUrygMETM8oCcOIwMxtgsr1z3P0aZmYGZH/n+M9IWhi7iYi/7vGIzMysT8v2VNVvMqZLSUbrW9nz4ZiZWV+X7amqX2a+l3Q78Me8RGRmZn1atjcAdjQFGNOTgZiZWf+QbR/HJnbv43iTZEAlMzMbYLI9VVWW70DMzKx/yOpUlaQzJY3KeF8u6YwslpsjaaGkJZKu6mT+REmPSHpB0lxJEzLm/U5Sk6TfdFjm55Je62RIWTMz6wXZ9nF8MyI2tL2JiCbgm90tkD7j6gbgZJIBoD4laXqHatcDt0TEYcA1JINFtfm/wAVdrP6rETEjfc3PchvMzKwHZJs4Oqu3p9NcRwNLImJpROwE7gBO71BnOvD7dPrRzPnp+B+bsozPzMx6SbaJo17Sv0raP339KzBvD8uMB5ZnvG9IyzI9D5yVTp8JlEmqzCKe76Snt74raUhnFSRdLKleUn1jY2MWqzQzs2xkmzj+FtgJ3EnSctgOXNoDn38lMEvSc8AskgGiWvawzNXAgcBRwGi6uLorIm6MiLqIqKuuru6BUM3MDLK/qmoL8I7O7T1YAdRkvJ+QlmWudyVpi0PSCODstP+ku1hWpZM70kehXJljXGZm9h5ke1XVQ5LKM95XSHpwD4s9A0yRNFnSYOBc4L4O661Kh6SFpCVxUxaxjEv/FXAGsCCbbTAzs56R7amqqsyWQESsZw93jkdEM3AZ8CDwF+CuiHhJ0jWSTkurzbALlToAABDpSURBVAYWSloEjAW+07a8pD+QjP9xkqQGSR9JZ90q6UXgRaAK+HaW22BmZj0g24cctkqqjYhlAJIm0cnTcjuKiPuB+zuUfSNj+m7g7i6WPb6L8hM7Kzczs96RbeL4OvBHSY8BAo4HLs5bVGZm1mdl2zn+O0l1JMniOeBeYFs+AzMzs74p24ccfgG4nOTKqPnAMcCT7D6UrJmZDQDZdo5fTnLfxBsR8UFgJtDtZbNmZrZ3yjZxbI+I7QCShkTEK8C0/IVlZmZ9Vbad4w3pfRz3Ag9JWg+8kb+wzMysr8q2c/zMdPJbkh4FRgG/y1tUZmbWZ2Xb4mgXEY/lIxAzM+sf3u2Y42ZmNkA5cZiZWU6cOMzMLCdOHGZmlhMnDjMzy4kTh5mZ5cSJw8zMcuLEYWZmOXHiMDOznOQ1cUiaI2mhpCWSrupk/kRJj0h6QdJcSRMy5v1OUpOk33RYZrKkp9N13pmOZ25mZr0kb4lDUhFwA3AyMB34lKTpHapdD9wSEYcB1wDXZsz7v8AFnaz6n4HvRsQBwHrg8z0du5mZdS2fLY6jgSURsTQidgJ3AKd3qDMd+H06/Wjm/Ih4BNiUWVmSSAaPahun/GbgjJ4P3czMupLPxDEeWJ7xviEty/Q8cFY6fSZQJqmym3VWAk0R0dzNOgGQdLGkekn1jY2NOQdvZmadK3Tn+JXALEnPAbOAFUBLT6w4Im6MiLqIqKuuru6JVZqZGe/iseo5WAHUZLyfkJa1i4iVpC0OSSOAsyOiuyFp1wLlkorTVsc71mlmZvmVzxbHM8CU9CqowcC5wH2ZFSRVSWqL4Wrgpu5WGBFB0hfy8bTos8CvezRqMzPrVt4SR9oiuAx4EPgLcFdEvCTpGkmnpdVmAwslLQLGAt9pW17SH4BfACdJapD0kXTW14CvSFpC0ufxn/naBjMzeyclP+L3bnV1dVFfX1/oMMzM+hVJ8yKirmN5oTvHzcysn3HiMDOznDhxmJlZTpw4zMwsJ04cZmaWEycOMzPLiROHmZnlxInDzMxy4sRhZmY5ceIwM7OcOHGYmVlOnDjMzCwnThxmZpYTJw4zM8uJE4eZmeXEicPMzHLixGFmZjnJa+KQNEfSQklLJF3VyfyJkh6R9IKkuZImZMz7rKTF6euzGeVz03XOT19j8rkNZma2u+J8rVhSEXAD8CGgAXhG0n0R8XJGteuBWyLiZkknAtcCF0gaDXwTqAMCmJcuuz5d7ryI8FiwZmYFkM8Wx9HAkohYGhE7gTuA0zvUmQ78Pp1+NGP+R4CHImJdmiweAubkMVYzM8tSPhPHeGB5xvuGtCzT88BZ6fSZQJmkyiyW/Vl6mup/S1JnHy7pYkn1kuobGxvfy3aYmVmGQneOXwnMkvQcMAtYAbTsYZnzIuJQ4Pj0dUFnlSLixoioi4i66urqnozZzGxAy2fiWAHUZLyfkJa1i4iVEXFWRMwEvp6WNXW3bES0/bsJuI3klJiZmfWSfCaOZ4ApkiZLGgycC9yXWUFSlaS2GK4GbkqnHwQ+LKlCUgXwYeBBScWSqtJlS4CPAQvyuA1mZtZB3hJHRDQDl5Ekgb8Ad0XES5KukXRaWm02sFDSImAs8J102XXAP5Ekn2eAa9KyISQJ5AVgPkkr5Cf52gYzM3snRUShY8i7urq6qK/31btmZrmQNC8i6jqWF7pz3MzM+hknDjMzy4kTh5mZ5cSJw8zMcuLEYWZmOXHiMDOznDhxmJlZTpw4zMwsJ04cZmaWEycOMzPLiROHmZnlxInDzMxy4sRhZmY5ceIwM7OcOHGYmVlOnDjMzCwnxYUOwMzMshQBOzbCtvWwdV3yb3evrevggnugvKZHw8hr4pA0B/geUAT8NCKu6zB/Isk449XAOuD8iGhI530W+Me06rcj4ua0/Ejg58BQ4H7g8hgIwxia2d6jtfXtBLCtLQE0dZMQMupES9frHVwGQytgaHny7z6H5CX8vCUOSUXADcCHgAbgGUn3RcTLGdWuB26JiJslnQhcC1wgaTTwTaAOCGBeuux64EfARcDTJIljDvBAvrbDzKxLra2wY0N6wO/sV/+6d7YAtq2H7U0QrV2vd8jItw/+Qytg1HgYOvrt90MrYFiH96XlUDy4VzY7ny2Oo4ElEbEUQNIdwOlAZuKYDnwlnX4UuDed/gjwUESsS5d9CJgjaS4wMiKeSstvAc7AicPM3ovWFti+IeOXfxcH/HckhSaS37ZdGDJy94P7qJp3HvCHVnRICuVQVNJrm/5u5DNxjAeWZ7xvAN7Xoc7zwFkkp7POBMokVXax7Pj01dBJ+TtIuhi4GKC2tvZdb4SZ9SO7JYDODvidJICt65Jluk0Ao2BYxoG+YmInB/wOrYDSUX0+Abxbhe4cvxL4oaQLgceBFUA3J/CyFxE3AjcC1NXVuQ/ErD9pad49AXR1wO+YFLZv6H69paN2P+BXTO78tE9mndJRUFToQ2Xfks+9sQLI7MqfkJa1i4iVJC0OJI0Azo6IJkkrgNkdlp2bLj+hu3WaWR/S0pycz9/TAb9jUug2AejtBDBsdPKq3L+L0z4ZSaF0FAwq6rVN35vlM3E8A0yRNJnk4H4u8OnMCpKqgHUR0QpcTXKFFcCDwP+RVJG+/zBwdUSsk7RR0jEkneOfAX6Qx20wM4CWXW9f9bOnjt/2V1PScdwl7d4BPKwKKqd03fmbeQrICaCg8pY4IqJZ0mUkSaAIuCkiXpJ0DVAfEfeRtCqulRQkp6ouTZddJ+mfSJIPwDVtHeXAJbx9Oe4DuGPcLHstu7I45dMxKTQll452RYOSK3raDuwjxkD1tO47f4eNTvoNBvke5P5IA+EWiLq6uqivry90GGY9p3lnFzd9ddUKSFsLOzd1vU4N2sPVPm2tgPLdy5wA9lqS5kVEXcdy9/iYFVLzjtwv/9y2HnZu7nqdKtr9wF42DsYe3Plpn90SwEgnAMuKE4flX0Rys1NrC7Q2J69oyXifWd769nRrWida3lnWvo628tYO6+6qrGUPn9dF2TviyObzOm5fJ9vS3SWgg4p3P7CPHA9jD8loBZR33hcwZCRIvfbntYHHiaMnRWR5oOt48GrppmxPB9YuDrbv6sDaUwfyTralrxlU/PZLRUln66Di3f9VZllRRt10ueLBMGhY1+vodL3FySmhzHV0dWpoSJkTgPVJThzd+Z8r4PU/Zv8LsrtHCBSCBu1+oBs0qMPBssNBcbeDWuYBsjRjHV0cRNvW3eV693QQ7eLA2unn7SGGPdb16Riz98KJozvlNbDPoV38Cs04KHVWltVBtKcOrF3E5l+rZpYHThzdOf7vCx2BmVmf4za7mZnlxInDzMxy4sRhZmY5ceIwM7OcOHGYmVlOnDjMzCwnThxmZpYTJw4zM8vJgHisuqRG4I13uXgVsKYHw+kpjis3jis3jis3e2tcEyOiumPhgEgc74Wk+s6eR19ojis3jis3jis3Ay0un6oyM7OcOHGYmVlOnDj27MZCB9AFx5Ubx5Ubx5WbARWX+zjMzCwnbnGYmVlOnDjMzCwnAzpxSJojaaGkJZKu6mT+EEl3pvOfljQpY97VaflCSR/p5bi+IullSS9IekTSxIx5LZLmp6/7ejmuCyU1Znz+FzLmfVbS4vT12V6O67sZMS2S1JQxLy/7S9JNklZLWtDFfEn6fhrzC5KOyJiXz321p7jOS+N5UdITkg7PmPd6Wj5fUn0vxzVb0oaMv9U3MuZ1+/fPc1xfzYhpQfp9Gp3Oy+f+qpH0aHoceEnS5Z3Uyd93LCIG5AsoAl4F9gMGA88D0zvUuQT4cTp9LnBnOj09rT8EmJyup6gX4/ogMCyd/lJbXOn7zQXcXxcCP+xk2dHA0vTfinS6orfi6lD/b4GbemF/nQAcASzoYv5HgQcAAccAT+d7X2UZ17Ftnwec3BZX+v51oKpA+2s28Jv3+vfv6bg61D0V+H0v7a9xwBHpdBmwqJP/j3n7jg3kFsfRwJKIWBoRO4E7gNM71DkduDmdvhs4SZLS8jsiYkdEvAYsSdfXK3FFxKMRsTV9+xQwoYc++z3F1Y2PAA9FxLqIWA88BMwpUFyfAm7voc/uUkQ8DqzrpsrpwC2ReAoolzSO/O6rPcYVEU+knwu9993KZn915b18L3s6rl75bgFExKqIeDad3gT8BRjfoVrevmMDOXGMB5ZnvG/gnTu+vU5ENAMbgMosl81nXJk+T/Krok2ppHpJT0k6o4diyiWus9Nm8d2SanJcNp9xkZ7Smwz8PqM4X/trT7qKO5/7Klcdv1sB/P+S5km6uADxvF/S85IekHRwWtYn9pekYSQH319mFPfK/lJyCn0m8HSHWXn7jhXnGqT1HZLOB+qAWRnFEyNihaT9gN9LejEiXu2lkP4HuD0idkj6G5LW2om99NnZOBe4OyJaMsoKub/6LEkfJEkcx2UUH5fuqzHAQ5JeSX+R94ZnSf5WmyV9FLgXmNJLn52NU4E/RURm6yTv+0vSCJJkdUVEbOzJdXdnILc4VgA1Ge8npGWd1pFUDIwC1ma5bD7jQtJfAV8HTouIHW3lEbEi/XcpMJfkl0ivxBURazNi+SlwZLbL5jOuDOfS4VRCHvfXnnQVdz73VVYkHUby9zs9Ita2lWfsq9XAPfTc6dk9ioiNEbE5nb4fKJFURR/YX6nuvlt52V+SSkiSxq0R8atOquTvO5aPjpv+8CJpbS0lOXXR1ql2cIc6l7J75/hd6fTB7N45vpSe6xzPJq6ZJB2CUzqUVwBD0ukqYDE91FGYZVzjMqbPBJ6KtzvjXkvjq0inR/dWXGm9A0k6K9Ub+ytd5yS67uw9hd07Lv+c732VZVy1JH12x3YoHw6UZUw/Aczpxbj2afvbkRyAl6X7Lqu/f77iSuePIukHGd5b+yvd9luAf+umTt6+Yz22c/vji+Sqg0UkB+Gvp2XXkPyKBygFfpH+R/ozsF/Gsl9Pl1sInNzLcT0MvAXMT1/3peXHAi+m/3leBD7fy3FdC7yUfv6jwIEZy/51uh+XAJ/rzbjS998CruuwXN72F8mvz1XALpJzyJ8Hvgh8MZ0v4IY05heBul7aV3uK66fA+ozvVn1avl+6n55P/8Zf7+W4Lsv4bj1FRmLr7O/fW3GldS4kuVgmc7l876/jSPpQXsj4W320t75jfuSImZnlZCD3cZiZ2bvgxGFmZjlx4jAzs5w4cZiZWU6cOMzMLCdOHGZ9XPpk2N8UOg6zNk4cZmaWEycOsx4i6XxJf07HX/gPSUWSNqfjgbykZOyU6rTujPTBii9IukdSRVp+gKSH04f5PStp/3T1I9IHR74i6db0Kc1mBeHEYdYDJB0EnAN8ICJmAC3AeSSPm6iPiIOBx4BvpovcAnwtIg4juau3rfxW4IaIOJzkzvZVaflM4AqSsWD2Az6Q940y64KfjmvWM04ieajjM2ljYCiwGmgF7kzr/DfwK0mjgPKIeCwtvxn4haQyYHxE3AMQEdsB0vX9OSIa0vfzSZ6f9Mf8b5bZOzlxmPUMATdHxNW7FUr/u0O9d/uMnx0Z0y34/64VkE9VmfWMR4CPp2MvIGl0OnDUIODjaZ1PA3+MiA3AeknHp+UXAI9FMpJbQ9uAUkrGvB/Wq1thlgX/ajHrARHxsqR/JBnxbRDJ01QvBbYAR6fzVpP0gwB8FvhxmhiWAp9Lyy8A/kPSNek6PtGLm2GWFT8d1yyPJG2OiBGFjsOsJ/lUlZmZ5cQtDjMzy4lbHGZmlhMnDjMzy4kTh5mZ5cSJw8zMcuLEYWZmOfl/5eQ6xdy933MAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_Qt39ZmL6ZU",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 295
        },
        "outputId": "b99a4565-e5b4-4e30-c868-e23b670d3087"
      },
      "source": [
        "plt.plot(history.history['loss'])\n",
        "plt.plot(history.history['val_loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train', 'val'], loc='upper left')\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xdZZ3v8c8vlyZtbm2TQNukpSltgbYghVC5iCDXAgN1BpWrB0a0o8JRh9HXwcGZ40Gc4cg5M6MjKtXhjM4IyEWcqpRyK6BCoSlWeoHeW5oUaFp6S+9JfuePtfbOSrqS7tCs7Fy+79crr+79rLX2/mV1J9+s51nrWebuiIiIdJST7QJERKRvUkCIiEgsBYSIiMRSQIiISCwFhIiIxFJAiIhILAWESA8ws383s7szXHeDmV10tK8jkjQFhIiIxFJAiIhILAWEDBph187XzOwNM9tjZv9mZsea2Twz221mz5rZiMj6V5nZcjPbYWYvmNlJkWXTzez1cLtfAIUd3uvPzGxJuO3LZnbKB6z5c2a2xszeN7O5ZjYmbDcz+2cz22Jmu8xsqZlNC5ddbmYrwtoazOyrH2iHyaCngJDB5mrgYmAycCUwD/hboJLg5+FLAGY2GXgI+Eq47Eng12Y2xMyGAL8C/gMYCTwavi7httOBB4C/AsqB+4G5ZlbQnULN7ALgH4FPAaOBjcDD4eJLgI+G30dZuM62cNm/AX/l7iXANOD57ryvSIoCQgabf3X399y9Afgd8Kq7/9Hd9wNPANPD9a4Bfuvuz7j7IeD/AEOBs4EzgXzgX9z9kLs/BiyKvMds4H53f9XdW9z9p8CBcLvuuAF4wN1fd/cDwNeBs8xsPHAIKAFOBMzd33T3d8LtDgFTzKzU3be7++vdfF8RQAEhg897kcf7Yp4Xh4/HEPzFDoC7twKbgKpwWYO3n+lyY+TxccDfhN1LO8xsBzA23K47OtbQRHCUUOXuzwPfB+4DtpjZHDMrDVe9Grgc2GhmL5rZWd18XxFAASHSmc0Ev+iBoM+f4Jd8A/AOUBW2pYyLPN4EfNvdh0e+hrn7Q0dZQxFBl1UDgLt/z91PB6YQdDV9LWxf5O6zgGMIusIe6eb7igAKCJHOPAJcYWYXmlk+8DcE3UQvA68AzcCXzCzfzP4CmBHZ9sfA583sw+FgcpGZXWFmJd2s4SHgL83s1HD84h8IusQ2mNkZ4evnA3uA/UBrOEZyg5mVhV1ju4DWo9gPMogpIERiuPtK4EbgX4GtBAPaV7r7QXc/CPwFcDPwPsF4xS8j29YBnyPoAtoOrAnX7W4NzwJ/BzxOcNRyPHBtuLiUIIi2E3RDbQPuDZd9GthgZruAzxOMZYh0m+mGQSIiEkdHECIiEksBISIisRQQIiISSwEhIiKx8rJdQE+pqKjw8ePHZ7sMEZF+ZfHixVvdvTJu2YAJiPHjx1NXV5ftMkRE+hUz29jZMnUxiYhILAWEiIjEUkCIiEisATMGEefQoUPU19ezf//+bJeSuMLCQqqrq8nPz892KSIyQAzogKivr6ekpITx48fTfuLNgcXd2bZtG/X19dTU1GS7HBEZIAZ0F9P+/fspLy8f0OEAYGaUl5cPiiMlEek9AzoggAEfDimD5fsUkd4zoLuYRLJu21pY8yzs3QaWCzk54b+5bf/m5IHltG9L/xvXnnqNvAzW7died+R19ceGhBQQCduxYwcPPvggX/ziF7u13eWXX86DDz7I8OHDE6pMEtHSDJtehVVPBV9bV2W7ou6zuBDrbnhlEkgx79Pp+3W2bqQ9Jy/zdY/6/TIN5/4dtgqIhO3YsYMf/OAHhwVEc3MzeXmd7/4nn3wy6dKkp+zfGRwlrHwK1jwD+7ZDTj6MPwdqPwOTZ8KI8dDaAt7S4d/WLtqbY5a1xqzbob21OfN1O7ant+3G+3n4nrHrpr6/g118380xbUd4/37DMjxa7E4gxaxbMQkuvqvHq1dAJOyOO+5g7dq1nHrqqeTn51NYWMiIESN46623WLVqFR//+MfZtGkT+/fv58tf/jKzZ88G2qYOaWpq4rLLLuMjH/kIL7/8MlVVVfzXf/0XQ4cOzfJ3Nsi9vy4IhFXzYOPLwS+5oSNh0qVwwkw4/gIoLGu/TW4e+pHrIbFhdqTAjTxvbc583S5Du5uv0e33a+nwfbZC88HDQ3RIcSK7edB8Wv/Xr5ezYvOuHn3NKWNK+Z9XTu1ynXvuuYdly5axZMkSXnjhBa644gqWLVuWPh31gQceYOTIkezbt48zzjiDq6++mvLy8navsXr1ah566CF+/OMf86lPfYrHH3+cG2+8sUe/FzmClmaofw1WzmvfdVRxApx1K0y+DMbOCP6ak+Tl5EDOkGxXMeANmoDoK2bMmNHuWoXvfe97PPHEEwBs2rSJ1atXHxYQNTU1nHrqqQCcfvrpbNiwodfqHdT274Q1zwWBsPrpsOsoD45LdR1dCiMnZLtKkcQMmoA40l/6vaWoqCj9+IUXXuDZZ5/llVdeYdiwYZx//vmx1zIUFBSkH+fm5rJv375eqXVQSncdPQUb/xB2HY2ASZcEYwkTLzy860hkgEo0IMxsJvBdIBf4ibvf02H554FbgRagCZjt7ivM7GLgHmAIcBD4mrs/n2StSSkpKWH37t2xy3bu3MmIESMYNmwYb731FgsXLuzl6iToOloUjCWsfAq2rgza1XUkklxAmFkucB9wMVAPLDKzue6+IrLag+7+o3D9q4B/AmYCW4Er3X2zmU0D5gNVSdWapPLycs455xymTZvG0KFDOfbYY9PLZs6cyY9+9CNOOukkTjjhBM4888wsVjqIdNV1dPrNwSCzuo5EEj2CmAGscfd1AGb2MDALSAeEu0dHjYsAD9v/GGlfDgw1swJ3P5BgvYl58MEHY9sLCgqYN29e7LLUOENFRQXLli1Lt3/1q1/t8foGhffXB4Gwcp66jkQylGRAVAGbIs/rgQ93XMnMbgVuJ+hOuiDmda4GXo8LBzObDcwGGDduXA+ULANGawtsei3oOlo1HxrfCtrTXUczoXpGeOqpiMTJ+k+Hu98H3Gdm1wPfAG5KLTOzqcD/Bi7pZNs5wByA2tpaT75a6dP274K1zwVjCaufhn3vh11HZ8NpN6nrSKSbkgyIBmBs5Hl12NaZh4Efpp6YWTXwBPDf3H1tIhVK/9eu6+hlaD0U6Tq6FI6/EIZquhKRDyLJgFgETDKzGoJguBa4PrqCmU1y99Xh0yuA1WH7cOC3wB3u/ocEa5T+prUlOOsodcFauutoMpz5BTjhMnUdifSQxH6K3L3ZzG4jOAMpF3jA3Zeb2V1AnbvPBW4zs4uAQ8B22rqXbgMmAn9vZn8ftl3i7luSqlf6sFTX0ar5QdfR3m3tu44mXwrlx2e7SpEBJ9E/s9z9SeDJDm1/H3n85U62uxu4O8napI/bvqFtrqMNfwi6jgqHB11HJ8xU15FIL9BxeB9TXFxMU1NTtsvofe26juZD45tBu7qORLJGP22SPft3wdrn2y5YS3UdjTsLTvuH4FRUdR2JZI0CImF33HEHY8eO5dZbbwXgm9/8Jnl5eSxYsIDt27dz6NAh7r77bmbNmpXlSntJuuvoKdjwe3UdifRhgycg5t0B7y7t2dccdTJcdk+Xq1xzzTV85StfSQfEI488wvz58/nSl75EaWkpW7du5cwzz+Sqq64amPeVbm2B+rq2uY5SXUflk+DMz4dzHX1YXUcifZB+KhM2ffp0tmzZwubNm2lsbGTEiBGMGjWKv/7rv+all14iJyeHhoYG3nvvPUaNGpXtcnvGgd3t5zpK3Y/5uLPVdSTSjwyegDjCX/pJ+uQnP8ljjz3Gu+++yzXXXMPPf/5zGhsbWbx4Mfn5+YwfPz52mu9+ZfvGtgvW2nUdXRzOdXSRuo5E+pnBExBZdM011/C5z32OrVu38uKLL/LII49wzDHHkJ+fz4IFC9i4cWO2S+y+dNdROJ6wJZyDUV1HIgOGfnp7wdSpU9m9ezdVVVWMHj2aG264gSuvvJKTTz6Z2tpaTjzxxGyXmJkDu4OzjlJzHe3d2tZ1dMm3g1NR1XUkMmAoIHrJ0qVtA+QVFRW88sorsev1uWsgUl1HqbOOWg6q60hkkFBASHutLdCwuG2uo3TX0USYMTs4Shh7prqORAYB/ZRLW9fRqvnBV8euo8kzoWJitqsUkV424APC3Qfm9QUduHfzdhg73o7MdZTqOiqDiRcHRwkTLwymzRaRQWtAB0RhYSHbtm2jvLx8QIeEu7Nt2zYKCws7X6m1BRpeb7tgbcvyoF1dRyLSiQH926C6upr6+noaGxuzXUriCgsLqa6ubt94oKltrqPDuo7uDk5FVdeRiHRiQAdEfn4+NTU12S6jd+14OwiDlfNgw+/UdSQiH9iADohBobU1OOuos66jyTNh3JmQm5/dOkWk31FA9EcHmmDdgvCCtfmwpzHoOhp3lrqORKTHKCD6ix2bInMddeg6mjwTJl2kriMR6VEKiL6qtRU2v952wdp7y4L2kcer60hEekWiAWFmM4HvArnAT9z9ng7LPw/cCrQATcBsd18RLvs6cEu47EvuPj/JWvuEI3YdzYSKSdmuUkQGicQCwsxygfuAi4F6YJGZzU0FQOhBd/9RuP5VwD8BM81sCnAtMBUYAzxrZpPdvSWperMm1XW06ilY/ztoOQAFZUGX0eTwrKNhI7NdpYgMQkkeQcwA1rj7OgAzexiYBaQDwt13RdYvAlKXA88CHnb3A8B6M1sTvl78DHf9SZddR59T15GI9BlJBkQVsCnyvB74cMeVzOxW4HZgCHBBZNuFHbatitl2NjAbYNy4cT1SdCIO7oG1C4JTUVc9DXu2hF1HZ8LF3wquT1DXkYj0MVkfpHb3+4D7zOx64BvATd3Ydg4wB6C2trabkxElbGd9eJQwH9a/pK4jEel3kgyIBmBs5Hl12NaZh4EffsBts6+1FTb/se2CtffC+z+MnABnfBZOmBkMNqvrSET6iSQDYhEwycxqCH65XwtcH13BzCa5++rw6RVA6vFc4EEz+yeCQepJwGsJ1vrBxHYd5QRBcPG32s46GsATBYrIwJVYQLh7s5ndBswnOM31AXdfbmZ3AXXuPhe4zcwuAg4B2wm7l8L1HiEY0G4Gbu0zZzDtrA8vWHuqfdfRxAvDuY4uUteRiAwI1u37CPRRtbW1XldX1/MvHO06WvUUvBvpOpp8mbqORKRfM7PF7l4btyzrg9R90sE9sO6FYJB59dPQ9F6k6+iucK4jdR2JyMCmgEjZ2dA2wKyuIxERBQQ7NsHD17XvOjrjszD50uDGOuo6EpFBSgFRMhqKR8HFn1TXkYhIhAIiNw9ufCzbVYiI9Dk52S5ARET6JgWEiIjEUkCIiEgsBYSIiMRSQIiISCwFhIiIxFJAiIhILAWEiIjEUkCIiEgsBYSIiMRSQIiISCwFhIiIxFJAiIhILAWEiIjESjQgzGymma00szVmdkfM8tvNbIWZvWFmz5nZcZFl3zGz5Wb2ppl9z0w3aRAR6U2JBYSZ5QL3AZcBU4DrzGxKh9X+CNS6+ynAY8B3wm3PBs4BTgGmAWcA5yVVq4iIHC7JI4gZwBp3X+fuB4GHgVnRFdx9gbvvDZ8uBKpTi4BCYAhQAOQD7yVYq4iIdJBkQFQBmyLP68O2ztwCzANw91eABcA74dd8d3+z4wZmNtvM6sysrrGxsccKFxGRPjJIbWY3ArXAveHzicBJBEcUVcAFZnZux+3cfY6717p7bWVlZW+WLCIy4CUZEA3A2Mjz6rCtHTO7CLgTuMrdD4TNfw4sdPcmd28iOLI4K8FaRUSkgyQDYhEwycxqzGwIcC0wN7qCmU0H7icIhy2RRW8D55lZnpnlEwxQH9bFJCIiyUksINy9GbgNmE/wy/0Rd19uZneZ2VXhavcCxcCjZrbEzFIB8hiwFlgK/An4k7v/OqlaRUTkcObu2a6hR9TW1npdXV22yxAR6VfMbLG718Yt6xOD1CIi0vcoIEREJJYCQkREYikgREQklgJCRERiKSBERCSWAkJERGIpIEREJJYCQkREYikgREQklgJCRERiKSBERCSWAkJERGIpIEREJJYCQkREYikgREQklgJCRERiKSBERCRWogFhZjPNbKWZrTGzO2KW325mK8zsDTN7zsyOiywbZ2ZPm9mb4Trjk6xVRETaSywgzCwXuA+4DJgCXGdmUzqs9keg1t1PAR4DvhNZ9jPgXnc/CZgBbEmqVhEROVySRxAzgDXuvs7dDwIPA7OiK7j7AnffGz5dCFQDhEGS5+7PhOs1RdYTEZFekGRAVAGbIs/rw7bO3ALMCx9PBnaY2S/N7I9mdm94RCIiIr0ko4Awsy+bWakF/s3MXjezS3qqCDO7EagF7g2b8oBzga8CZwATgJtjtpttZnVmVtfY2NhT5YiICJkfQXzG3XcBlwAjgE8D9xxhmwZgbOR5ddjWjpldBNwJXOXuB8LmemBJ2D3VDPwKOK3jtu4+x91r3b22srIyw29FREQykWlAWPjv5cB/uPvySFtnFgGTzKzGzIYA1wJz272o2XTgfoJw2NJh2+FmlvqtfwGwIsNaRUSkB2QaEIvN7GmCgJhvZiVAa1cbhH/53wbMB94EHnH35WZ2l5ldFa52L1AMPGpmS8xsbrhtC0H30nNmtpQgjH7cze9NRESOgrn7kVcyywFOBda5+w4zGwlUu/sbSReYqdraWq+rq8t2GSIi/YqZLXb32rhlmR5BnAWsDMPhRuAbwM6eKlBERPqeTAPih8BeM/sQ8DfAWoIL2UREZIDKNCCaPeiLmgV8393vA0qSK0tERLItL8P1dpvZ1wlObz03HJPIT64sERHJtkyPIK4BDhBcD/EuwTUN93a9iYiI9GcZBUQYCj8Hyszsz4D97q4xCBGRASzTqTY+BbwGfBL4FPCqmX0iycJERCS7Mh2DuBM4I3W1c3iF87MEU3SLiMgAlOkYRE6HqTC2dWNbERHphzI9gnjKzOYDD4XPrwGeTKYkERHpCzIKCHf/mpldDZwTNs1x9yeSK0tERLIt0yMI3P1x4PEEaxERkT6ky4Aws91A3Gx+Bri7lyZSlYiIZF2XAeHumk5DRGSQ0plIIiISSwEhIiKxFBAiIhJLASEiIrEUECIiEivRgDCzmWa20szWmNkdMctvN7MVZvaGmT1nZsd1WF5qZvVm9v0k6xQRkcMlFhBmlgvcB1wGTAGuM7MpHVb7I1Dr7qcQTPz3nQ7LvwW8lFSNIiLSuSSPIGYAa9x9nbsfBB4muGVpmrsvcPe94dOFBDciAsDMTgeOBZ5OsEYREelEkgFRBWyKPK8P2zpzCzAPILyl6f8FvtrVG5jZbDOrM7O6xsbGoyxXRESi+sQgtZndCNTSdhvTLwJPunt9V9u5+xx3r3X32srKyqTLFBEZVDKerO8DaADGRp5Xh23tmNlFBDckOs/dD4TNZwHnmtkXgWJgiJk1ufthA90iIpKMJANiETDJzGoIguFa4ProCmY2HbgfmBm9IZG73xBZ52aCgWyFg4hIL0qsi8ndm4HbgPnAm8Aj7r7czO4ys6vC1e4lOEJ41MyWmNncpOoREZHuMfe42bz7n9raWq+rq8t2GSIi/YqZLXb32rhlfWKQWkRE+h4FhIiIxFJAiIhILAWEiIjEUkCIiEgsBYSIiMRSQIiISCwFhIiIxFJAiIhILAWEiIjEUkCIiEgsBYSIiMRSQIiISCwFhIiIxFJAiIhILAWEiIjEUkCIiEgsBQSwY+/BbJcgItLn5GW7gGzbue8Qp971DGPKCplWVcbJVWVMC78qSwqyXZ6ISNYkGhBmNhP4LpAL/MTd7+mw/Hbgs0Az0Ah8xt03mtmpwA+BUqAF+La7/yKpOv/28hNZ2rCLZQ07eXrFe+n2UaWFYViUcnIYHseUFiZVhohIn2LunswLm+UCq4CLgXpgEXCdu6+IrPMx4FV332tmXwDOd/drzGwy4O6+2szGAIuBk9x9R2fvV1tb63V1dUdd9+79h1i+OQiLZQ07Wdqwk3Vb95DaTZUlBemjjJPD8BhVWoiZHfV7i4j0NjNb7O61ccuSPIKYAaxx93VhEQ8Ds4B0QLj7gsj6C4Ebw/ZVkXU2m9kWoBLoNCB6SklhPmdOKOfMCeXptqYDzbz5zi6W1reFxgsrt9AahkZF8ZDgSGNMGBzVZYwpU2iISP+WZEBUAZsiz+uBD3ex/i3AvI6NZjYDGAKsjVk2G5gNMG7cuKOptUvFBXmcMX4kZ4wfmW7bezASGuERx+9Wb6UlTI2RRUOYOqata2paVRnVI4YqNESk3+gTg9RmdiNQC5zXoX008B/ATe7e2nE7d58DzIGgi6kXSk0bNiSP048byenHtYXG/kMtvPnOrvRRxtKGXcx5aR3NYWgMH5bPtDFlTI2MaYwbOUyhISJ9UpIB0QCMjTyvDtvaMbOLgDuB89z9QKS9FPgtcKe7L0ywzh5TmJ/L9HEjmD5uRLpt/6EWVr67m6UNO1m+OQiOB36/nkMtQWiUFOYxbUzQLRV0U5UyvryInByFhohkV5IBsQiYZGY1BMFwLXB9dAUzmw7cD8x09y2R9iHAE8DP3P2xBGtMXGF+Lh8aO5wPjR2ebjvQ3MKqd5tYFgbGsoad/PsfNnCwJThIKinIY8qY0nan3U6oUGiISO9KLCDcvdnMbgPmE5zm+oC7Lzezu4A6d58L3AsUA4+G3Sxvu/tVwKeAjwLlZnZz+JI3u/uSpOrtTQV5uZxcHRw1XBe2HWppZdV7u8Ozp3axtGEn/7lwIweag9AoGpJ7WGgcX1lMrkJDRBKS2Gmuva2nTnPtSw61tLJmS1O7U25XvLOL/YeC0BiaH4TGyVVlwYB4dRkTK4vJy9UF8iKSmWyd5ipHKT83h5NGl3LS6FI+WRsM57S0Omsbm8Kzp4LgeKRuE3sPtgBQkBdskxoEn1pVyuRjS8hXaIhIN+kIYgBoaXXWb90TOXtqJys276LpQDMAQ/JyOGlUSXoKkZOryph8bAlD8hQaIoNdV0cQCogBqrXV2bBtT3j21K70Ecfu/UFo5OcaJ4wqaZt7akwZJ4wqoTA/N8uVi0hvUkAIEITGpu1700cZqQHxnfsOAZCXY0w+tiQ9hci0qjJOGl2q0BAZwBQQ0il3p377vnRgpP7dvjcIjdwcY9Ixxe3OnpoyupShQxQaIgOBAkK6xd1p2LGPZQ272oXGtj3BfTNyDCaGoZG6yG/K6FKKCnTOg0h/o7OYpFvMjOoRw6geMYyZ00YBQWi8u2v/YXNP/fL1hnAbOL6ymGmRazWmjCmlpDA/m9+KiBwFBYRkxMwYXTaU0WVDuWTqqHT7e7v2tzvKWLjufX61ZHN6+YSKovQ9NVJnUZUqNET6BQWEHJVjSws5trSQC086Nt3WuPtAu4v76ja8z9w/tYXG+PJhTA2PMlIX+Q0fNiQb5YtIFxQQ0uMqSwr42InH8LETj0m3bWs6kO6aWlq/kz9t2sFv33gnvXzsyKFhWLQFx4gihYZINikgpFeUFxdw3uRKzptcmW7bvudgeDV422D4k0vfTS+vGj40fbvXVPdURbHuEy7SWxQQkjUjioZw7qRKzp3UFho79x5KT4ueushv/vK2+4SPLitsd7vXaVVlHFOi+4SLJEEBIX1K2bB8zp5YwdkTK9Jtu/YfYnnDrnbB8eyb76XvE35saUHb7V7DI41jSwt0IyaRo6SAkD6vtDCfs44v56zj298nfMXmXZErwneyoN19wgs4OeyeSg2Ij9Z9wkW6RQEh/VJxQR4zakYyo6b9fcJXpAbCw3GNF1c1pkOjvGhIGBal6QFx3SdcpHMKCBkwhg3Jo3b8SGrHt4XGvoMtvPlu29lTyzbv4v4X2+4TPmJYfnoAfFp4BtXYkQoNEVBAyAA3dEgup40bwWkd7hP+1ru7212r8ZPfrUvfJ7y0MK/deMbJVWWMGzlMt3yVQUcBIYNOYX4up44dzqkx9wlvO3tqJ/8vep/wwrzgrn2RU25rynWfcBnYFBAitL9PeMrB5uA+4W1nT+3ip69s5GB4n/DigrzgPuFjyji5OgiPmgrdJ1wGjkRnczWzmcB3gVzgJ+5+T4fltwOfBZqBRuAz7r4xXHYT8I1w1bvd/addvZdmc5XekLpPePTsqeh9wocNyWXK6NJ2XVTHVxbpPuHSZ2Vlum8zywVWARcD9cAi4Dp3XxFZ52PAq+6+18y+AJzv7teY2UigDqgFHFgMnO7u2zt7PwWEZEtzSytrG/e0m7Rw+eZd7DsU3Ce8MD+HiccUM6GimAmVRUyoLGZCRRE1FUWaIl2yLlvTfc8A1rj7urCIh4FZQDog3H1BZP2FwI3h40uBZ9z9/XDbZ4CZwEMJ1ivygeTl5nDCqBJOGFXC1adXA6n7hIdjGvW7WNPYxOtvb+fXb2wm+jfZqNLCMDSKqAkD5PiKYqpGDFVXlWRdkgFRBWyKPK8HPtzF+rcA87rYtqrjBmY2G5gNMG7cuKOpVaRH5eYYE48pYeIxJfz59Lb2/Yda2LhtL+sam1i3dQ9rG5tY17iHuUs2syu8XzjAkNwcjisf1u6II/WvJjGU3tInjm/N7EaC7qTzurOdu88B5kDQxZRAaSI9qjA/N320EeXuvL/nIOu27kmHx7rGPazZ0sTzb21Jn4ILwbUb0dCoqSji+MoixpUPoyBPt4KVnpNkQDQAYyPPq8O2dszsIuBO4Dx3PxDZ9vwO276QSJUifYCZUV5cQHlxAWdELvSDYIxj0/Z9rN8aHG2sbQxC5MVVjTy6uD69Xo7B2JHDqKkoiox3FHF8ZTHHlGhuKum+JAep8wgGqS8k+IW/CLje3ZdH1pkOPAbMdPfVkfaRBAPTp4VNrxMMUr/f2ftpkFoGo937D7E+PNqIHnms37onPUgOUDQkl5rKtuAIjjqKNVAu2RmkdvdmM7sNmE9wmusD7r7czO4C6tx9LnAvUAw8Gv5187a7X+Xu75vZtwhCBeCursJBZLAqKcznlOrhnFI9vF17a2twD/EgLJqCo46te7ocKK9JjXNooFxCiV4H0Zt0BCGSmbiB8tRRyM59h9LrxQ+UB0chGigfOPS37SgAAApGSURBVLJ1mquI9EE9PVCe6rrSQPnAo4AQEeDIA+X12/exLoOB8uoRw9JHGqmB8gkVxbqJUz+kgBCRI8rLzWF8RRHjK4q44MT2y9oNlKeOPhr38Oq69zsdKK+paDvDSgPlfZf+V0TkqBxpoHx9GBqpgfI/boofKE+FRmqgfEJFEdUjhmmgPIsUECKSiJwcY8zwoYwZPpRzIvcYh8MHyoOjjyZ+88Y7XQ6Upy4K1EB571BAiEivO9JAearLam045tHVQHlN5OwqDZT3LAWEiPQZ0YHy2iMMlKfGO15a1chjRxooD6/x0EB59yggRKRfyGSgfP3WtjOsNFB+9LRHRKTf62yg3L3tivLUQPn6TgbKjy0taH/PDg2UKyBEZOAyM0aXDWV0WecD5empSDIYKE/fs2OQDJQrIERkUOpqoHz73kPpbqq1kYsD4wbKo3NYpY5AjhsgA+UKCBGRCDNjZNEQRhaN7LGB8prKIo7vhwPlCggRkQwdaaB8w9a9rEt3WXU9UF5T0TYBYl8dKO9b1YiI9FMlhfmcXF3GydVl7do7DpSnLgxcsmk7vznSQHkYINkaKFdAiIgkqD8PlCsgRESypLOBciCYej0yUL6+k4Hy4cPyOXdSJf963fQer08BISLSBx1poHx9eLOndVv3MHxofiI1KCBERPqR6ED5x048JtH3ykn01UVEpN9KNCDMbKaZrTSzNWZ2R8zyj5rZ62bWbGaf6LDsO2a23MzeNLPvWX85cVhEZIBILCDMLBe4D7gMmAJcZ2ZTOqz2NnAz8GCHbc8GzgFOAaYBZwDnJVWriIgcLskxiBnAGndfB2BmDwOzgBWpFdx9Q7istcO2DhQCQwAD8oH3EqxVREQ6SLKLqQrYFHleH7Ydkbu/AiwA3gm/5rv7mx3XM7PZZlZnZnWNjY09ULKIiKT0yUFqM5sInARUE4TKBWZ2bsf13H2Ou9e6e21lZWVvlykiMqAlGRANwNjI8+qwLRN/Dix09yZ3bwLmAWf1cH0iItKFJANiETDJzGrMbAhwLTA3w23fBs4zszwzyycYoD6si0lERJJjHp0pqqdf3Oxy4F+AXOABd/+2md0F1Ln7XDM7A3gCGAHsB95196nhGVA/AD5KMGD9lLvffoT3agQ2HkW5FcDWo9g+Kaqre1RX96iu7hmIdR3n7rF99IkGRH9iZnXuXpvtOjpSXd2jurpHdXXPYKurTw5Si4hI9ikgREQklgKizZxsF9AJ1dU9qqt7VFf3DKq6NAYhIiKxdAQhIiKxFBAiIhJrwAdEBlOOF5jZL8Llr5rZ+Miyr4ftK83s0l6u63YzW2Fmb5jZc2Z2XGRZi5ktCb8yvfiwp+q62cwaI+//2ciym8xsdfh1Uy/X9c+RmlaZ2Y7IsiT31wNmtsXMlnWy3MLp6teE/5enRZYlub+OVNcNYT1LzexlM/tQZNmGsH2JmdX1cl3nm9nOyP/X30eWdfkZSLiur0VqWhZ+pkaGy5LcX2PNbEH4u2C5mX05Zp3kPmPuPmC/CC7QWwtMIJgZ9k/AlA7rfBH4Ufj4WuAX4eMp4foFQE34Orm9WNfHgGHh4y+k6gqfN2Vxf90MfD9m25HAuvDfEeHjEb1VV4f1/zvBhZmJ7q/wtT8KnAYs62T55QRTxRhwJvBq0vsrw7rOTr0fwZT8r0aWbQAqsrS/zgd+c7SfgZ6uq8O6VwLP99L+Gg2cFj4uAVbF/Ewm9hkb6EcQ6SnH3f0gkJpyPGoW8NPw8WPAhWZmYfvD7n7A3dcDa8LX65W63H2Bu+8Nny4kmMsqaZnsr85cCjzj7u+7+3bgGWBmluq6Dnioh967S+7+EvB+F6vMAn7mgYXAcDMbTbL764h1ufvL4ftC732+MtlfnTmaz2ZP19Wbn6933P318PFugimHOs6KndhnbKAHRCZTjqfXcfdmYCdQnuG2SdYVdQvBXwgphRZMc77QzD7eQzV1p66rw0PZx8wsNSFjn9hfYVdcDfB8pDmp/ZWJzmpPcn91V8fPlwNPm9liM5udhXrOMrM/mdk8M5satvWJ/WVmwwh+yT4eae6V/WVB9/d04NUOixL7jCV5wyDpAWZ2I1BL+zvqHefuDWY2AXjezJa6+9peKunXwEPufsDM/org6OuCXnrvTFwLPObuLZG2bO6vPs3MPkYQEB+JNH8k3F/HAM+Y2VvhX9i94XWC/68mC+Zy+xUwqZfeOxNXAn9w9+jRRuL7y8yKCULpK+6+qydfuysD/QgikynH0+uYWR5QBmzLcNsk68LMLgLuBK5y9wOpdndvCP9dB7xA8FdFr9Tl7tsitfwEOD3TbZOsK+JaOhz+J7i/MtFZ7Unur4yY2SkE/4ez3H1bqj2yv7YQTKbZU12rR+TuuzyY4h93fxLIN7MK+sD+CnX1+Upkf1kwo/XjwM/d/ZcxqyT3GUtiYKWvfBEcIa0j6HJIDWxN7bDOrbQfpH4kfDyV9oPU6+i5QepM6ppOMCg3qUP7CKAgfFwBrKaHBusyrGt05HHqvh0QDIStD+sbET4e2Vt1heudSDBgaL2xvyLvMZ7OB12voP0A4mtJ768M6xpHMK52dof2IqAk8vhlYGYv1jUq9f9H8Iv27XDfZfQZSKqucHkZwThFUW/tr/B7/xnwL12sk9hnrMd2bl/9IhjhX0Xwy/bOsO0ugr/KIbj39aPhD8trwITItneG260ELuvlup4luA/3kvBrbth+NrA0/AFZCtzSy3X9I7A8fP8FwImRbT8T7sc1wF/2Zl3h828C93TYLun99RDBbXEPEfTx3gJ8Hvh8uNyA+8K6lwK1vbS/jlTXT4Dtkc9XXdg+IdxXfwr/n+/s5bpui3y+FhIJsLjPQG/VFa5zM8GJK9Htkt5fHyEY43gj8n91eW99xjTVhoiIxBroYxAiIvIBKSBERCSWAkJERGIpIEREJJYCQkREYikgRPqAcBbT32S7DpEoBYSIiMRSQIh0g5ndaGavhXP/329muWbWFN6PYrkF9+6oDNc9NZwg8A0ze8LMRoTtE83s2XBCutfN7Pjw5YvDCRDfMrOfh7MKi2SNAkIkQ2Z2EnANcI67nwq0ADcQTLFQ5+5TgReB/xlu8jPgf7j7KQRXuKbafw7c5+4fIrjS+52wfTrwFYJ7kUwAzkn8mxLpgmZzFcnchQSTEy4K/7gfCmwBWoFfhOv8J/BLMysDhrv7i2H7T4FHzawEqHL3JwDcfT9A+HqvuXt9+HwJwdxAv0/+2xKJp4AQyZwBP3X3r7drNPu7Dut90PlrDkQet6CfT8kydTGJZO454BPhvP+Y2cjwBkU5wCfCda4Hfu/uO4HtZnZu2P5p4EUP7gpWn7pxkQX3RB/Wq9+FSIb0F4pIhtx9hZl9g+DuYTkEM3/eCuwBZoTLthCMUwDcBPwoDIB1wF+G7Z8G7jezu8LX+GQvfhsiGdNsriJHycya3L0423WI9DR1MYmISCwdQYiISCwdQYiISCwFhIiIxFJAiIhILAWEiIjEUkCIiEis/w/tQVCmB3iRdQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49Rf_xd5MpjX"
      },
      "source": [
        "#**6-Textual Data Preprocessing** [6]\n",
        "Congratulations for going far in this Keras tutorial. After working with numeric, categorical and image data, it is time to handle textual data. Deep learning has proved itself in being very effective with Natural Language Processing (NLP) tasks. Before we dive into how to build NLP models, let us learn how to handle and preprocess textual data with Keras.\n",
        "\n",
        "We already know that any machine learning model needs data to represented in a numeric format. Therefore, in this article we will learn how to load, preprocess, tokenize, convert and pad textual data into numeric sequences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rEDMVWWNXxlf"
      },
      "source": [
        "##**6.1 Dataset**\n",
        "\n",
        "In the following coding exercises we will be playing with the Spam SMS dataset. The dataset is stored in a CSV file that contains two fields: Label and SMS. The SMS is the message text, while the label indicates whether the SMS is a spam or not. Kindly download from here, and load it as shown in the following code snippet:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SW7Aa7zypOSe",
        "colab": {
          "resources": {
            "http://localhost:8080/nbextensions/google.colab/files.js": {
              "data": "Ly8gQ29weXJpZ2h0IDIwMTcgR29vZ2xlIExMQwovLwovLyBMaWNlbnNlZCB1bmRlciB0aGUgQXBhY2hlIExpY2Vuc2UsIFZlcnNpb24gMi4wICh0aGUgIkxpY2Vuc2UiKTsKLy8geW91IG1heSBub3QgdXNlIHRoaXMgZmlsZSBleGNlcHQgaW4gY29tcGxpYW5jZSB3aXRoIHRoZSBMaWNlbnNlLgovLyBZb3UgbWF5IG9idGFpbiBhIGNvcHkgb2YgdGhlIExpY2Vuc2UgYXQKLy8KLy8gICAgICBodHRwOi8vd3d3LmFwYWNoZS5vcmcvbGljZW5zZXMvTElDRU5TRS0yLjAKLy8KLy8gVW5sZXNzIHJlcXVpcmVkIGJ5IGFwcGxpY2FibGUgbGF3IG9yIGFncmVlZCB0byBpbiB3cml0aW5nLCBzb2Z0d2FyZQovLyBkaXN0cmlidXRlZCB1bmRlciB0aGUgTGljZW5zZSBpcyBkaXN0cmlidXRlZCBvbiBhbiAiQVMgSVMiIEJBU0lTLAovLyBXSVRIT1VUIFdBUlJBTlRJRVMgT1IgQ09ORElUSU9OUyBPRiBBTlkgS0lORCwgZWl0aGVyIGV4cHJlc3Mgb3IgaW1wbGllZC4KLy8gU2VlIHRoZSBMaWNlbnNlIGZvciB0aGUgc3BlY2lmaWMgbGFuZ3VhZ2UgZ292ZXJuaW5nIHBlcm1pc3Npb25zIGFuZAovLyBsaW1pdGF0aW9ucyB1bmRlciB0aGUgTGljZW5zZS4KCi8qKgogKiBAZmlsZW92ZXJ2aWV3IEhlbHBlcnMgZm9yIGdvb2dsZS5jb2xhYiBQeXRob24gbW9kdWxlLgogKi8KKGZ1bmN0aW9uKHNjb3BlKSB7CmZ1bmN0aW9uIHNwYW4odGV4dCwgc3R5bGVBdHRyaWJ1dGVzID0ge30pIHsKICBjb25zdCBlbGVtZW50ID0gZG9jdW1lbnQuY3JlYXRlRWxlbWVudCgnc3BhbicpOwogIGVsZW1lbnQudGV4dENvbnRlbnQgPSB0ZXh0OwogIGZvciAoY29uc3Qga2V5IG9mIE9iamVjdC5rZXlzKHN0eWxlQXR0cmlidXRlcykpIHsKICAgIGVsZW1lbnQuc3R5bGVba2V5XSA9IHN0eWxlQXR0cmlidXRlc1trZXldOwogIH0KICByZXR1cm4gZWxlbWVudDsKfQoKLy8gTWF4IG51bWJlciBvZiBieXRlcyB3aGljaCB3aWxsIGJlIHVwbG9hZGVkIGF0IGEgdGltZS4KY29uc3QgTUFYX1BBWUxPQURfU0laRSA9IDEwMCAqIDEwMjQ7CgpmdW5jdGlvbiBfdXBsb2FkRmlsZXMoaW5wdXRJZCwgb3V0cHV0SWQpIHsKICBjb25zdCBzdGVwcyA9IHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCk7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICAvLyBDYWNoZSBzdGVwcyBvbiB0aGUgb3V0cHV0RWxlbWVudCB0byBtYWtlIGl0IGF2YWlsYWJsZSBmb3IgdGhlIG5leHQgY2FsbAogIC8vIHRvIHVwbG9hZEZpbGVzQ29udGludWUgZnJvbSBQeXRob24uCiAgb3V0cHV0RWxlbWVudC5zdGVwcyA9IHN0ZXBzOwoKICByZXR1cm4gX3VwbG9hZEZpbGVzQ29udGludWUob3V0cHV0SWQpOwp9CgovLyBUaGlzIGlzIHJvdWdobHkgYW4gYXN5bmMgZ2VuZXJhdG9yIChub3Qgc3VwcG9ydGVkIGluIHRoZSBicm93c2VyIHlldCksCi8vIHdoZXJlIHRoZXJlIGFyZSBtdWx0aXBsZSBhc3luY2hyb25vdXMgc3RlcHMgYW5kIHRoZSBQeXRob24gc2lkZSBpcyBnb2luZwovLyB0byBwb2xsIGZvciBjb21wbGV0aW9uIG9mIGVhY2ggc3RlcC4KLy8gVGhpcyB1c2VzIGEgUHJvbWlzZSB0byBibG9jayB0aGUgcHl0aG9uIHNpZGUgb24gY29tcGxldGlvbiBvZiBlYWNoIHN0ZXAsCi8vIHRoZW4gcGFzc2VzIHRoZSByZXN1bHQgb2YgdGhlIHByZXZpb3VzIHN0ZXAgYXMgdGhlIGlucHV0IHRvIHRoZSBuZXh0IHN0ZXAuCmZ1bmN0aW9uIF91cGxvYWRGaWxlc0NvbnRpbnVlKG91dHB1dElkKSB7CiAgY29uc3Qgb3V0cHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKG91dHB1dElkKTsKICBjb25zdCBzdGVwcyA9IG91dHB1dEVsZW1lbnQuc3RlcHM7CgogIGNvbnN0IG5leHQgPSBzdGVwcy5uZXh0KG91dHB1dEVsZW1lbnQubGFzdFByb21pc2VWYWx1ZSk7CiAgcmV0dXJuIFByb21pc2UucmVzb2x2ZShuZXh0LnZhbHVlLnByb21pc2UpLnRoZW4oKHZhbHVlKSA9PiB7CiAgICAvLyBDYWNoZSB0aGUgbGFzdCBwcm9taXNlIHZhbHVlIHRvIG1ha2UgaXQgYXZhaWxhYmxlIHRvIHRoZSBuZXh0CiAgICAvLyBzdGVwIG9mIHRoZSBnZW5lcmF0b3IuCiAgICBvdXRwdXRFbGVtZW50Lmxhc3RQcm9taXNlVmFsdWUgPSB2YWx1ZTsKICAgIHJldHVybiBuZXh0LnZhbHVlLnJlc3BvbnNlOwogIH0pOwp9CgovKioKICogR2VuZXJhdG9yIGZ1bmN0aW9uIHdoaWNoIGlzIGNhbGxlZCBiZXR3ZWVuIGVhY2ggYXN5bmMgc3RlcCBvZiB0aGUgdXBsb2FkCiAqIHByb2Nlc3MuCiAqIEBwYXJhbSB7c3RyaW5nfSBpbnB1dElkIEVsZW1lbnQgSUQgb2YgdGhlIGlucHV0IGZpbGUgcGlja2VyIGVsZW1lbnQuCiAqIEBwYXJhbSB7c3RyaW5nfSBvdXRwdXRJZCBFbGVtZW50IElEIG9mIHRoZSBvdXRwdXQgZGlzcGxheS4KICogQHJldHVybiB7IUl0ZXJhYmxlPCFPYmplY3Q+fSBJdGVyYWJsZSBvZiBuZXh0IHN0ZXBzLgogKi8KZnVuY3Rpb24qIHVwbG9hZEZpbGVzU3RlcChpbnB1dElkLCBvdXRwdXRJZCkgewogIGNvbnN0IGlucHV0RWxlbWVudCA9IGRvY3VtZW50LmdldEVsZW1lbnRCeUlkKGlucHV0SWQpOwogIGlucHV0RWxlbWVudC5kaXNhYmxlZCA9IGZhbHNlOwoKICBjb25zdCBvdXRwdXRFbGVtZW50ID0gZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQob3V0cHV0SWQpOwogIG91dHB1dEVsZW1lbnQuaW5uZXJIVE1MID0gJyc7CgogIGNvbnN0IHBpY2tlZFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgaW5wdXRFbGVtZW50LmFkZEV2ZW50TGlzdGVuZXIoJ2NoYW5nZScsIChlKSA9PiB7CiAgICAgIHJlc29sdmUoZS50YXJnZXQuZmlsZXMpOwogICAgfSk7CiAgfSk7CgogIGNvbnN0IGNhbmNlbCA9IGRvY3VtZW50LmNyZWF0ZUVsZW1lbnQoJ2J1dHRvbicpOwogIGlucHV0RWxlbWVudC5wYXJlbnRFbGVtZW50LmFwcGVuZENoaWxkKGNhbmNlbCk7CiAgY2FuY2VsLnRleHRDb250ZW50ID0gJ0NhbmNlbCB1cGxvYWQnOwogIGNvbnN0IGNhbmNlbFByb21pc2UgPSBuZXcgUHJvbWlzZSgocmVzb2x2ZSkgPT4gewogICAgY2FuY2VsLm9uY2xpY2sgPSAoKSA9PiB7CiAgICAgIHJlc29sdmUobnVsbCk7CiAgICB9OwogIH0pOwoKICAvLyBXYWl0IGZvciB0aGUgdXNlciB0byBwaWNrIHRoZSBmaWxlcy4KICBjb25zdCBmaWxlcyA9IHlpZWxkIHsKICAgIHByb21pc2U6IFByb21pc2UucmFjZShbcGlja2VkUHJvbWlzZSwgY2FuY2VsUHJvbWlzZV0pLAogICAgcmVzcG9uc2U6IHsKICAgICAgYWN0aW9uOiAnc3RhcnRpbmcnLAogICAgfQogIH07CgogIGNhbmNlbC5yZW1vdmUoKTsKCiAgLy8gRGlzYWJsZSB0aGUgaW5wdXQgZWxlbWVudCBzaW5jZSBmdXJ0aGVyIHBpY2tzIGFyZSBub3QgYWxsb3dlZC4KICBpbnB1dEVsZW1lbnQuZGlzYWJsZWQgPSB0cnVlOwoKICBpZiAoIWZpbGVzKSB7CiAgICByZXR1cm4gewogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbXBsZXRlJywKICAgICAgfQogICAgfTsKICB9CgogIGZvciAoY29uc3QgZmlsZSBvZiBmaWxlcykgewogICAgY29uc3QgbGkgPSBkb2N1bWVudC5jcmVhdGVFbGVtZW50KCdsaScpOwogICAgbGkuYXBwZW5kKHNwYW4oZmlsZS5uYW1lLCB7Zm9udFdlaWdodDogJ2JvbGQnfSkpOwogICAgbGkuYXBwZW5kKHNwYW4oCiAgICAgICAgYCgke2ZpbGUudHlwZSB8fCAnbi9hJ30pIC0gJHtmaWxlLnNpemV9IGJ5dGVzLCBgICsKICAgICAgICBgbGFzdCBtb2RpZmllZDogJHsKICAgICAgICAgICAgZmlsZS5sYXN0TW9kaWZpZWREYXRlID8gZmlsZS5sYXN0TW9kaWZpZWREYXRlLnRvTG9jYWxlRGF0ZVN0cmluZygpIDoKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJ24vYSd9IC0gYCkpOwogICAgY29uc3QgcGVyY2VudCA9IHNwYW4oJzAlIGRvbmUnKTsKICAgIGxpLmFwcGVuZENoaWxkKHBlcmNlbnQpOwoKICAgIG91dHB1dEVsZW1lbnQuYXBwZW5kQ2hpbGQobGkpOwoKICAgIGNvbnN0IGZpbGVEYXRhUHJvbWlzZSA9IG5ldyBQcm9taXNlKChyZXNvbHZlKSA9PiB7CiAgICAgIGNvbnN0IHJlYWRlciA9IG5ldyBGaWxlUmVhZGVyKCk7CiAgICAgIHJlYWRlci5vbmxvYWQgPSAoZSkgPT4gewogICAgICAgIHJlc29sdmUoZS50YXJnZXQucmVzdWx0KTsKICAgICAgfTsKICAgICAgcmVhZGVyLnJlYWRBc0FycmF5QnVmZmVyKGZpbGUpOwogICAgfSk7CiAgICAvLyBXYWl0IGZvciB0aGUgZGF0YSB0byBiZSByZWFkeS4KICAgIGxldCBmaWxlRGF0YSA9IHlpZWxkIHsKICAgICAgcHJvbWlzZTogZmlsZURhdGFQcm9taXNlLAogICAgICByZXNwb25zZTogewogICAgICAgIGFjdGlvbjogJ2NvbnRpbnVlJywKICAgICAgfQogICAgfTsKCiAgICAvLyBVc2UgYSBjaHVua2VkIHNlbmRpbmcgdG8gYXZvaWQgbWVzc2FnZSBzaXplIGxpbWl0cy4gU2VlIGIvNjIxMTU2NjAuCiAgICBsZXQgcG9zaXRpb24gPSAwOwogICAgd2hpbGUgKHBvc2l0aW9uIDwgZmlsZURhdGEuYnl0ZUxlbmd0aCkgewogICAgICBjb25zdCBsZW5ndGggPSBNYXRoLm1pbihmaWxlRGF0YS5ieXRlTGVuZ3RoIC0gcG9zaXRpb24sIE1BWF9QQVlMT0FEX1NJWkUpOwogICAgICBjb25zdCBjaHVuayA9IG5ldyBVaW50OEFycmF5KGZpbGVEYXRhLCBwb3NpdGlvbiwgbGVuZ3RoKTsKICAgICAgcG9zaXRpb24gKz0gbGVuZ3RoOwoKICAgICAgY29uc3QgYmFzZTY0ID0gYnRvYShTdHJpbmcuZnJvbUNoYXJDb2RlLmFwcGx5KG51bGwsIGNodW5rKSk7CiAgICAgIHlpZWxkIHsKICAgICAgICByZXNwb25zZTogewogICAgICAgICAgYWN0aW9uOiAnYXBwZW5kJywKICAgICAgICAgIGZpbGU6IGZpbGUubmFtZSwKICAgICAgICAgIGRhdGE6IGJhc2U2NCwKICAgICAgICB9LAogICAgICB9OwogICAgICBwZXJjZW50LnRleHRDb250ZW50ID0KICAgICAgICAgIGAke01hdGgucm91bmQoKHBvc2l0aW9uIC8gZmlsZURhdGEuYnl0ZUxlbmd0aCkgKiAxMDApfSUgZG9uZWA7CiAgICB9CiAgfQoKICAvLyBBbGwgZG9uZS4KICB5aWVsZCB7CiAgICByZXNwb25zZTogewogICAgICBhY3Rpb246ICdjb21wbGV0ZScsCiAgICB9CiAgfTsKfQoKc2NvcGUuZ29vZ2xlID0gc2NvcGUuZ29vZ2xlIHx8IHt9OwpzY29wZS5nb29nbGUuY29sYWIgPSBzY29wZS5nb29nbGUuY29sYWIgfHwge307CnNjb3BlLmdvb2dsZS5jb2xhYi5fZmlsZXMgPSB7CiAgX3VwbG9hZEZpbGVzLAogIF91cGxvYWRGaWxlc0NvbnRpbnVlLAp9Owp9KShzZWxmKTsK",
              "ok": true,
              "headers": [
                [
                  "content-type",
                  "application/javascript"
                ]
              ],
              "status": 200,
              "status_text": ""
            }
          },
          "base_uri": "https://localhost:8080/",
          "height": 72
        },
        "outputId": "5ab65c8e-52ea-4cbd-afdd-b86a467d0a36"
      },
      "source": [
        "from google.colab import files\n",
        "uploaded = files.upload()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-f08dc4da-bf07-4e18-be18-3f41e3084577\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-f08dc4da-bf07-4e18-be18-3f41e3084577\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script src=\"/nbextensions/google.colab/files.js\"></script> "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Saving sms.csv to sms.csv\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZbBe053qYIB0"
      },
      "source": [
        "import pandas as pd\n",
        "data = pd.read_csv('sms.csv',delimiter=',',encoding='latin-1')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QrbpLk6lq9ae",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "e3fe6d07-5d66-4505-ce62-85bafdf45f78"
      },
      "source": [
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v1</th>\n",
              "      <th>v2</th>\n",
              "      <th>Unnamed: 2</th>\n",
              "      <th>Unnamed: 3</th>\n",
              "      <th>Unnamed: 4</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>ham</td>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>ham</td>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>spam</td>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>ham</td>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>ham</td>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     v1  ... Unnamed: 4\n",
              "0   ham  ...        NaN\n",
              "1   ham  ...        NaN\n",
              "2  spam  ...        NaN\n",
              "3   ham  ...        NaN\n",
              "4   ham  ...        NaN\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oSxTtEpZrMg8"
      },
      "source": [
        "#Drop the columns that are not required for the neural network.\n",
        "data.drop(['v1','Unnamed: 2', 'Unnamed: 3', 'Unnamed: 4'],axis=1,inplace=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "d9JuSz1Rrpq9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "10e9b907-5120-4a2f-f9c9-1886775a39a5"
      },
      "source": [
        "data.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>v2</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Ok lar... Joking wif u oni...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>U dun say so early hor... U c already then say...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  v2\n",
              "0  Go until jurong point, crazy.. Available only ...\n",
              "1                      Ok lar... Joking wif u oni...\n",
              "2  Free entry in 2 a wkly comp to win FA Cup fina...\n",
              "3  U dun say so early hor... U c already then say...\n",
              "4  Nah I don't think he goes to usf, he lives aro..."
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "08TehW6kHDAt"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=10KE2RVl5itEzmnP7bPDi-CmUF4EvgwVi)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s5NFKqJBiiWz"
      },
      "source": [
        "## **6.2 Tokenizing**\n",
        "Normally the first step in textual data preprocessing is splitting sentences into words/tokens. This could easily be done using the Keras tokenizer class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5swzcygiu9E"
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer\n",
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(data['SMS'])\n",
        "data['Tokens'] = tokenizer.texts_to_sequences(data['SMS'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pj1mDa_Vi10W"
      },
      "source": [
        "Let us check the results:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kH0PxIWwlBWR"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1cClKRjqV0-XUqqokNrAMPcgrdp3Wzhf3\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vAAIxBkWlZMi"
      },
      "source": [
        "See how words got replaced by numbers! Well each word now has an index that represents it in a sentence.\n",
        "\n",
        "In order to get what each index stands for, we can use the tokenizer index_word property:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y-L2mzDzlhpo"
      },
      "source": [
        "print(tokenizer.index_word[49])\n",
        "# output: go"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3aZTnO21li_J"
      },
      "source": [
        "print(tokenizer.word_index['go'])\n",
        "# output: 49"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEYobj2xlsS7"
      },
      "source": [
        "We have one more thing to take care of. See how each SMS has different number of words! This results in different sequence lengths; which may cause problems in some models. The solution is simple: Padding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Z-Iq0fQlzCp"
      },
      "source": [
        "## **6.3- Sequence Padding**\n",
        "\n",
        "Padding means adding zero values to the left or the right of a sequence. We simply decide the maximum sequence length and the position of the padding. But what if the sequence length is already greater than the maximum length? Well here just truncate! By the end we will have sequences with equal lengths. Here is how to do it:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F_QCkXONmTvQ"
      },
      "source": [
        "from keras.preprocessing.sequence import pad_sequences\n",
        "sequences = pad_sequences(data['Tokens'], maxlen=40, padding='pre', truncating='post')\n",
        "data['Padded'] = sequences.tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_WoQLiIkmaBQ"
      },
      "source": [
        "Simple right? Here are the final results:\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1RXDJ-n9RaMCJlp6ZeHLGh-domm2oEG3S\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "APk9TBzpulGU"
      },
      "source": [
        "#**7-Recurrent Neural Networks** [7]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pj6Z5B0eu5hb"
      },
      "source": [
        "## **7.1 Introduction**\n",
        "In this part of the series, we will introduce Recurrent Neural Networks aka RNNs that made a major breakthrough in predictive analytics for sequential data. This article covers RNNs on both conceptual and practical levels. We will start with the definition of RNNs, why and when they are used, then we will build an RNN ourselves for sentiment analysis.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dmSyxCtGvMxP"
      },
      "source": [
        "##**7.2-Why RNNs?**\n",
        "So far we have been working with regular tabular data. This data has no real notion of a sequence. That means that it does not matter whether we shuffle the fields or not, the model will still be able to train correctly. In case of sequential data it all changes. The order of inputs is very important in sequential data, and any order change would drastically affect the meaning behind the input. For example: We cannot switch the frames in a video, clips in an audio, or words in a sentence without messing up the whole meaning.\n",
        "\n",
        "The networks we learned so far do not take this ordering into consideration. We will illustrate the idea with an example. Given a dataset of sentences: **I love orange juice. I hate this movie. The sky is blue…,** build a model that predicts the sentiment behind each sentence. This means you have to know whether the author is having a positive, negative or neutral opinion.\n",
        "\n",
        "In classic fully connected networks we can solve this situation as follows:\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jWQ72jbCxJ3I"
      },
      "source": [
        "You can guess that this solution is not so efficient for many reasons:\n",
        "\n",
        "- The ordering between the words is not well captured in the network\n",
        "- The length may vary from one sentence to another\n",
        "- Neurons do not really understand the relation between words in the sentence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ezb_dJrSy1vE"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1ryILpVOwFPKIj933CEPak38TvNDX8itc\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "56Di20SIxr17"
      },
      "source": [
        "##7.3-What are RNNs?\n",
        "RNNs are designed much differently. Have a look at an alternative network for the same problem:\n",
        "\n",
        "The red arrows represent the activation value from each neuron. The first red arrow is the initial activation value (can be set randomly). All the following are calculated based on the weights of the current word and the activation of the previous one. Notice how each neuron is connected to the one before using the activation value. This what makes RNNs powerful. RNNs enable the network to understand the ordering of each word according to its position in a sentence.\n",
        "\n",
        "The figure represents a very simple RNN architecture. Actually there are many others. Some enable bidirectional connections between words, others output multiple values. In short we can classify they according to the number of inputs and number of outputs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S56Kf6QrzVCQ"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1u8Ycv7NOnWyILT0Qu9lQTY07oGXzZBEU)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sW5Z7e5c0KkK"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1Vyzajueie6e5AhtYTczSwaMHyE3TQGYo)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGyo1y6h0WB2"
      },
      "source": [
        "- Many to many RNNs are useful for translation. We give a sentence of many words, and expect the network to provide many translated words in different language.\n",
        "- Many to one is the same one we have designed for sentiment analysis. We give multiple input tokens and expect one value as result.\n",
        "- One to many is used often for sequence generation. You could give the network one input and it will generate a sequence based on what it learned. Example: Give a word and generate news, poetry, music, etc.\n",
        "\n",
        "These different types are somehow advanced, and we will cover them in future articles. For now, let us focus on our many to one sentiment analysis network."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MocjcE-kIOPj"
      },
      "source": [
        "## **7.4 Implementation**\n",
        "\n",
        "We are going to test our RNNs with the Spam SMS dataset. For more information about the dataset and how to preprocess the data please refer to the previous post in this series .Deep Learning with Keras – Part 6: Textual Data Preprocessing.\n",
        "\n",
        "We have reached this far in the previous tutorial:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LVRMUAYAIoTO"
      },
      "source": [
        "import pandas as pd\n",
        "data = pd.read_csv('./data/sms.csv')\n",
        "from keras.preprocessing.text import Tokenizer\n",
        "tokenizer = Tokenizer(num_words=10000)\n",
        "tokenizer.fit_on_texts(data['SMS'])\n",
        "data['Tokens'] = tokenizer.texts_to_sequences(data['SMS'])\n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "sequences = pad_sequences(data['Tokens'], maxlen=40, padding='pre', truncating='post')\n",
        "data['Padded'] = sequences.tolist()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6YMdsCeIsLF"
      },
      "source": [
        "The goal is to extend the above code by adding and training the RNN model. But let us review what we did first. We started by loading the data, we used a tokenizer to break the sentences into tokens, we specified the num_words=10000 to get only the top 10000 words in the document, we padded the sequences so that they can all have a length of 40.\n",
        "\n",
        "Next we have to split our data to training and testing. We will do it using the sklearn library:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S4Yd5OhAI2wu"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = train_test_split(data['Padded'], data['Label'], test_size=0.3)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GBGLQKm8JaLn"
      },
      "source": [
        "The X is the data[‘Padded’] and the Y is the data[‘Label’]. If we have a look at the shape of each set you will see that they have the shapes: ((3900,), (1672,), (3900,), (1672,)). So we need to fix that in order to match the shape of: number of samples * sequence length (which is 40 in our case). This is how it is done:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oa5vJyBAJib6"
      },
      "source": [
        "import numpy as np\n",
        "X_train = np.vstack(X_train.values)\n",
        "X_test = np.vstack(X_test.values)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "smLx1jhQJmXW"
      },
      "source": [
        "Time to define the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HHtXL5EPJn8m"
      },
      "source": [
        "from keras import layers, models\n",
        "model = models.Sequential()\n",
        "model.add(layers.Embedding(10000, 128, input_length=40))\n",
        "model.add(layers.LSTM(32, activation='tanh'))\n",
        "model.add(layers.Dense(1, activation='sigmoid'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VFKWOtgwKAwo"
      },
      "source": [
        "First, the Embedding layer is a special layer used especially for text. We gave it the following parameters:\n",
        "\n",
        "- number of words/tokens in the data: in our case we chose to take 10000 words from the dataset so this is our number\n",
        "- the output embedding size: an embedding is a vector that represents the characteristics of each word. Here we chose to extract 128 characteristics from each word. (More about this layer will be discussed in future tutorials)\n",
        "- the length of each sentence: which is the padded sequence length (40 in our case)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bi9e7XqmKQYd"
      },
      "source": [
        "After the embedding layer we added an LSTM layer. LSTM (Long short term memory) is a special type of RNN that proved to have a very good performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VwynLI2hKhTI"
      },
      "source": [
        "model.compile('adam', 'binary_crossentropy', metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SPImEN4pKkWH"
      },
      "source": [
        "Time for training:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Pl-EhinpKmQV"
      },
      "source": [
        "model.fit(X_train, y_train, epochs=5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jBHH6NzwKslk"
      },
      "source": [
        "In only 5 ephocs, we were able to reach a 99% accuracy! Is it too good to be true? Let us evaluate it on the testing data:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tmsmRWTZKvKm"
      },
      "source": [
        "model.evaluate(X_train, y_train)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MYP4GVGL6BLn"
      },
      "source": [
        "# **8-Introduction of Deep learning with Kera**[12]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Avi4DYxClDZ"
      },
      "source": [
        "##**8.1 Introducing Keras**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c6szHRU-6Lgl"
      },
      "source": [
        "## **8.1.1-Counting parameters**\n",
        "You've just created a neural network. But you're going to create a new one now, taking some time to think about the weights of each layer. The Keras Dense layer and the Sequential model are already loaded for you to use.\n",
        "This is the network you will be creating:\n",
        "Instantiate a new Sequential() model.\n",
        "Add a Dense() layer with five neurons and three neurons as input.\n",
        "Add a final dense layer with one neuron and no activation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2anhNrA66E0"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1ZFYOC6z-mIdTaH_UTsl30CNN-XT2YQ0G)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0YPnBqUsAuaa"
      },
      "source": [
        "# Instantiate a new Sequential model\n",
        "model = Sequential()\n",
        "# Add a Dense layer with five neurons and three inputs\n",
        "model.add(Dense(10, input_shape=(2,), activation=\"relu\"))\n",
        "# Add a final Dense layer with one neuron and no activation\n",
        "model.add(Dense(1))\n",
        "# Summarize your model\n",
        "model.summary()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wk9PjmfJBBic"
      },
      "source": [
        "![](https://drive.google.com/uc?export=view&id=1Rf4tyCTeYeeCWUYm22m8s6K-Q8ECMSzj)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vpdpU1CZ6vye"
      },
      "source": [
        "# **9-Hyperparameter Tuning Methods With Keras Tuner**[18]\n",
        "\n",
        "The same analogy is true for building a highly accurate model. Where getting the best hyperparameters using the hyperparameter tuning packages such as keras tuner changes everything.\n",
        "\n",
        "After many such iterations. I asked myself when all the machine learning or deep learning algorithms implementation is available online, why only a few people are getting high in the leaderboard. \n",
        "\n",
        "The reason is straightforward: they know what the best parameters to use to build the model for the given datasets are.\n",
        "\n",
        "How can we get the best parameters then? \n",
        "For that, we need to perform the hyperparameter tuning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7EZf42f773wd"
      },
      "source": [
        "## **9-2 Questions While Building Machine Learning or Deep Learning Models?[18]**\n",
        "\n",
        "One of the most important things we need to question ourselves when we try to build any machine learning or deep learning models or computer vision models is \n",
        "\n",
        "- What depth of the tree is good for the decision tree?\n",
        "- What is the best method to calculate the root for a random forest model? \n",
        "- How many hidden layers do we have to add to our neural network model?\n",
        "- How many neurons do we need to use in the RNN hidden layer?\n",
        "- Which active functions will give better results for models to build on the provided data? \n",
        "\n",
        "If you observe carefully, all these questions have the same pattern.\n",
        "\n",
        "- Could you guess what it is?\n",
        "\n",
        "The pattern is we don’t have a specific answer for all these questions. The answer to these questions entirely depend on the data we are using. \n",
        "\n",
        "Which means the answers will vary from problem to problem. As for each problem, the dataset we are going to use is entirely different.\n",
        "\n",
        "In simple words, everyone needs to know which hyperparameter values we need to be selected to get a higher model performance."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g05i-0UY9I2_"
      },
      "source": [
        "## **9.2 What Are Hyperparameters?[18]**\n",
        "\n",
        "In the beginning stage of implementing the machine learning (ML) or deep learning (DL) models, people tend to confuse about selecting hyperparameters for their ML or DL models. \n",
        "\n",
        "What are these hyperparameters?\n",
        "\n",
        "Every machine learning or deep learning algorithm has two kinds of parameters. \n",
        "\n",
        "- Trainable Parameters\n",
        "- Model Hyperparamters\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kkmF0dhL-lml"
      },
      "source": [
        "**Trainable Parameters**\n",
        "\n",
        "Trainable parameters are the configuration variables learned by the algorithms during the model's training. This means these values are learned from the training data, which can not set manually.\n",
        "\n",
        "For example, weights and bias in neural networks are the trainable model parameters.\n",
        "\n",
        "**Model Hyperparameters**\n",
        "\n",
        "Model Hyperparameters are the external configuration to the model, and we have to provide before starting or beginning of training a model. \n",
        "\n",
        "These values are not estimated from the training data. But these values are used to estimate model trainable parameters.\n",
        "\n",
        "For example. While building random forest or decision tree algorithm models, we need to define how many decision trees we need for model training. \n",
        "\n",
        "When we build neural networks, we need to determine how many hidden layers will give better performance after training the model by optimising the loss functions. \n",
        "\n",
        "There are many hyperparameters like this. Few or well-known hyperparameters are related to neural networks.\n",
        "\n",
        "- Number of hidden layers and number of units in each hidden layer\n",
        "Dropout\n",
        "- Initialization of Network weights\n",
        "- Activation functions\n",
        "- Learning rate\n",
        "- Number of epochs\n",
        "- Batch size, etc.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cfomFOA3_fYl"
      },
      "source": [
        "## **9.3 -What Is Hyperparameter Tuning?**[18]\n",
        "\n",
        "We don’t know the most suitable or best values for all these model hyperparameters for our problem statements.\n",
        "\n",
        "We may take hyperparameter values from other problem statements. Which are already solved by others with high-performance value. Else we can search for the best value by using trial and error techniques.\n",
        "\n",
        "Hyperparameter tuning is the process of searching for the best values for the hyperparameters of the ideal model.\n",
        "\n",
        "Till now, you know what the hyperparameters and hyperparameter tuning are. In the next section, we will discuss why this hyperparameter tuning is essential for our model building.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3LrshfURphNd"
      },
      "source": [
        "## **9.4 Importance Of Hyperparameter Tuning**[18]\n",
        "\n",
        "Tuning the model hyperparameters is essential because hyperparameters directly control the training ML/DL models' behavior. Also, it shows a significant impact on the trained model, which accurately predict results on unseen data. \n",
        "\n",
        "In other words, Helps in building the high performance models, when we quantify popular evaluation metrics.\n",
        "\n",
        "The right choice of the hyperparameter values while tuning the model will shine the trained model effectively.\n",
        "\n",
        "Nowadays, Artificial intelligence and its subfields, such as machine learning, deep learning, natural language processing, and computer vision techniques, are providing the most reliable & accurate results.\n",
        "\n",
        "These solutions simplify the traditional/old process techniques in many industries.  But building these models require more resources like space and time consuming and computationally expensive.\n",
        "\n",
        "So finding the best values for hyperparameters of our model is significant to save our resources. \n",
        "\n",
        "Choosing an appropriate or most suitable hyperparameter for our ML/DL problem statements is the critical task or step while building an effective and accurate model. Most of the ML/DL models' performance depends on their hyperparameter values.\n",
        "\n",
        "To get efficient results from the problem statements of machine learning or deep learning algorithms. You should focus more while tuning the model hyperparameters.\n",
        "\n",
        "Tuning of the hyperparameter requires search space. This helps in searching for the best hyperparameters from our choice and range of hyperparameters.\n",
        "\n",
        "This process will become computationally expensive with the increment of the number of hyperparameters. \n",
        "\n",
        "So while selecting hyperparameters, we always need to keep in mind which hyperparameters have an impact on the model’s performance.\n",
        "\n",
        "Few points where exactly hyperparameter tuning will show its impact to improve model performance.\n",
        "\n",
        "- Accelerate accuracy on the test/validation dataset\n",
        "- Reduce the number of model parameters\n",
        "- Reduce the number of hidden layers\n",
        "- Faster inference speed"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "41IWiMJL5qsL"
      },
      "source": [
        "## **9.5 -Popular Hyperparameter Tuning Methods [18]**\n",
        "\n",
        "Machine learning or deep learning model tuning is a kind of optimization problem. We have different types of hyperparameters for each model.\n",
        "\n",
        "Our goal here is to find the best combination of those hyperparameter values. These values can help to minimize model loss or maximize the model accuracy values.\n",
        "\n",
        "There are few hyperparameter tuning techniques available for us. In this section, we will discuss a few of those techniques.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MAd8iPY86H0U"
      },
      "source": [
        "**Manual search**\n",
        "\n",
        "In the manual search, we choose a few model hyperparameters and their values by our experience or by random guess.\n",
        "\n",
        "Build a model with guessed hyperparameter values and then check performance value, either loss or accuracy, or any other performance metrics. This is a repeated process until we are satisfied with our model performance value.\n",
        "\n",
        "**Grid search**\n",
        "\n",
        "This is a very traditional technique for hyperparameter tuning. In the Grid search method, we can set up a grid of specific model hyperparameters and then train/ test our problem statement model on every combination of values. \n",
        "\n",
        "**Random search**\n",
        "\n",
        "This is another kind of hyperparameter tuning technique. Here also, we are set up a grid of specific model hyperparameter values the same as the grid search, but here we are selecting the combination of hyperparameter values randomly. \n",
        "\n",
        "A random search tuning technique is faster than a grid search, but a grid search is more effective than a random search.\n",
        "\n",
        "Because grid search utilizes all combinations of selected hyperparameter values, but a random search misses a few of those combinations, maybe that combination will improve model performance value.\n",
        "\n",
        "But both tuning techniques, Grid and Random search don't use prior information from the past experiments to select the next set of hyperparameter value combinations.\n",
        "\n",
        "**Bayesian optimization**\n",
        "\n",
        "The setting of best values for model hyperparameters will improve the model's performance because most of the ML/DL models frequently require tuning processes. We discussed this point already; we just remember basic points for you. \n",
        "\n",
        "Above discussed, hyperparameter tuning techniques such as grid and random search use full space, which one allocated for available hyperparameter values in an isolated way without paying attention to the previous results or previous combinations.\n",
        "\n",
        "The tuning process will become a time-consuming task in these both tuning techniques if the number of hyperparameter values increases. Means search space will gradually increase along with the number of hyperparameters.\n",
        "\n",
        "In the case of Bayesian optimization tuning techniques, tuning processes will reduce the time spent to get optimal values for the model hyperparameters and also produce better generalization results on the test data.\n",
        "\n",
        "The Bayesian optimization technique will consider the previous set of hyperparameters while choosing a new set of hyperparameters to evaluate model performance.\n",
        "\n",
        "This bayesian optimization technique is more effective than previous tuning techniques.\n",
        "\n",
        "Till now, we know what kind of process we use while tuning the model hyperparameters.\n",
        "\n",
        "In this article, we decided to discuss the Keras tuner library for this tuning purpose. Next few minutes, we are focusing on that; you can get an idea about the Keras tuner library, a few methods of it.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3CqAbhDzBZ_E"
      },
      "source": [
        "##**9.6 -Introduction to Keras tuner [18]**\n",
        "\n",
        "Keras tuner is a library to perform hyperparameter tuning with Tensorflow 2.0. This library solves the pain points of searching for the best suitable hyperparameter values for our ML/DL models. \n",
        "\n",
        "In short, Keras tuner aims to find the most significant values for hyperparameters of specified ML/DL models with the help of the tuners.\n",
        "\n",
        "These tuners are like searching agents to find the right hyperparameter values. Keras tuner comes with the above-mentioned tuning techniques such as **random search, Bayesian optimization, etc.**\n",
        "\n",
        "Present Keras Tuner provides four kinds of tuners.\n",
        "\n",
        "- **Bayesian Optimization**\n",
        "- **Hyperband**\n",
        "- **Random search**\n",
        "- **Sklearn**\n",
        "\n",
        "We will discuss all these tuner snippets and other useful methods to build tuners in the next section.\n",
        "\n",
        "In the coming section, we plan to discuss a list of steps to find the best model using a tuner.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "57PrkvmDw2Ff"
      },
      "source": [
        "## **9.7 Selecting Best Model Using Keras Tuner**[18]\n",
        "\n",
        "In this section will learn all steps you need to perform to get the best hyperparameter values for our model. Let's start discussing a list of steps.\n",
        "\n",
        "**Step 1: Install Libraries**\n",
        "\n",
        "First of all, install all required libraries to build our machine learning or deep learning model.\n",
        "\n",
        "**Step 2: import Libraries**\n",
        "\n",
        "After installing libraries, we need to import those libraries and default libraries to use them.\n",
        "\n",
        "**Step 3: Load dataset**\n",
        "\n",
        "Whatever dataset we need to use for our problem statement to build our model, we have to load that dataset.\n",
        "\n",
        "**Step 4: Preprocessing**\n",
        "\n",
        "The real-time dataset doesn't have any good quality means it has noise, missing values, etc. in this step, we need to apply a few preprocessing steps to make it quality data.\n",
        "\n",
        "**Step 5: Splitting data**\n",
        "\n",
        "We split the whole dataset into the train, test sets or train, test, validation sets.\n",
        "\n",
        "Train dataset: Used for training a model while building a model. The model will learn patterns from this data.\n",
        "\n",
        "Test dataset: This is for testing our trained model. We take performance value based on model results after applying on test data because it will apply a learning pattern on test data.\n",
        "\n",
        "Validation dataset: this is entirely new data. For example, if our dataset has information about bank customers from 2000 to 2010.\n",
        "\n",
        "We take 2000 to 2009 bank information for training and testing datasets; we have 2010 information or data for validation trained models.\n",
        "\n",
        "**Step 6: Define the model**\n",
        "\n",
        "After splitting the dataset into a train and test, we have to define an ML/DL model for our problem statement.\n",
        "\n",
        "**Step 7: Search Space Definition**\n",
        "\n",
        "In this step, we have to provide values for our model hyperparameters—which means which hyperparameter we want to optimize with their values in some range or few values.\n",
        "\n",
        "Step 8: Searching for the best model with good hyperparameter values\n",
        "Here in this Keras, the tuner will train the model with the combination of different hyperparameter values. \n",
        "\n",
        "Step 9: Get The Best Model\n",
        "After completing the 8th step, we can get the best model with corresponding hyperparameter values and performance values.\n",
        "\n",
        "In the next section, we will show how you can find the best hyperparameter with code.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IFCe592HxkDL"
      },
      "source": [
        "## **9.8 Hyperparameter Tuning With Keras Tuner[18]**\n",
        "Now we will give you the codes to build various Keras tuner methods. You just need to copy these and change them according to your needs. Let’s start with the bayesian optimization tuning methods.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ZqhfaGgx0Rj"
      },
      "source": [
        "**Bayesian Optimization Tuning Implementation With Kera**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4t4no6Rlx6x6"
      },
      "source": [
        "kerastuner.tuners.bayesian.BayesianOptimization(hypermodel, objective, max_trials, num_initial_points=2, seed=None, \n",
        "hyperparameters=None, tune_new_entries=True, allow_new_entries=True, **kwargs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WEybCaEGyMc2"
      },
      "source": [
        "Arguments Explanation:\n",
        "\n",
        "**hypermodel:** Instance of HyperModel class (or callable that takes hyperparameters and returns a Model instance).\n",
        "\n",
        "**objective:** String. Name of model metric to minimize or maximize, e.g. \"val_accuracy\".\n",
        "\n",
        "**max_trials:** Int. Total number of trials (model configurations) to test at most. Note that the oracle may interrupt the search before max_trial models have been tested if the search space has been exhausted.\n",
        "\n",
        "**num_initial_points:** Int. The number of randomly generated samples as initial training data for Bayesian optimization.\n",
        "\n",
        "**alpha:** Float or array-like. Value added to the diagonal of the kernel matrix during fitting.\n",
        "\n",
        "**beta:** Float. The balancing factor of exploration and exploitation. The larger it is, the more explorative it is.\n",
        "\n",
        "**seed:** Int. Random seed.\n",
        "\n",
        "**hyperparameters:** HyperParameters class instance. Used to override (or register in advance) hyperparamters in the search space.\n",
        "\n",
        "**tune_new_entries:** Whether hyperparameter entries that are requested by the hypermodel but that were not specified in hyperparameters should be added to the search space, or not. If not, then the default value for these parameters will be used.\n",
        "\n",
        "**allow_new_entries:** Whether the hypermodel is allowed to request hyperparameter entries not listed in hyperparameters.\n",
        "\n",
        "**kwargs:** Keyword arguments relevant to all Tuner subclasses. Please see the docstring for Tuner."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DH6U8JCBzefS"
      },
      "source": [
        "**Hyperband tuning implementation with Keras**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2oIa_m6hznjl"
      },
      "source": [
        "kerastuner.tuners.hyperband.Hyperband(hypermodel, objective, max_epochs, factor=3, hyperband_iterations=1, seed=None, \n",
        "hyperparameters=None, tune_new_entries=True, allow_new_entries=True, **kwargs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JOxKagl4zpZd"
      },
      "source": [
        "Arguments Explanation\n",
        "\n",
        "hypermodel: Instance of HyperModel class (or callable that takes hyperparameters and returns a Model instance).\n",
        "\n",
        "objective: String. Name of model metric to minimize or maximize, e.g. \"val_accuracy\".\n",
        "\n",
        "max_epochs: Int. The maximum number of epochs to train one model. It is recommended to set this to a value slightly higher than the expected time to convergence for your largest Model, and to use early stopping during training (for example, via tf.keras.callbacks.EarlyStopping).\n",
        "\n",
        "factor: Int. Reduction factor for the number of epochs and number of models for each bracket.\n",
        "\n",
        "hyperband_iterations: Int >= 1. The number of times to iterate over the full Hyperband algorithm. One iteration will run approximately max_epochs * (math.log(max_epochs, factor) ** 2) cumulative epochs across all trials. It is recommended to set this to as high a value as is within your resource budget.\n",
        "\n",
        "seed: Int. Random seed.\n",
        "\n",
        "hyperparameters: HyperParameters class instance. Can be used to override (or register in advance) hyperparamters in the search space.\n",
        "\n",
        "tune_new_entries: Whether hyperparameter entries that are requested by the hypermodel but that were not specified in hyperparameters should be added to the search space, or not. If not, then the default value for these parameters will be used.\n",
        "\n",
        "allow_new_entries: Whether the hypermodel is allowed to request hyperparameter entries not listed in hyperparameters.\n",
        "\n",
        "**kwargs: Keyword arguments relevant to all Tuner subclasses. Please see the docstring for Tuner."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZClL3Tdiz2R8"
      },
      "source": [
        "**Random search tuning implementation with Keras**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QfhhmC8zz5NV"
      },
      "source": [
        "kerastuner.tuners.randomsearch.RandomSearch(hypermodel, objective, max_trials, seed=None, \n",
        "hyperparameters=None, tune_new_entries=True, allow_new_entries=True, **kwargs)\n",
        " "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TC2BY-nT0Gj0"
      },
      "source": [
        "Arguments Explanation:\n",
        "\n",
        "hypermodel: Instance of HyperModel class (or callable that takes hyperparameters and returns a Model instance).\n",
        "\n",
        "objective: String. Name of model metric to minimize or maximize, e.g. \"val_accuracy\".\n",
        "\n",
        "max_trials: Int. Total number of trials (model configurations) to test at most. Note that the oracle may interrupt the search before max_trial models have been tested.\n",
        "\n",
        "seed: Int. Random seed.\n",
        "\n",
        "hyperparameters: HyperParameters class instance. Can be used to override (or register in advance) hyperparamters in the search space.\n",
        "\n",
        "tune_new_entries: Whether hyperparameter entries that are requested by the hypermodel but that were not specified in hyperparameters should be added to the search space, or not. If not, then the default value for these parameters will be used.\n",
        "\n",
        "allow_new_entries: Whether the hypermodel is allowed to request hyperparameter entries not listed in hyperparameters.\n",
        "\n",
        "**kwargs: Keyword arguments relevant to all Tuner subclasses. Please see the docstring for Tuner."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tw2QmnUU0JQ-"
      },
      "source": [
        "**Sklearn tuning implementation with Keras**\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k6iZg7wQ0OcE"
      },
      "source": [
        "kerastuner.tuners.sklearn.Sklearn(oracle, hypermodel, scoring=None, metrics=None, cv=KFold(n_splits=5, \n",
        "random_state=1, shuffle=True), **kwargs)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QHaZLU2y0Tjk"
      },
      "source": [
        "Arguments Explanation:\n",
        "\n",
        "oracle: An instance of the kerastuner.Oracle class. Note that for this Tuner, the objective for the Oracle should always be set to Objective('score', direction='max'). Also, Oracles that exploit Neural-Network-specific training (e.g. Hyperband) should not be used with this Tuner.\n",
        "\n",
        "hypermodel: Instance of HyperModel class (or callable that takes a Hyperparameters object and returns a Model instance).\n",
        "\n",
        "scoring: An sklearn scoring function. For more information, see sklearn.metrics.make_scorer. If not provided, the Model's default scoring will be used via model.score. Note that if you are searching across different Model families, the default scoring for these Models will often be different. In this case you should supply scoring here in order to make sure your Models are being scored on the same metric.\n",
        "\n",
        "metrics: Additional sklearn.metrics functions to monitor during search. Note that these metrics do not affect the search process.\n",
        "\n",
        "cv: An sklearn.model_selection Splitter class. Used to determine how samples are split up into groups for cross-validation.\n",
        "\n",
        "**kwargs: Keyword arguments relevant to all Tuner subclasses. Please see the docstring for Tuner.\n",
        "\n",
        "In the above section we have given the codes for various keras tuners. The below explains you use the keras tuners in your model building tasks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hQMMbtCF0iTt"
      },
      "source": [
        "## Function for building \n",
        "## A basic classification model\n",
        "\n",
        "def build_model(hp):\n",
        "  ## Input : hyperparameter tuner\n",
        "  ## Output : Model object\n",
        "\n",
        "  ## Initialize sequence model\n",
        "  model = keras.Sequential()\n",
        "  \n",
        "  ## Add embedding layer\n",
        "  model.add(layers.Embedding(vocab_size, embedding_dim, input_length=max_length))\n",
        "  \n",
        "  ## convert 2D input to 1D input\n",
        "  model.add(layers.GlobalAveragePooling1D())\n",
        "  \n",
        "  ## for loop for hidden layers\n",
        "  for i in range(hp.Int('num_layers', 1, 20)):\n",
        "    ## Add hidden layer and \n",
        "    ## finding units for each layer\n",
        "    model.add(layers.Dense(units=hp.Int('units' + str(i),\n",
        "                                      min_value=32,\n",
        "                                      max_value=512,\n",
        "                                      step=32),\n",
        "                         activation='relu'))\n",
        "    ## Add output layer\n",
        "    model.add(layers.Dense(1, activation='sigmoid'))\n",
        "    ## Add optimizers and metrics\n",
        "    model.compile(\n",
        "      optimizer=keras.optimizers.Adam(\n",
        "          hp.Choice('learning_rate',\n",
        "                    values=[1e-2, 1e-3, 1e-4])),\n",
        "                loss='binary_crossentropy',\n",
        "                metrics=['accuracy'])\n",
        "  return model\n",
        "  \n",
        "  \n",
        "## Searching for the best model with good hyperparameter values\n",
        "tuner = RandomSearch(\n",
        "    build_model,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=5,\n",
        "    executions_per_trial=3,\n",
        "    directory='my_dir',\n",
        "    project_name='sms_spam_classification')\n",
        "    \n",
        "## summary of the search space like \n",
        "## which hyperparameter will optimize \n",
        "tuner.search_space_summary()\n",
        "\n",
        "tuner.search(train_sent, train_target,epochs=5,validation_data=(test_sent, test_target))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vm7uTbEWCAei"
      },
      "source": [
        "#**References**\n",
        "\n",
        " [1] Deep learning with keras Tutorila part 1\n",
        "https://www.marktechpost.com/2019/06/11/deep-learning-with-keras-tutorial-part-1/?fbclid=IwAR0Mbtuas8dNItrxRQvAmZG2OwYXOe6JkjTLmZUNFokVcDyLYHIlENSOze0\n",
        "\n",
        "[2] Regression with Keras (Deep Learning with Keras – Part 3)\n",
        "https://www.marktechpost.com/2019/06/17/regression-with-keras-deep-learning-with-keras-part-3/?fbclid=IwAR0WEGYwi1nlfK_eaH1qK3JD8ITMYc82Q15uT_vC4DnNUOgJboEn8KLr5_Y\n",
        "\n",
        "[3] Data Pre-processing for Deep Learning models (Deep Learning with Keras – Part 2)\n",
        "https://www.marktechpost.com/2019/06/14/data-pre-processing-for-deep-learning-models-deep-learning-with-keras-part-2/?fbclid=IwAR22EF2_QJDOXwc1bFrgiA-ciKzwkbznmcXm8rf8xMJSqkZpKaAsay-_ZuU\n",
        "\n",
        "[4] Deep Learning with Keras – Part 4: Classification\n",
        "https://www.marktechpost.com/2019/06/24/deep-learning-with-keras-part-4-classification/?fbclid=IwAR1NzkaFT4Ys-QsafSGqx7AgXpwiFt183FU57K1lr19VFubux3I7is1y9og\n",
        "\n",
        "[5] Deep Learning with Keras – Part 5: Convolutional Neural Networks\n",
        "https://www.marktechpost.com/2019/07/04/deep-learning-with-keras-part-5-convolutional-neural-networks/?fbclid=IwAR3TveW1lbQ7um9EyL5LG1Mb0SEU5JjDxumHMC5EgCCA23c2A75Dkyv-qio\n",
        "\n",
        "[6] Deep Learning with Keras – Part 6: Textual Data Preprocessing\n",
        "https://www.marktechpost.com/2019/09/13/deep-learning-with-keras-part-6-textual-data-preprocessing/?fbclid=IwAR04cX9X8R6_Lu88p3NfYDnZlY3h1YRQoPT85viLPEZlDuSb75T-zskyC94\n",
        "\n",
        "[7]Deep Learning with Keras – Part 7: Recurrent Neural Networks\n",
        "https://www.marktechpost.com/2019/10/01/deep-learning-with-keras-part-7-recurrent-neural-networks/?fbclid=IwAR2DdCwqkUfDsHHe-n4VVWiZBtqwIdMqHw38z7qc82koBo6yLrW9wyxwNLU\n",
        "\n",
        "[8] Image Classification With CNN\n",
        "https://medium.com/swlh/image-classification-with-cnn-4f2a501faadb\n",
        "\n",
        "[9]What is One Hot Encoding | One Hot Encoding | Machine Learning | Data Magic\n",
        "https://www.youtube.com/watch?v=pWq2-PCS8Rg&feature=share&fbclid=IwAR2iMNmccJicoanxjzkCHFa0vWx7GnHMBRG6IltzG2HKuvonkZtCEG7WHuY\n",
        "\n",
        "[[10]Explanation of Keras for Deep Learning and Using It in Real World Problem ](https://laconicml.com/keras-deep-learning/)\n",
        "\n",
        "[11]Introduction to Deep Learning with Keras\n",
        "https://learn.datacamp.com/courses/introduction-to-deep-learning-with-keras\n",
        "\n",
        "[12] Deep-Learning-with-Keras\n",
        "https://github.com/PacktPublishing/Deep-Learning-with-Keras\n",
        "\n",
        "[13] Machine-Learning-Scientist-with-Python-by-DataCamp\n",
        "https://github.com/abdelrahmaan/Machine-Learning-Scientist-with-Python-by-DataCamp\n",
        "\n",
        "[14]Deep Learning with Keras - Python\n",
        "https://www.youtube.com/playlist?list=PLVBorYCcu-xX3Ppjb_sqBd_Xf6GqagQyl\n",
        "\n",
        "[15] Explanation of Keras for Deep Learning and Using It in Real World Problem\n",
        "https://laconicml.com/keras-deep-learning/\n",
        "\n",
        "[16]Your First Deep Learning Project in Python with Keras Step-By-Step\n",
        "\n",
        "https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/?fbclid=IwAR0BDm2fEAOSNXKwvc-Fulx0nxdbmFoLDUrTShvwCz0cIVXAFkqWR5iQp6k\n",
        "\n",
        "[[17] Keras Tutorial for Beginners: Deep Learning in Python with Example\n",
        "What is Keras?](https://morioh.com/p/64add0864eec)\n",
        "\n",
        "[[18] Four Popular Hyperparameter Tuning Methods With Keras Tuner](https://dataaspirant.com/hyperparameter-tuning-with-keras-tuner/?fbclid=IwAR3c0z1rAOyWRpIp9DfS1aB46_P4CHSHreOnN32fzmxImw1kZddPVgS1o-g)\n",
        "\n"
      ]
    }
  ]
}