08_introduction_to_nlp_in_tensorflow.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "08_introduction_to_nlp_in_tensorflow.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8nFrwTxHTxSa"
      },
      "source": [
        "# 08. Natural Language Processing with TensorFlow¶"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JGJXuWsIUw5X"
      },
      "source": [
        "The main goal of natural language processing (NLP) is to derive information from natural language.\n",
        "\n",
        "Natural language is a broad term but you can consider it to cover any of the following:\n",
        "\n",
        "- Text (such as that contained in an email, blog post, book, Tweet)\n",
        "- Speech (a conversation you have with a doctor, voice commands you give to a smart speaker)\n",
        "\n",
        "Under the umbrellas of text and speech there are many different things you might want to do.\n",
        "\n",
        "If you're building an email application, you might want to scan incoming emails to see if they're spam or not spam (classification).\n",
        "\n",
        "If you're trying to analyse customer feedback complaints, you might want to discover which section of your business they're for.\n",
        "\n",
        "🔑 Note: Both of these types of data are often referred to as sequences (a sentence is a sequence of words). So a common term you'll come across in NLP problems is called seq2seq, in other words, finding information in one sequence to produce another sequence (e.g. converting a speech command to a sequence of text-based steps).\n",
        "\n",
        "To get hands-on with NLP in TensorFlow, we're going to practice the steps we've used previously but this time with text data:\n",
        "\n",
        "Text -> turn into numbers -> build a model -> train the model to find patterns -> use patterns (make predictions)\n",
        "\n",
        "📖 Resource: For a great overview of NLP and the different problems within it, read the article[A Simple Introduction to Natural Language Processing.](https://becominghuman.ai/a-simple-introduction-to-natural-language-processing-ea66a1747b32)\n",
        "\n",
        "**What we're going to cover\n",
        "Let's get specific hey?**\n",
        "\n",
        "- Downloading a text dataset\n",
        "- Visualizing text data\n",
        "- Converting text into numbers using tokenization\n",
        "- Turning our tokenized text into an embedding\n",
        "- Modelling a text dataset\n",
        "  - Starting with a baseline (TF-IDF)\n",
        "  - Building several deep learning text models\n",
        "    - Dense, LSTM, GRU, Conv1D, Transfer learning\n",
        "- Comparing the performance of each our models\n",
        "- Combining our models into an ensemble\n",
        "- Saving and loading a trained model\n",
        "- Find the most wrong predictions"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VNHPL4o0WA0p"
      },
      "source": [
        "## Get helper functions\n",
        "In past modules, we've created a bunch of helper functions to do small tasks required for our notebooks.\n",
        "\n",
        "Rather than rewrite all of these, we can import a script and load them in from there.\n",
        "\n",
        "The script containing our helper functions can be found on GitHub."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IoMR6yRKTZeo",
        "outputId": "a3910a16-32a1-4f01-9171-9a5de590b4ba"
      },
      "source": [
        "# Download helper functions script\n",
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-23 07:07:38--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10246 (10K) [text/plain]\n",
            "Saving to: ‘helper_functions.py’\n",
            "\n",
            "helper_functions.py 100%[===================>]  10.01K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-09-23 07:07:38 (90.1 MB/s) - ‘helper_functions.py’ saved [10246/10246]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Id1ihOXRWFxs"
      },
      "source": [
        "# Import series of helper function for this notebook\n",
        "from helper_functions import unzip_data, create_tensorboard_callback, plot_loss_curves, compare_historys"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5cvItFxMWZ19"
      },
      "source": [
        "## Download a text dataset\n",
        "Let's start by download a text dataset. We'll be using the Real or Not? datset from Kaggle which contains text-based Tweets about natural disasters.\n",
        "\n",
        "The Real Tweets are actually about diasters, for example:\n",
        "\n",
        "- Jetstar and Virgin forced to cancel Bali flights again because of ash from Mount Raung volcano\n",
        "\n",
        "The Not Real Tweets are Tweets not about diasters (they can be on anything), for example:\n",
        "\n",
        "- 'Education is the most powerful weapon which you can use to change the world.' Nelson #Mandela #quote\n",
        "\n",
        "For convenience, the dataset has been downloaded from Kaggle (doing this requires a Kaggle account) and uploaded as a downloadable zip file.\n",
        "\n",
        "🔑 Note: The original downloaded data has not been altered to how you would download it from Kaggle."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ONqSTRpPWNx7",
        "outputId": "43c94109-5e5c-4730-99f2-7a8a41983d8b"
      },
      "source": [
        "# Download data (same as from Kaggle)\n",
        "!wget \"https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\"\n",
        "\n",
        "# Unzip data\n",
        "unzip_data(\"nlp_getting_started.zip\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2021-09-23 07:07:40--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.250.141.128, 142.251.2.128, 2607:f8b0:4023:c0b::80, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.250.141.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip’\n",
            "\n",
            "\rnlp_getting_started   0%[                    ]       0  --.-KB/s               \rnlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.005s  \n",
            "\n",
            "2021-09-23 07:07:40 (122 MB/s) - ‘nlp_getting_started.zip’ saved [607343/607343]\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yPDrUqNOWrM_"
      },
      "source": [
        "Unzipping nlp_getting_started.zip gives the following 3 .csv files:\n",
        "\n",
        "- sample_submission.csv - an example of the file you'd submit to the Kaggle competition of your model's predictions.\n",
        "- train.csv - training samples of real and not real diaster Tweets.\n",
        "- test.csv - testing samples of real and not real diaster Tweets.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dS10s2bUW8w5"
      },
      "source": [
        "## Visualizing a text dataset\n",
        "Once you've acquired a new dataset to work with, what should you do first?\n",
        "\n",
        "Explore it? Inspect it? Verify it? Become one with it?\n",
        "\n",
        "All correct.\n",
        "\n",
        "Remember the motto: visualize, visualize, visualize.\n",
        "\n",
        "Right now, our text data samples are in the form of .csv files. For an easy way to make them visual, let's turn them into pandas DataFrame's.\n",
        "\n",
        "\n",
        "> 📖 Reading: You might come across text datasets in many different formats. Aside from CSV files (what we're working with), you'll probably encounter .txt files and .json files too. For working with these type of files, I'd recommend reading the two following articles by RealPython:\n",
        "\n",
        "How to Read and Write Files in Python\n",
        "\n",
        "Working with JSON Data in Python\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "m1lPLHnoWlFH",
        "outputId": "9c42c624-10c7-439a-f690-687c014650e1"
      },
      "source": [
        "# Turn .csv files into pandas dataframe\n",
        "import pandas as pd\n",
        "train_df = pd.read_csv(\"train.csv\")\n",
        "test_df = pd.read_csv(\"test.csv\")\n",
        "train_df.head()"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Our Deeds are the Reason of this #earthquake M...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Forest fire near La Ronge Sask. Canada</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>All residents asked to 'shelter in place' are ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>13,000 people receive #wildfires evacuation or...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just got sent this photo from Ruby #Alaska as ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword  ...                                               text target\n",
              "0   1     NaN  ...  Our Deeds are the Reason of this #earthquake M...      1\n",
              "1   4     NaN  ...             Forest fire near La Ronge Sask. Canada      1\n",
              "2   5     NaN  ...  All residents asked to 'shelter in place' are ...      1\n",
              "3   6     NaN  ...  13,000 people receive #wildfires evacuation or...      1\n",
              "4   7     NaN  ...  Just got sent this photo from Ruby #Alaska as ...      1\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E11sJwO-Xjd5"
      },
      "source": [
        "\n",
        "The training data we downloaded is probably shuffled already. But just to be sure, let's shuffle it again."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "4hZHEAX8Xct-",
        "outputId": "6a740524-0347-4cfc-c339-baf5f26a6dce"
      },
      "source": [
        "# Shuffle the training data\n",
        "train_df_shuffled = train_df.sample(frac=1, random_state=42)# shuffle with random_state=42 for reproducibility\n",
        "train_df_shuffled.head()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2644</th>\n",
              "      <td>3796</td>\n",
              "      <td>destruction</td>\n",
              "      <td>NaN</td>\n",
              "      <td>So you have a new weapon that can cause un-ima...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2227</th>\n",
              "      <td>3185</td>\n",
              "      <td>deluge</td>\n",
              "      <td>NaN</td>\n",
              "      <td>The f$&amp;amp;@ing things I do for #GISHWHES Just...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5448</th>\n",
              "      <td>7769</td>\n",
              "      <td>police</td>\n",
              "      <td>UK</td>\n",
              "      <td>DT @georgegalloway: RT @Galloway4Mayor: ÛÏThe...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>132</th>\n",
              "      <td>191</td>\n",
              "      <td>aftershock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Aftershock back to school kick off was great. ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6845</th>\n",
              "      <td>9810</td>\n",
              "      <td>trauma</td>\n",
              "      <td>Montgomery County, MD</td>\n",
              "      <td>in response to trauma Children of Addicts deve...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id  ... target\n",
              "2644  3796  ...      1\n",
              "2227  3185  ...      0\n",
              "5448  7769  ...      1\n",
              "132    191  ...      0\n",
              "6845  9810  ...      0\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "HKG4dgPaXvLh",
        "outputId": "35f0abd2-de53-437f-ce96-9e4d6f104e6b"
      },
      "source": [
        "# The test data doesn't have a target (that's what we'd try to predict)\n",
        "test_df.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   id keyword location                                               text\n",
              "0   0     NaN      NaN                 Just happened a terrible car crash\n",
              "1   2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2   3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3   9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4  11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7UMzgDGtYWGp",
        "outputId": "5fb5a322-150d-4c54-aba4-9f8de7cdcb94"
      },
      "source": [
        "# How many examples of each class?\n",
        "train_df[\"target\"].value_counts()"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    4342\n",
              "1    3271\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IBY85JOOYwDC"
      },
      "source": [
        "\n",
        "Since we have two target values, we're dealing with a **binary classification** problem.\n",
        "\n",
        "It's fairly balanced too, about 60% negative class (target = 0) and 40% positive class (target = 1).\n",
        "\n",
        "Where,\n",
        "\n",
        "- 1 = a real disaster Tweet\n",
        "- 0 = not a real disaster Tweet\n",
        "\n",
        "And what about the total number of samples we have?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1tP_JclFYemh",
        "outputId": "2acc7878-4256-41b5-87ee-c34a8baefe67"
      },
      "source": [
        "# How many samples total?\n",
        "print(f\"Total training samples:- {len(train_df)}\")\n",
        "print(f\"Total test samples:- {len(test_df)}\")\n",
        "print(f\"Total samples:- {len(train_df) + len(test_df)}\")"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total training samples:- 7613\n",
            "Total test samples:- 3263\n",
            "Total samples:- 10876\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lE0onfLtZmsP"
      },
      "source": [
        "Alright, seems like we've got a decent amount of training and test data. If anything, we've got an abundance of testing examples, usually a split of 90/10 (90% training, 10% testing) or 80/20 is suffice.\n",
        "\n",
        "Okay, time to visualize, let's write some code to visualize random text samples.\n",
        "\n",
        "> 🤔 Question: Why visualize random samples? You could visualize samples in order but this could lead to only seeing a certain subset of data. Better to visualize a substantial quantity (100+) of random samples to get an idea of the different kinds of data you're working with. In machine learning, never underestimate the power of randomness.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XDXH1dLCZOys",
        "outputId": "e295bc37-8c0b-464e-e06b-bdf3266f7c77"
      },
      "source": [
        "# Let's visualize some random trainning examples\n",
        "import random\n",
        "random_index = random.randint(0, len(train_df)-5)# create random indexes not higher than the total number of samples\n",
        "for row in train_df_shuffled[[\"text\", \"target\"]][random_index:random_index+5].itertuples():\n",
        "  _, text, target = row\n",
        "  print(f\"Target : {target}\", \"(real disaster)\" if target > 0 else \"(not real disaster)\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"---\\n\")"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target : 1 (real disaster)\n",
            "Text:\n",
            "28 Oct 1895: 'Bush Fires.' http://t.co/zCKXtFc9PT\n",
            "\n",
            "---\n",
            "\n",
            "Target : 1 (real disaster)\n",
            "Text:\n",
            "@ScotRail i be seen them turn a blind eye to a bloke drinking and smoking on during the curfew time cos'its not worth the hassle'\n",
            "\n",
            "---\n",
            "\n",
            "Target : 0 (not real disaster)\n",
            "Text:\n",
            "sorry but that mousse inundation was fuckin hilarious https://t.co/tUai0ZwGXU\n",
            "\n",
            "---\n",
            "\n",
            "Target : 0 (not real disaster)\n",
            "Text:\n",
            "The Great Upheaval By Winik Jay http://t.co/Ef4swP9SXZ http://t.co/Nb7MAAAOfs\n",
            "\n",
            "---\n",
            "\n",
            "Target : 1 (real disaster)\n",
            "Text:\n",
            "Former heroin addict shares story as city leaders sound alarm: City officials said emergency teams responded t... http://t.co/GZxIPMOknB\n",
            "\n",
            "---\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4qwZ1EKbluM"
      },
      "source": [
        "## Split data into training and validation sets\n",
        "Since the test set has no labels and we need a way to evalaute our trained models, we'll split off some of the training data and create a validation set.\n",
        "\n",
        "When our model trains (tries patterns in the Tweet samples), it'll only see data from the training set and we can see how it performs on unseen data using the validation set.\n",
        "\n",
        "We'll convert our splits from pandas Series datatypes to lists of strings (for the text) and lists of ints (for the labels) for ease of use later.\n",
        "\n",
        "To split our training dataset and create a validation dataset, we'll use Scikit-Learn's train_test_split() method and dedicate 10% of the training samples to the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wBlyEQtYZ-wD"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "\n",
        "train_sentences, val_sentences, train_labels, val_labels = train_test_split(train_df_shuffled[\"text\"].to_numpy(), train_df_shuffled[\"target\"].to_numpy(), test_size = 0.1, random_state=42)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4NhFq4bxcJn7",
        "outputId": "fdc30579-d949-443f-fec7-2ca211fb7e03"
      },
      "source": [
        "# Check the lengths\n",
        "len(train_sentences), len(train_labels), len(val_sentences), len(val_labels)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(6851, 6851, 762, 762)"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ufYNjjYJcLvs",
        "outputId": "5908cb47-876f-4eae-9fba-f91fd269b783"
      },
      "source": [
        "# View the first 10 training sentences and their labels\n",
        "train_sentences[:10], train_labels[:10]"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array(['@mogacola @zamtriossu i screamed after hitting tweet',\n",
              "        'Imagine getting flattened by Kurt Zouma',\n",
              "        '@Gurmeetramrahim #MSGDoing111WelfareWorks Green S welfare force ke appx 65000 members har time disaster victim ki help ke liye tyar hai....',\n",
              "        \"@shakjn @C7 @Magnums im shaking in fear he's gonna hack the planet\",\n",
              "        'Somehow find you and I collide http://t.co/Ee8RpOahPk',\n",
              "        '@EvaHanderek @MarleyKnysh great times until the bus driver held us hostage in the mall parking lot lmfao',\n",
              "        'destroy the free fandom honestly',\n",
              "        'Weapons stolen from National Guard Armory in New Albany still missing #Gunsense http://t.co/lKNU8902JE',\n",
              "        '@wfaaweather Pete when will the heat wave pass? Is it really going to be mid month? Frisco Boy Scouts have a canoe trip in Okla.',\n",
              "        'Patient-reported outcomes in long-term survivors of metastatic colorectal cancer - British Journal of Surgery http://t.co/5Yl4DC1Tqt'],\n",
              "       dtype=object), array([0, 0, 1, 0, 0, 1, 1, 0, 1, 1]))"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eyaj-9_Wd2do"
      },
      "source": [
        "## Converting text into numbers\n",
        "Wonderful! We've got a training set and a validation set containing Tweets and labels.\n",
        "\n",
        "Our labels are in numerical form (0 and 1) but our Tweets are in string form.\n",
        "\n",
        "> 🤔 Question: What do you think we have to do before we can use a machine learning algorithm with our text data?\n",
        "\n",
        "If you answered something along the lines of \"turn it into numbers\", you're correct. A machine learning algorithm requires its inputs to be in numerical form.\n",
        "\n",
        "In NLP, there are two main concepts for turning text into numbers:\n",
        "\n",
        "- Tokenization - A straight mapping from word or character or sub-word to a numerical value. There are three main levels of tokenization:\n",
        "  - Using word-level tokenization with the sentence \"I love TensorFlow\" might result in \"I\" being 0, \"love\" being 1 and \"TensorFlow\" being 2. In this case, every word in a sequence considered a single token.\n",
        "  - Character-level tokenization, such as converting the letters A-Z to values 1-26. In this case, every character in a sequence considered a single token.\n",
        "  - Sub-word tokenization is in between word-level and character-level tokenization. It involves breaking invidual words into smaller parts and then converting those smaller parts into numbers. For example, \"my favourite food is pineapple pizza\" might become \"my, fav, avour, rite, fo, oo, od, is, pin, ine, app, le, piz, za\". After doing this, these sub-words would then be mapped to a numerical value. In this case, every word could be considered multiple tokens.\n",
        "- Embeddings - An embedding is a representation of natural language which can be learned. Representation comes in the form of a feature vector. For example, the word \"dance\" could be represented by the 5-dimensional vector [-0.8547, 0.4559, -0.3332, 0.9877, 0.1112]. It's important to note here, the size of the feature vector is tuneable. There are two ways to use embeddings:\n",
        "  - Create your own embedding - Once your text has been turned into numbers (required for an embedding), you can put them through an embedding layer (such as tf.keras.layers.Embedding) and an embedding representation will be learned during model training.\n",
        "  - Reuse a pre-learned embedding - Many pre-trained embeddings exist online. These pre-trained embeddings have often been learned on large corpuses of text (such as all of Wikipedia) and thus have a good underlying representation of natural language. You can use a pre-trained embedding to initialize your model and fine-tune it to your own specific task."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eqt4nV7leZGo"
      },
      "source": [
        "> 🤔 Question: What level of tokenzation should I use? What embedding should should I choose?\n",
        "\n",
        "It depends on your problem. You could try character-level tokenization/embeddings and word-level tokenization/embeddings and see which perform best. You might even want to try stacking them (e.g. combining the outputs of your embedding layers using tf.keras.layers.concatenate).\n",
        "\n",
        "If you're looking for pre-trained word embeddings, Word2vec embeddings, GloVe embeddings and many of the options available on TensorFlow Hub are great places to start.\n",
        "\n",
        "> 🔑 Note: Much like searching for a pre-trained computer vision model, you can search for pre-trained word embeddings to use for your problem. Try searching for something like \"use pre-trained word embeddings in TensorFlow\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tKQ64agZe4RY"
      },
      "source": [
        "## Text vectorization (tokenization)\n",
        "Enough talking about tokenization and embeddings, let's create some.\n",
        "\n",
        "We'll practice tokenzation (mapping our words to numbers) first.\n",
        "\n",
        "To tokenize our words, we'll use the helpful preprocessing layer tf.keras.layers.experimental.preprocessing.TextVectorization.\n",
        "\n",
        "The TextVectorization layer takes the following parameters:\n",
        "\n",
        "- max_tokens - The maximum number of words in your vocabulary (e.g. 20000 or the number of unique words in your text), includes a value for OOV (out of vocabulary) tokens.\n",
        "- standardize - Method for standardizing text. Default is \"lower_and_strip_punctuation\" which lowers text and removes all punctuation marks.\n",
        "- split - How to split text, default is \"whitespace\" which splits on spaces.\n",
        "- ngrams - How many words to contain per token split, for example, ngrams=2 splits tokens into continuous sequences of 2.\n",
        "- output_mode - How to output tokens, can be \"int\" (integer mapping), \"binary\" (one-hot encoding), \"count\" or \"tf-idf\". See documentation for more.\n",
        "- output_sequence_length - Length of tokenized sequence to output. For example, if output_sequence_length=150, all tokenized sequences will be 150 tokens long.\n",
        "- pad_to_max_tokens - Defaults to False, if True, the output feature axis will be padded to max_tokens even if the number of unique tokens in the vocabulary is less than max_tokens. Only valid in certain modes, see docs for more.\n",
        "\n",
        "Let's see it in action."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SI4ozfWecYZ_"
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "# Note: in TensorFlow 2.6+, you no longer need \"layers.experimental.preprocessing\"\n",
        "# you can use: \"tf.keras.layers.TextVectorization\", see https://github.com/tensorflow/tensorflow/releases/tag/v2.6.0 for more\n",
        "\n",
        "# Use the default TextVectorization variables\n",
        "text_vectorizer = TextVectorization(max_tokens=None, # how many words in the vocabulary (all of the different words in your text)\n",
        "                                    standardize=\"lower_and_strip_punctuation\", # how to process text\n",
        "                                    split=\"whitespace\", # how to split tokens\n",
        "                                    ngrams=None, # create groups of n-words?\n",
        "                                    output_mode=\"int\", # how to map tokens to numbers\n",
        "                                    output_sequence_length=None) # how long should the output sequence of tokens be?\n",
        "                                    # pad_to_max_tokens=True) # Not valid if using max_tokens=None"
      ],
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xuEkhbI4i363"
      },
      "source": [
        "\n",
        "We've initialized a TextVectorization object with the default settings but let's customize it a little bit for our own use case.\n",
        "\n",
        "In particular, let's set values for max_tokens and output_sequence_length.\n",
        "\n",
        "For max_tokens (the number of words in the vocabulary), multiples of 10,000 (10,000, 20,000, 30,000) or the exact number of unique words in your text (e.g. 32,179) are common values.\n",
        "\n",
        "For our use case, we'll use 10,000.\n",
        "\n",
        "And for the output_sequence_length we'll use the average number of tokens per Tweet in the training set. But first, we'll need to find it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qTtlkRwzikhB",
        "outputId": "2205dc3c-50c6-4505-ddbf-523c3e9323ae"
      },
      "source": [
        "# Find average number of tokens (words) in training Tweets\n",
        "round(sum([len(i.split()) for i in train_sentences])/len(train_sentences))"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yi_5XhuSjNPV"
      },
      "source": [
        "\n",
        "Now let's create another TextVectorization object using our custom parameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5iTbpc_CjJju"
      },
      "source": [
        "# Setup text vectorization with custom variables\n",
        "max_vocab_length = 10000 # max number of words to have in our vocabulary\n",
        "max_length = 15 # max length our sequences will be (e.g. how many words from a Tweet does our model see?)\n",
        "\n",
        "text_vectorizer = TextVectorization(max_tokens=max_vocab_length,\n",
        "                                    output_mode=\"int\",\n",
        "                                    output_sequence_length=max_length)"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ePpLEUbjW_k"
      },
      "source": [
        "\n",
        "Beautiful!\n",
        "\n",
        "To map our TextVectorization instance text_vectorizer to our data, we can call the adapt() method on it whilst passing it our training text."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g0SDB_1ojRiA"
      },
      "source": [
        "# Fit the text vectorizer to the training data\n",
        "text_vectorizer.adapt(train_sentences)"
      ],
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_MdBpRcVjpM4"
      },
      "source": [
        "\n",
        "Training data mapped! Let's try our text_vectorizer on a custom sentence (one similar to what you might see in the training data).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8VaFeRHfjjpi",
        "outputId": "0628c3c6-404c-4da4-9c15-370f8e364020"
      },
      "source": [
        "# Create sample sentence and tokenize it\n",
        "sample_sentence = \"There's a flood in my street!\"\n",
        "text_vectorizer([sample_sentence])"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[264,   3, 232,   4,  13, 698,   0,   0,   0,   0,   0,   0,   0,\n",
              "          0,   0]])>"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "owWOuAGEkHmj"
      },
      "source": [
        "\n",
        "Wonderful, it seems we've got a way to turn our text into numbers (in this case, word-level tokenization). Notice the 0's at the end of the returned tensor, this is because we set output_sequence_length=15, meaning no matter the size of the sequence we pass to text_vectorizer, it always returns a sequence with a length of 15.\n",
        "\n",
        "How about we try our text_vectorizer on a few random sentences?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5gEba6rCj27T",
        "outputId": "97191c55-4dd2-4e59-e8fb-6a2858034e04"
      },
      "source": [
        "# Choose a random sentence from the training dataset and tokenize it\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "      \\n\\nVectorized version:\")\n",
        "text_vectorizer([random_sentence])"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text:\n",
            "Posted a new song: 'Earthquake' http://t.co/RfTyyZ4GwJ http://t.co/lau0Ay7ahV      \n",
            "\n",
            "Vectorized version:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15), dtype=int64, numpy=\n",
              "array([[2794,    3,   50,  528,  290,    1,    1,    0,    0,    0,    0,\n",
              "           0,    0,    0,    0]])>"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O3fnZfIGkRCy"
      },
      "source": [
        "\n",
        "Looking good!\n",
        "\n",
        "Finally, we can check the unique tokens in our vocabulary using the get_vocabulary() method."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XUW43_HTkLL_",
        "outputId": "a452a78e-f4df-4629-cd6e-0c16fa94f57a"
      },
      "source": [
        "# Get the unique words in the vocabulary\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "top_5_words = words_in_vocab[:5] # most common tokens (notice the [UNK] token for \"unknown\" words)\n",
        "bottom_5_words = words_in_vocab[-5:] # least common tokens\n",
        "print(f\"Number of words in vocab: {len(words_in_vocab)}\")\n",
        "print(f\"Top 5 most common words: {top_5_words}\") \n",
        "print(f\"Bottom 5 least common words: {bottom_5_words}\")"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Number of words in vocab: 10000\n",
            "Top 5 most common words: ['', '[UNK]', 'the', 'a', 'in']\n",
            "Bottom 5 least common words: ['pages', 'paeds', 'pads', 'padres', 'paddytomlinson1']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TefBqZtvlOPB"
      },
      "source": [
        "## Creating an Embedding using an Embedding Layer\n",
        "We've got a way to map our text to numbers. How about we go a step further and turn those numbers into an embedding?\n",
        "\n",
        "The powerful thing about an embedding is it can be learned during training. This means rather than just being static (e.g. 1 = I, 2 = love, 3 = TensorFlow), a word's numeric representation can be improved as a model goes through data samples.\n",
        "\n",
        "We can see what an embedding of a word looks like by using the tf.keras.layers.Embedding layer.\n",
        "\n",
        "The main parameters we're concerned about here are:\n",
        "\n",
        "- input_dim - The size of the vocabulary (e.g. len(text_vectorizer.get_vocabulary()).\n",
        "- output_dim - The size of the output embedding vector, for example, a value of 100 outputs a feature vector of size 100 for each word.\n",
        "- embeddings_initializer - How to initialize the embeddings matrix, default is \"uniform\" which randomly initalizes embedding matrix with uniform distribution. This can be changed for using pre-learned embeddings.\n",
        "- input_length - Length of sequences being passed to embedding layer.\n",
        "\n",
        "Knowing these, let's make an embedding layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ra-VgGqkkq1Y",
        "outputId": "67c5a065-5f8c-47e8-d425-7b1ba7f59cb3"
      },
      "source": [
        "from tensorflow.keras import layers\n",
        "\n",
        "embedding = layers.Embedding(input_dim=max_vocab_length, # set input shape\n",
        "                             output_dim=128, # set size of embedding vector\n",
        "                             embeddings_initializer=\"uniform\", # default, intialize randomly\n",
        "                             input_length=max_length) # how long is each input\n",
        "\n",
        "embedding"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.layers.embeddings.Embedding at 0x7f55c0368550>"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QEjzK2zAlhb5"
      },
      "source": [
        "Excellent, notice how embedding is a TensoFlow layer? This is important because we can use it as part of a model, meaning its parameters (word representations) can be updated and improved as the model learns.\n",
        "\n",
        "How about we try it out on a sample sentence?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1cFtnsculai_",
        "outputId": "4ef9cd71-1918-42dc-93e5-0ae84258fd52"
      },
      "source": [
        "# Get a random sentence from training set\n",
        "random_sentence = random.choice(train_sentences)\n",
        "print(f\"Original text:\\n{random_sentence}\\\n",
        "      \\n\\nEmbedded version:\")\n",
        "\n",
        "# Embed the random sentence (turn it into numerical representation)\n",
        "sample_embed = embedding(text_vectorizer([random_sentence]))\n",
        "sample_embed"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original text:\n",
            "I liked a @YouTube video http://t.co/jK7nPdpWRo J. Cole - Fire Squad (2014 Forest Hills Drive)      \n",
            "\n",
            "Embedded version:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              "array([[[-0.00716193, -0.02650405,  0.02041295, ..., -0.04058369,\n",
              "         -0.03402247,  0.02494038],\n",
              "        [ 0.00496794,  0.00242374,  0.02006559, ..., -0.04879434,\n",
              "          0.01200615,  0.04201773],\n",
              "        [ 0.00677536, -0.03712019,  0.03888628, ...,  0.03081367,\n",
              "          0.01221542,  0.01712577],\n",
              "        ...,\n",
              "        [ 0.00417843, -0.03483133, -0.00729876, ..., -0.04269493,\n",
              "         -0.0411445 , -0.01571548],\n",
              "        [-0.0361839 , -0.03684834, -0.0428664 , ..., -0.02726002,\n",
              "          0.03933952,  0.03323427],\n",
              "        [ 0.03563151,  0.0395606 , -0.0499089 , ..., -0.04685975,\n",
              "          0.03444735, -0.0488968 ]]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yEVbSrZ6l2O3"
      },
      "source": [
        "\n",
        "Each token in the sentence gets turned into a length 128 feature vector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c_tvvnnRloEz",
        "outputId": "ac84d982-5885-431c-d656-e982b74cc149"
      },
      "source": [
        "# check out a single token's embedding\n",
        "sample_embed[0][0]"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(128,), dtype=float32, numpy=\n",
              "array([-0.00716193, -0.02650405,  0.02041295,  0.0047984 ,  0.03575281,\n",
              "        0.04289151, -0.0409149 , -0.0387967 , -0.00757719,  0.04998019,\n",
              "       -0.00564221, -0.02587198, -0.00291819, -0.02956009,  0.03884791,\n",
              "        0.04477042, -0.03016808, -0.01663459, -0.03669066,  0.00379875,\n",
              "        0.00751417, -0.0314176 , -0.03247337, -0.02020364,  0.01714686,\n",
              "       -0.04189996, -0.03886171,  0.00877973,  0.04752528,  0.02405348,\n",
              "        0.01685539,  0.00320488,  0.03936727,  0.03105387,  0.00040805,\n",
              "        0.02107142,  0.03653432,  0.04339736, -0.03586557, -0.00958591,\n",
              "        0.00090111, -0.01565786, -0.02416469, -0.04010974,  0.04627817,\n",
              "       -0.02180747,  0.02876098,  0.01719259,  0.02061646,  0.00611862,\n",
              "       -0.00021799,  0.01273168,  0.04731908, -0.04739612,  0.02211263,\n",
              "        0.03974586,  0.01973727, -0.01564374,  0.03678901,  0.0102218 ,\n",
              "       -0.02923754, -0.03018866, -0.0421363 ,  0.02533856, -0.03025615,\n",
              "        0.02924805,  0.03220591, -0.01121243, -0.04294541,  0.03968605,\n",
              "        0.03194788, -0.0453007 , -0.03506307, -0.03246828,  0.02583135,\n",
              "       -0.01725994,  0.0479677 , -0.02735629,  0.01193266,  0.04370179,\n",
              "       -0.04231239,  0.03536825,  0.04193281,  0.00251198,  0.04600587,\n",
              "        0.03677816,  0.00624425, -0.00592124,  0.00513113,  0.02432207,\n",
              "        0.00942392, -0.03733387, -0.04678851,  0.00206884, -0.04967065,\n",
              "       -0.03421028, -0.01664829, -0.03129921, -0.01655629, -0.04201989,\n",
              "        0.02094724,  0.00910398, -0.00463371, -0.02220262,  0.03316534,\n",
              "       -0.03014826, -0.01752721, -0.00432205,  0.04179979,  0.00624203,\n",
              "        0.01467681,  0.01885703, -0.01215645, -0.00111375, -0.04864636,\n",
              "        0.01980004,  0.01424739, -0.01581484,  0.04764793,  0.03759414,\n",
              "       -0.01533209, -0.002695  ,  0.00354955,  0.01185212,  0.0356787 ,\n",
              "       -0.04058369, -0.03402247,  0.02494038], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CS_UCVZ3mJa0"
      },
      "source": [
        "These values might not mean much to us but they're what our computer sees each word as. When our model looks for patterns in different samples, these values will be updated as necessary.\n",
        "\n",
        "> 🔑 Note: The previous two concepts (tokenization and embeddings) are the foundation for many NLP tasks. So if you're not sure about anything, be sure to research and conduct your own experiments to further help your understanding."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iFob4VN1mcn6"
      },
      "source": [
        "## Modelling a text dataset\n",
        "\n",
        "Now that we've got a way to turn our text data into numbers, we can start to build machine learning models to model it.\n",
        "\n",
        "To get plenty of practice, we're going to build a series of different models, each as its own experiment. We'll then compare the results of each model and see which one performed best.\n",
        "\n",
        "More specifically, we'll be building the following:\n",
        "\n",
        "- Model 0: Naive Bayes (baseline)\n",
        "- Model 1: Feed-forward neural network (dense model)\n",
        "- Model 2: LSTM model\n",
        "- Model 3: GRU model\n",
        "- Model 4: Bidirectional-LSTM model\n",
        "- Model 5: 1D Convolutional Neural Network\n",
        "- Model 6: TensorFlow Hub Pretrained Feature Extractor\n",
        "- Model 7: Same as model 6 with 10% of training data\n",
        "\n",
        "Model 0 is the simplest to acquire a baseline which we'll expect each other of the other deeper models to beat.\n",
        "\n",
        "Each experiment will go through the following steps:\n",
        "\n",
        "- Construct the model\n",
        "- Train the model\n",
        "- Make predictions with the model\n",
        "- Track prediction evaluation metrics for later comparison\n",
        "\n",
        "Let's get started"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0mzKiwLg2Z60"
      },
      "source": [
        "\n",
        "## Model 0: Getting a baseline\n",
        "As with all machine learning modelling experiments, it's important to create a baseline model so you've got a benchmark for future experiments to build upon.\n",
        "\n",
        "To create our baseline, we'll create a Scikit-Learn Pipeline using the TF-IDF (term frequency-inverse document frequency) formula to convert our words to numbers and then model them with the Multinomial Naive Bayes algorithm. This was chosen via referring to the Scikit-Learn machine learning map.\n",
        "\n",
        "> 📖 Reading: The ins and outs of TF-IDF algorithm is beyond the scope of this notebook, however, the curious reader is encouraged to check out the Scikit-Learn documentation for more."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9AORoRrcl8Sp",
        "outputId": "0bfc956d-a2fe-4f13-8996-10964b06f756"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "model_0 = Pipeline([\n",
        "                    (\"tfidf\", TfidfVectorizer()), # convert words to numbers using tfidf\n",
        "                    (\"clf\", MultinomialNB()) # model the text\n",
        "])\n",
        "\n",
        "# Fit the pipeline to the training data\n",
        "model_0.fit(train_sentences, train_labels)"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Pipeline(memory=None,\n",
              "         steps=[('tfidf',\n",
              "                 TfidfVectorizer(analyzer='word', binary=False,\n",
              "                                 decode_error='strict',\n",
              "                                 dtype=<class 'numpy.float64'>,\n",
              "                                 encoding='utf-8', input='content',\n",
              "                                 lowercase=True, max_df=1.0, max_features=None,\n",
              "                                 min_df=1, ngram_range=(1, 1), norm='l2',\n",
              "                                 preprocessor=None, smooth_idf=True,\n",
              "                                 stop_words=None, strip_accents=None,\n",
              "                                 sublinear_tf=False,\n",
              "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
              "                                 tokenizer=None, use_idf=True,\n",
              "                                 vocabulary=None)),\n",
              "                ('clf',\n",
              "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
              "         verbose=False)"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YG7jOLIR3W7a"
      },
      "source": [
        "\n",
        "The benefit of using a shallow model like Multinomial Naive Bayes is that training is very fast.\n",
        "\n",
        "Let's evaluate our model and find our baseline metric."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZQ4lEQhT3QYS",
        "outputId": "f7a6e39e-f1b7-4b2e-8fa8-fb2e12a5703d"
      },
      "source": [
        "baseline_score = model_0.score(val_sentences, val_labels)\n",
        "print(f\"Our baseline model achieves an accuracy of: {baseline_score*100:.2f}%\")"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Our baseline model achieves an accuracy of: 79.27%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qKLDmE5t3n-x",
        "outputId": "ef2d8c6b-fa44-4eed-d938-28b3775a6de5"
      },
      "source": [
        "# Make predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds[:10]"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 1, 1, 0, 0, 1, 1, 1, 1, 0])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qkmiZggk35tx"
      },
      "source": [
        "## Creating an evaluation function for our model experiments\n",
        "We could evaluate these as they are but since we're going to be evaluating several models in the same way going forward, let's create a helper function which takes an array of predictions and ground truth labels and computes the following:\n",
        "\n",
        "- Accuracy\n",
        "- Precision\n",
        "- Recall\n",
        "- F1-score\n",
        "\n",
        "> 🔑 Note: Since we're dealing with a classification problem, the above metrics are the most appropriate. If we were working with a regression problem, other metrics such as MAE (mean absolute error) would be a better choice."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RkUGMtTC30O_"
      },
      "source": [
        "# Function to evaluate : accuracy, precision, recall, f1_score\n",
        "from sklearn.metrics import accuracy_score, precision_recall_fscore_support\n",
        "\n",
        "def calculate_results(y_true, y_pred):\n",
        "  \"\"\"\n",
        "  Calculates model accuracy, precision, recall and f1 score of a binary classification model.\n",
        "\n",
        "  Args:\n",
        "  -----\n",
        "  y_true = true labels in the form of a 1D array\n",
        "  y_pred = predicted labels in the form of a 1D array\n",
        "\n",
        "  Returns a dictionary of accuracy, precision, recall, f1-score.\n",
        "  \"\"\"\n",
        "  model_accuracy = accuracy_score(y_true, y_pred) * 100\n",
        "\n",
        "  model_precision, model_recall, model_f1, _ = precision_recall_fscore_support(y_true, y_pred, average=\"weighted\")\n",
        "  model_results = {\"accuracy\": model_accuracy,\n",
        "                   \"precision\": model_precision,\n",
        "                   \"recall\": model_recall,\n",
        "                   \"f1_score\": model_f1}\n",
        "\n",
        "  return model_results\n"
      ],
      "execution_count": 26,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3mlUqOnp5Cnp",
        "outputId": "e471b42d-58a5-448a-d468-bfabd9cb1b86"
      },
      "source": [
        "# Get baseline results\n",
        "baseline_results = calculate_results(val_labels, baseline_preds)\n",
        "baseline_results"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.26509186351706,\n",
              " 'f1_score': 0.7862189758049549,\n",
              " 'precision': 0.8111390004213173,\n",
              " 'recall': 0.7926509186351706}"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNagLzAS5nOJ"
      },
      "source": [
        "## Model 1: A simple dense model\n",
        "The first \"deep\" model we're going to build is a single layer dense model. In fact, it's barely going to have a single layer.\n",
        "\n",
        "It'll take our text and labels as input, tokenize the text, create an embedding, find the average of the embedding (using Global Average Pooling) and then pass the average through a fully connected layer with one output unit and a sigmoid activation function.\n",
        "\n",
        "If the previous sentence sounds like a mouthful, it'll make sense when we code it out (remember, if in doubt, code it out).\n",
        "\n",
        "And since we're going to be building a number of TensorFlow deep learning models, we'll import our create_tensorboard_callback() function from helper_functions.py to keep track of the results of each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gzumx4JA5P8a"
      },
      "source": [
        "# Create tensorboard callback (need to create a a new one for each model)\n",
        "from helper_functions import create_tensorboard_callback\n",
        "\n",
        "# Create directory to save tensorboard logs\n",
        "SAVE_DIR = \"model_logs\""
      ],
      "execution_count": 28,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7GNuSlyX5795"
      },
      "source": [
        "Now we've got a TensorBoard callback function ready to go, let's build our first deep model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BwlNiEiw56aQ"
      },
      "source": [
        "# Build model with functional API\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype = \"string\") # inputs are 1-dimensional strings\n",
        "x = text_vectorizer(inputs) # turn the input text into number\n",
        "x = embedding(x) # Create an embedding of the numerized numbers\n",
        "x = layers.GlobalAveragePooling1D()(x) # lower the dimensionality of the embedding (try running the model without this layer and see what happens)\n",
        "\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x) # create the output layer, want binary outputs so use sigmoid activation\n",
        "model_1 = tf.keras.Model(inputs, outputs, name=\"model_1_dense\")"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GYy4RQjC7Bs6"
      },
      "source": [
        "\n",
        "Looking good. Our model takes a 1-dimensional string as input (in our case, a Tweet), it then tokenizes the string using text_vectorizer and creates an embedding using embedding.\n",
        "\n",
        "We then (optionally) pool the outputs of the embedding layer to reduce the dimensionality of the tensor we pass to the output layer.\n",
        "\n",
        "> 🛠 Exercise: Try building model_1 with and without a GlobalAveragePooling1D() layer after the embedding layer. What happens? Why do you think this is?\n",
        "\n",
        "Finally, we pass the output of the pooling layer to a dense layer with sigmoid activation (we use sigmoid since our problem is binary classification).\n",
        "\n",
        "Before we can fit our model to the data, we've got to compile it. Since we're working with binary classification, we'll use \"binary_crossentropy\" as our loss function and the Adam optimizer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kvXo3gI36zKf"
      },
      "source": [
        "# Compile model\n",
        "model_1.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GXq49Lf17P8G",
        "outputId": "3c72d354-551d-467f-c21c-fc79eb4d9388"
      },
      "source": [
        "model_1.summary()"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xqivYXi47h8P"
      },
      "source": [
        "Most of the trainable parameters are contained within the embedding layer. Recall we created an embedding of size 128 (output_dim=128) for a vocabulary of size 10,000 (input_dim=10000), hence the 1,280,000 trainable parameters.\n",
        "\n",
        "Alright, our model is compiled, let's fit it to our training data for 5 epochs. We'll also pass our TensorBoard callback function to make sure our model's training metrics are logged.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UjoBx6pF7Sxm",
        "outputId": "cb614cca-ec11-4e48-d212-eac907755b3b"
      },
      "source": [
        "# Fit the model\n",
        "model_1_history = model_1.fit(train_sentences, # input sentences can be a list of strings due to text preprocessing layer built-in model\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(dir_name=SAVE_DIR, \n",
        "                                                                     experiment_name=\"simple_dense_model\")])"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/simple_dense_model/20210923-070742\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 4s 13ms/step - loss: 0.6097 - accuracy: 0.6981 - val_loss: 0.5349 - val_accuracy: 0.7520\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 11ms/step - loss: 0.4414 - accuracy: 0.8190 - val_loss: 0.4726 - val_accuracy: 0.7822\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 12ms/step - loss: 0.3462 - accuracy: 0.8602 - val_loss: 0.4563 - val_accuracy: 0.7966\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 12ms/step - loss: 0.2839 - accuracy: 0.8902 - val_loss: 0.4631 - val_accuracy: 0.7913\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 11ms/step - loss: 0.2373 - accuracy: 0.9114 - val_loss: 0.4812 - val_accuracy: 0.7835\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k7AtW69R7ywX"
      },
      "source": [
        "Nice! Since we're using such a simple model, each epoch processes very quickly.\n",
        "\n",
        "Let's check our model's performance on the validation set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g0oxNq207luc",
        "outputId": "04f1a27e-8da6-4cbc-b3e2-785fd8d792e4"
      },
      "source": [
        "# Check the results\n",
        "model_1.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 0s 2ms/step - loss: 0.4812 - accuracy: 0.7835\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.48121127486228943, 0.7834645509719849]"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZobhKvG8E0S"
      },
      "source": [
        "And since we tracked our model's training logs with TensorBoard, how about we visualize them?\n",
        "\n",
        "We can do so by uploading our TensorBoard log files (contained in the model_logs directory) to TensorBoard.dev.\n",
        "\n",
        "> 🔑 Note: Remember, whatever you upload to TensorBoard.dev becomes public. If there are training logs you don't want to share, don't upload them."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jxq59fGW8Xkm"
      },
      "source": [
        "We've built and trained our first deep model, the next step is to make some predictions with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "upJPwyTZ76qN",
        "outputId": "3743633e-6e7d-44cc-f3e1-38c1cfc354e8"
      },
      "source": [
        "# Make predictions\n",
        "model_1_pred_prbs = model_1.predict(val_sentences)\n",
        "model_1_pred_prbs[:10]"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.33054727],\n",
              "       [0.6935922 ],\n",
              "       [0.9979266 ],\n",
              "       [0.13741195],\n",
              "       [0.10586694],\n",
              "       [0.94053674],\n",
              "       [0.9100997 ],\n",
              "       [0.9922351 ],\n",
              "       [0.96177447],\n",
              "       [0.29720515]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ExBc2iHU8rP9"
      },
      "source": [
        "Since our final layer uses a sigmoid activation function, we get our predictions back in the form of probabilities.\n",
        "\n",
        "To convert them to prediction classes, we'll use tf.round(), meaning prediction probabilities below 0.5 will be rounded to 0 and those above 0.5 will be rounded to 1.\n",
        "\n",
        "> 🔑 Note: In practice, the output threshold of a sigmoid prediction probability doesn't necessarily have to 0.5. For example, through testing, you may find that a cut off of 0.25 is better for your chosen evaluation metrics. A common example of this threshold cutoff is the precision-recall tradeoff."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E_EswsfM8izk",
        "outputId": "ba44d41c-f432-4bc9-d434-30d1df6f6556"
      },
      "source": [
        "# Turn prediction probabilities into single-dimension tensor of floats\n",
        "model_1_preds = tf.squeeze(tf.round(model_1_pred_prbs))\n",
        "model_1_preds[:10]"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8w3vd2da8-9K"
      },
      "source": [
        "Now we've got our model's predictions in the form of classes, we can use our calculate_results() function to compare them to the ground truth validation labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AOqbfZPL9AbA",
        "outputId": "fdf2b57d-e245-41f2-cbf1-6063ddeceafa"
      },
      "source": [
        "# Calculate model_1 metrics\n",
        "model_1_results = calculate_results(val_labels, model_1_preds)\n",
        "model_1_results"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.34645669291339,\n",
              " 'f1_score': 0.780192419920975,\n",
              " 'precision': 0.7888731352165597,\n",
              " 'recall': 0.7834645669291339}"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YQKgGrfl9Pc0"
      },
      "source": [
        "\n",
        "How about we compare our first deep model to our baseline model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O2ms4kzk9Ipi",
        "outputId": "6a461486-a3b1-49e0-eadb-85c8e56a0f23"
      },
      "source": [
        "# Is our simple Keras model better than our baseline model?\n",
        "import numpy as np\n",
        "np.array(list(model_1_results.values())) > np.array(list(baseline_results.values()))"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([False, False, False, False])"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j65dzWm_9WFz"
      },
      "source": [
        "\n",
        "Since we'll be doing this kind of comparison (baseline compared to new model) quite a few times, let's create a function to help us out."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hvN6vrxh9R0Z",
        "outputId": "d6f42939-8173-428b-bfb1-54d7745a19c3"
      },
      "source": [
        "# Create a helper function to compare our baseline results to new model results\n",
        "def compare_baseline_to_new_results(baseline_results, new_model_results):\n",
        "  for key, value in baseline_results.items():\n",
        "    print(f\"Baseline {key}: {value:.2f}, New {key}: {new_model_results[key]:.2f}, Difference: {new_model_results[key]-value:.2f}\")\n",
        "\n",
        "compare_baseline_to_new_results(baseline_results=baseline_results, \n",
        "                                new_model_results=model_1_results)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 78.35, Difference: -0.92\n",
            "Baseline precision: 0.81, New precision: 0.79, Difference: -0.02\n",
            "Baseline recall: 0.79, New recall: 0.78, Difference: -0.01\n",
            "Baseline f1_score: 0.79, New f1_score: 0.78, Difference: -0.01\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YO516-3C-No5"
      },
      "source": [
        "## Visualizing learned embeddings\n",
        "Our first model (model_1) contained an embedding layer (embedding) which learned a way of representing words as feature vectors by passing over the training data.\n",
        "\n",
        "Hearing this for the first few times may sound confusing.\n",
        "\n",
        "So to further help understand what a text embedding is, let's visualize the embedding our model learned.\n",
        "\n",
        "To do so, let's remind ourselves of the words in our vocabulary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s4Z7LECM92ck",
        "outputId": "f8adf130-ef02-44b4-c53f-0d3f281221b1"
      },
      "source": [
        "# Get the vocabulary from the text vectorization layer\n",
        "words_in_vocab = text_vectorizer.get_vocabulary()\n",
        "len(words_in_vocab), words_in_vocab[:10]"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(10000, ['', '[UNK]', 'the', 'a', 'in', 'to', 'of', 'and', 'i', 'is'])"
            ]
          },
          "metadata": {},
          "execution_count": 39
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tJ62oQRD-qnm"
      },
      "source": [
        "And now let's get our embedding layer's weights (these are the numerical representations of each word)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "klYIkIJz-iMh",
        "outputId": "f8c86609-6b44-49b0-d815-c07465b997b0"
      },
      "source": [
        "model_1.summary()"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_1_dense\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 128)               0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,280,129\n",
            "Trainable params: 1,280,129\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OOCiymz_-smf",
        "outputId": "26fe473e-333a-4780-e05a-f77c980e676c"
      },
      "source": [
        "# Get the weight matrix of embedding layer \n",
        "# (these are the numerical patterns between the text in the training dataset the model has learned)\n",
        "embed_weights = model_1.get_layer(\"embedding\").get_weights()[0]\n",
        "print(embed_weights.shape) # same size as vocab size and embedding_dim (each word is a embedding_dim size vector)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(10000, 128)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vi_REaGx_dQ8"
      },
      "source": [
        "Now we've got these two objects, we can use the Embedding Projector tool to visualize our embedding.\n",
        "\n",
        "To use the Embedding Projector tool, we need two files:\n",
        "\n",
        "The embedding vectors (same as embedding weights).\n",
        "The meta data of the embedding vectors (the words they represent - our vocabulary).\n",
        "Right now, we've got of these files as Python objects. To download them to file, we're going to use the code example available on the TensorFlow word embeddings tutorial page."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I21wrLs5-2DU"
      },
      "source": [
        "# # Code below is adapted from: https://www.tensorflow.org/tutorials/text/word_embeddings#retrieve_the_trained_word_embeddings_and_save_them_to_disk\n",
        "# import io\n",
        "\n",
        "# # Create output writers\n",
        "# out_v = io.open(\"embedding_vectors.tsv\", \"w\", encoding=\"utf-8\")\n",
        "# out_m = io.open(\"embedding_metadata.tsv\", \"w\", encoding=\"utf-8\")\n",
        "\n",
        "# # Write embedding vectors and words to file\n",
        "# for num, word in enumerate(words_in_vocab):\n",
        "#   if num == 0: \n",
        "#      continue # skip padding token\n",
        "#   vec = embed_weights[num]\n",
        "#   out_m.write(word + \"\\n\") # write words to file\n",
        "#   out_v.write(\"\\t\".join([str(x) for x in vec]) + \"\\n\") # write corresponding word vector to file\n",
        "# out_v.close()\n",
        "# out_m.close()\n",
        "\n",
        "# # Download files locally to upload to Embedding Projector\n",
        "# try:\n",
        "#   from google.colab import files\n",
        "# except ImportError:\n",
        "#   pass\n",
        "# else:\n",
        "#   files.download(\"embedding_vectors.tsv\")\n",
        "#   files.download(\"embedding_metadata.tsv\")"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ttkUQfW1AzIL"
      },
      "source": [
        "Once you've downloaded the embedding vectors and metadata, you can visualize them using Embedding Vector tool:\n",
        "\n",
        "- Go to http://projector.tensorflow.org/\n",
        "- Click on \"Load data\"\n",
        "- Upload the two files you downloaded (embedding_vectors.tsv and embedding_metadata.tsv)\n",
        "- Explore\n",
        "- Optional: You can share the data you've created by clicking \"Publish\"\n",
        "\n",
        "What do you find?\n",
        "\n",
        "Are words with similar meanings close together?\n",
        "\n",
        "Remember, they might not be. The embeddings we downloaded are how our model interprets words, not necessarily how we interpret them.\n",
        "\n",
        "Also, since the embedding has been learned purely from Tweets, it may contain some strange values as Tweets are a very unique style of natural language.\n",
        "\n",
        "> 🤔 Question: Do you have to visualize embeddings every time?\n",
        "\n",
        "No. Although helpful for gaining an intuition of what natural language embeddings are, it's not completely necessary. Especially as the dimensions of your vocabulary and embeddings grow, trying to comprehend them would become an increasingly difficult task.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A-XMGR3VBZO1"
      },
      "source": [
        "## Recurrent Neural Networks (RNN's)\n",
        "For our next series of modelling experiments we're going to be using a special kind of neural network called a Recurrent Neural Network (RNN).\n",
        "\n",
        "The premise of an RNN is simple: use information from the past to help you with the future (this is where the term recurrent comes from). In other words, take an input (X) and compute an output (y) based on all previous inputs.\n",
        "\n",
        "This concept is especially helpful when dealing with sequences such as passages of natural language text (such as our Tweets).\n",
        "\n",
        "For example, when you read this sentence, you take into context the previous words when deciphering the meaning of the current word dog.\n",
        "\n",
        "See what happened there?\n",
        "\n",
        "I put the word \"dog\" at the end which is a valid word but it doesn't make sense in the context of the rest of the sentence.\n",
        "\n",
        "When an RNN looks at a sequence of text (already in numerical form), the patterns it learns are continually updated based on the order of the sequence.\n",
        "\n",
        "For a simple example, take two sentences:\n",
        "\n",
        "- Massive earthquake last week, no?\n",
        "- No massive earthquake last week.\n",
        "Both contain exactly the same words but have different meaning. The order of the words determines the meaning (one could argue punctuation marks also dictate the meaning but for simplicity sake, let's stay focused on the words).\n",
        "\n",
        "Recurrent neural networks can be used for a number of sequence-based problems:\n",
        "\n",
        "- One to one: one input, one output, such as image classification.\n",
        "- One to many: one input, many outputs, such as image captioning (image input, a sequence of text as caption output).\n",
        "- Many to one: many inputs, one outputs, such as text classification (classifying a Tweet as real diaster or not real diaster).\n",
        "- Many to many: many inputs, many outputs, such as machine translation (translating English to Spanish) or speech to text (audio wave as input, text as output).\n",
        "\n",
        "When you come across RNN's in the wild, you'll most likely come across variants of the following:\n",
        "\n",
        "- Long short-term memory cells (LSTMs).\n",
        "- Gated recurrent units (GRUs).\n",
        "- Bidirectional RNN's (passes forward and backward along a sequence, left to right and right to left).\n",
        "\n",
        "Going into the details of each these is beyond the scope of this notebook (we're going to focus on using them instead), the main thing you should know for now is that they've proven very effective at modelling sequences.\n",
        "\n",
        "For a deeper understanding of what's happening behind the scenes of the code we're about to write, I'd recommend the following resources:\n",
        "\n",
        "> 📖 Resources:\n",
        "\n",
        "MIT Deep Learning Lecture on Recurrent Neural Networks - explains the background of recurrent neural networks and introduces LSTMs.\n",
        "\n",
        "The Unreasonable Effectiveness of Recurrent Neural Networks by Andrej Karpathy - demonstrates the power of RNN's with examples generating various sequences.\n",
        "\n",
        "Understanding LSTMs by Chris Olah - an in-depth (and technical) look at the mechanics of the LSTM cell, possibly the most popular RNN building block."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hYi-DqYgCUP9"
      },
      "source": [
        "## Model 2: LSTM\n",
        "With all this talk of what RNN's are and what they're good for, I'm sure you're eager to build one.\n",
        "\n",
        "We're going to start with an LSTM-powered RNN.\n",
        "\n",
        "To harness the power of the LSTM cell (LSTM cell and LSTM layer are often used interchangably) in TensorFlow, we'll use tensorflow.keras.layers.LSTM().\n",
        "\n",
        "Our model is going to take on a very similar structure to model_1:\n",
        "\n",
        "> Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)\n",
        "\n",
        "The main difference will be that we're going to add an LSTM layer between our embedding and output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l5izZsNa_yyj",
        "outputId": "58b81954-a7ef-47aa-a38a-31b3e8170d6a"
      },
      "source": [
        "# Create LSTM model\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "print(x.shape)\n",
        "# x = layers.LSTM(64, return_sequences=True)(x) # return vector for each word in the Tweet (you can stack RNN cells as long as return_sequences=True)\n",
        "x = layers.LSTM(64)(x) # return vector for whole sequence\n",
        "print(x.shape)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer on top of output of LSTM cell\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_2 = tf.keras.Model(inputs, outputs, name=\"model_2_LSTM\")"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(None, 15, 128)\n",
            "(None, 64)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bU2-5uAEC6Wq"
      },
      "source": [
        "> 🔑 Note: Reading the documentation for the TensorFlow LSTM layer, you'll find a plethora of parameters. Many of these have been tuned to make sure they compute as fast as possible. The main ones you'll be looking to adjust are units (number of hidden units) and return_sequences (set this to True when stacking LSTM or other recurrent layers).\n",
        "\n",
        "Now we've got our LSTM model built, let's compile it using \"binary_crossentropy\" loss and the Adam optimizer.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "e1YMCXlJCwpF"
      },
      "source": [
        "# Compile the model\n",
        "model_2.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 44,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XRs0oF4_DHk4",
        "outputId": "7c977536-2c19-40f7-c43f-7652d955b65a"
      },
      "source": [
        "model_2.summary()"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_2_LSTM\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "lstm (LSTM)                  (None, 64)                49408     \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 1,329,473\n",
            "Trainable params: 1,329,473\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WB_rwlrPDTl_"
      },
      "source": [
        "\n",
        "Looking good! You'll notice a fair few more trainable parameters within our LSTM layer than model_1.\n",
        "\n",
        "If you'd like to know where this number comes from, I recommend going through the above resources as well the following on calculating the number of parameters in an LSTM cell:\n",
        "\n",
        "- Stack Overflow answer to calculate the number of parameters in an LSTM cell by Marcin Możejko\n",
        "\n",
        "- Calculating number of parameters in a LSTM unit and layer by Shridhar Priyadarshi\n",
        "\n",
        "Now our first RNN model's compiled let's fit it to our training data, validating it on the validation data and tracking its training parameters using our TensorBoard callback.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3rn4qeHGDJjT",
        "outputId": "93fb2dac-00ef-48a0-d065-f5655998f4a6"
      },
      "source": [
        "# Fit model\n",
        "model_2_history = model_2.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"LSTM\")])"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/LSTM/20210923-070757\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 7s 26ms/step - loss: 0.2192 - accuracy: 0.9175 - val_loss: 0.5134 - val_accuracy: 0.7769\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.1561 - accuracy: 0.9428 - val_loss: 0.7231 - val_accuracy: 0.7808\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 5s 23ms/step - loss: 0.1283 - accuracy: 0.9498 - val_loss: 0.7141 - val_accuracy: 0.7822\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.1063 - accuracy: 0.9602 - val_loss: 0.7586 - val_accuracy: 0.7808\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 5s 23ms/step - loss: 0.0867 - accuracy: 0.9657 - val_loss: 0.8765 - val_accuracy: 0.7717\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXml2atsEDkO"
      },
      "source": [
        "\n",
        "Nice! We've got our first trained RNN model using LSTM cells. Let's make some predictions with it.\n",
        "\n",
        "The same thing will happen as before, due to the sigmoid activiation function in the final layer, when we call the predict() method on our model, it'll return prediction probabilities rather than classes.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wKN3Js9zD0PC",
        "outputId": "e362a564-2e0e-4cd0-cedf-4185002fe1c0"
      },
      "source": [
        "# Make predictions on the validation dataset\n",
        "model_2_pred_probs = model_2.predict(val_sentences)\n",
        "model_2_pred_probs.shape, model_2_pred_probs[:10] # view the first 10"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((762, 1), array([[0.00922349],\n",
              "        [0.6965489 ],\n",
              "        [0.99966335],\n",
              "        [0.04516658],\n",
              "        [0.00108173],\n",
              "        [0.99935806],\n",
              "        [0.67222005],\n",
              "        [0.9997759 ],\n",
              "        [0.99967325],\n",
              "        [0.32209593]], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0lXVIgPkEFL7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6bb2e58a-2ec1-48ce-c0cc-bf5a524965af"
      },
      "source": [
        "# Round out predictions and reduce to 1-dimensional array\n",
        "model_2_preds = tf.squeeze(tf.round(model_2_pred_probs))\n",
        "model_2_preds[:10]"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qEl77_SxEHFg",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5fd5ce06-1398-433b-9463-8eef5add34f1"
      },
      "source": [
        "# Calculate LSTM model results\n",
        "model_2_results = calculate_results(y_true=val_labels,\n",
        "                                    y_pred=model_2_preds)\n",
        "model_2_results"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.16535433070865,\n",
              " 'f1_score': 0.7687482528669142,\n",
              " 'precision': 0.7751367943413853,\n",
              " 'recall': 0.7716535433070866}"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J_r07F43EJLK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b231f555-5721-4ec7-b413-b689db490710"
      },
      "source": [
        "# Compare model 2 to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_2_results)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 77.17, Difference: -2.10\n",
            "Baseline precision: 0.81, New precision: 0.78, Difference: -0.04\n",
            "Baseline recall: 0.79, New recall: 0.77, Difference: -0.02\n",
            "Baseline f1_score: 0.79, New f1_score: 0.77, Difference: -0.02\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "17vt6OpbGl6f"
      },
      "source": [
        "## Model 3: GRU\n",
        "Another popular and effective RNN component is the GRU or gated recurrent unit.\n",
        "\n",
        "The GRU cell has similar features to an LSTM cell but has less parameters.\n",
        "\n",
        "> 📖 Resource: A full explanation of the GRU cell is beyond the scope of this noteook but I'd suggest the following resources to learn more:\n",
        "\n",
        "- Gated Recurrent Unit Wikipedia page\n",
        "- Understanding GRU networks by Simeon Kostadinov\n",
        "To use the GRU cell in TensorFlow, we can call the tensorflow.keras.layers.GRU() class.\n",
        "\n",
        "The architecture of the GRU-powered model will follow the same structure we've been using:\n",
        "\n",
        "> Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)\n",
        "\n",
        "Again, the only difference will be the layer(s) we use between the embedding and the output."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JZXx2v7YEK9z"
      },
      "source": [
        "# Build an RNN using the GRU Cell\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.GRU(64, return_sequences=True) # stacking recurrent cells requires return_sequences=True\n",
        "x = layers.GRU(64)(x)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer after GRU cell\n",
        "\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_3 = tf.keras.Model(inputs, outputs, name=\"model_3_GRU\")"
      ],
      "execution_count": 51,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6i2TqElHUCv"
      },
      "source": [
        "TensorFlow makes it easy to use powerful components such as the GRU cell in our models. And now our third model is built, let's compile it, just as before.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wba5c6dNHQmf"
      },
      "source": [
        "# Compile the model\n",
        "model_3.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer = tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 52,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "deiIOb6YHfZn",
        "outputId": "73ed4beb-4401-437f-ed84-0db60a31f553"
      },
      "source": [
        "model_3.summary()"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_3_GRU\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_3 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "gru (GRU)                    (None, 64)                37248     \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 1,317,313\n",
            "Trainable params: 1,317,313\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWJqdlwzHlim"
      },
      "source": [
        "\n",
        "Notice the difference in number of trainable parameters between model_2 (LSTM) and model_3 (GRU). The difference comes from the LSTM cell having more trainable parameters than the GRU cell.\n",
        "\n",
        "We'll fit our model just as we've been doing previously. We'll also track our models results using our create_tensorboard_callback() function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HeBrKrG4Hgs3",
        "outputId": "f828a6bb-4c23-4d79-aa4b-1dd7bd50ccb6"
      },
      "source": [
        "#Fit model\n",
        "model_3_history = model_3.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data = (val_sentences, val_labels),\n",
        "                              callbacks= [create_tensorboard_callback(SAVE_DIR, \"GRU\")])"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/GRU/20210923-070824\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 7s 25ms/step - loss: 0.1595 - accuracy: 0.9381 - val_loss: 0.7895 - val_accuracy: 0.7782\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.0827 - accuracy: 0.9695 - val_loss: 0.7746 - val_accuracy: 0.7769\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.0690 - accuracy: 0.9734 - val_loss: 0.9798 - val_accuracy: 0.7782\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 5s 22ms/step - loss: 0.0606 - accuracy: 0.9758 - val_loss: 1.0863 - val_accuracy: 0.7782\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 5s 23ms/step - loss: 0.0521 - accuracy: 0.9765 - val_loss: 1.2444 - val_accuracy: 0.7769\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gppbQHJMIAqY"
      },
      "source": [
        "\n",
        "Due to the optimized default settings of the GRU cell in TensorFlow, training doesn't take long at all."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IxO7wSCnH2dN",
        "outputId": "a811de9d-7032-4f14-9a0f-82229f229ce1"
      },
      "source": [
        "# Make predictions on the validation data\n",
        "model_3_pred_probs = model_3.predict(val_sentences)\n",
        "model_3_pred_probs.shape, model_3_pred_probs[:10]"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((762, 1), array([[3.3012033e-04],\n",
              "        [8.3059061e-01],\n",
              "        [9.9992996e-01],\n",
              "        [8.7063491e-02],\n",
              "        [7.3562725e-05],\n",
              "        [9.9989176e-01],\n",
              "        [9.6812946e-01],\n",
              "        [9.9997228e-01],\n",
              "        [9.9994051e-01],\n",
              "        [9.9611545e-01]], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-F36YC9OICja",
        "outputId": "50edbe8b-547e-4ce5-c5ce-2e94c6acd736"
      },
      "source": [
        "# Convert prediction probabilities to prediction classes\n",
        "model_3_preds = tf.squeeze(tf.round(model_3_pred_probs))\n",
        "model_3_preds[:10]"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5Htps0DxIEii",
        "outputId": "90841750-926d-4f04-e235-ddf892ce2735"
      },
      "source": [
        "# Calcuate model_3 results\n",
        "model_3_results = calculate_results(y_true=val_labels, \n",
        "                                    y_pred=model_3_preds)\n",
        "model_3_results"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.69028871391076,\n",
              " 'f1_score': 0.7748796582613408,\n",
              " 'precision': 0.7786827971006937,\n",
              " 'recall': 0.7769028871391076}"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TVDbMfOOIGcc",
        "outputId": "650fe4a0-a9fe-4721-b741-01d2b1b5344f"
      },
      "source": [
        "# Compare to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_3_results)"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 77.69, Difference: -1.57\n",
            "Baseline precision: 0.81, New precision: 0.78, Difference: -0.03\n",
            "Baseline recall: 0.79, New recall: 0.78, Difference: -0.02\n",
            "Baseline f1_score: 0.79, New f1_score: 0.77, Difference: -0.01\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gea8X25yIhUa"
      },
      "source": [
        "## Model 4: Bidirectonal RNN model\n",
        "Look at us go! We've already built two RNN's with GRU and LSTM cells. Now we're going to look into another kind of RNN, the bidirectional RNN.\n",
        "\n",
        "A standard RNN will process a sequence from left to right, where as a bidirectional RNN will process the sequence from left to right and then again from right to left.\n",
        "\n",
        "Intuitively, this can be thought of as if you were reading a sentence for the first time in the normal fashion (left to right) but for some reason it didn't make sense so you traverse back through the words and go back over them again (right to left).\n",
        "\n",
        "In practice, many sequence models often see and improvement in performance when using bidirectional RNN's.\n",
        "\n",
        "However, this improvement in performance often comes at the cost of longer training times and increased model parameters (since the model goes left to right and right to left, the number of trainable parameters doubles).\n",
        "\n",
        "Okay enough talk, let's build a bidirectional RNN.\n",
        "\n",
        "Once again, TensorFlow helps us out by providing the tensorflow.keras.layers.Bidirectional class. We can use the Bidirectional class to wrap our existing RNNs, instantly making them bidirectional."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L_uMMeyzIJwH"
      },
      "source": [
        "#  Build a Bidirectional RNN in TensorFlow\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "# x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(x) # stacking RNN layers requires return_sequences=True\n",
        "x = layers.Bidirectional(layers.LSTM(64))(x) # bidirectional goes both ways so has double the parameters of a regular LSTM layer\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_4 = tf.keras.Model(inputs, outputs, name=\"model_4_Bidirectional\")"
      ],
      "execution_count": 59,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MjEE5j5gJKU_"
      },
      "source": [
        "> 🔑 Note: You can use the Bidirectional wrapper on any RNN cell in TensorFlow. For example, layers.Bidirectional(layers.GRU(64)) creates a bidirectional GRU cell.\n",
        "\n",
        "Our bidirectional model is built, let's compile it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "93BlVkDNJFuE"
      },
      "source": [
        "# Compile\n",
        "model_4.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Y3egMxD8JOFQ",
        "outputId": "21222006-9fba-47de-89e5-c1c87910050d"
      },
      "source": [
        "# Get a summary of our bidirectional model\n",
        "model_4.summary()"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_4_Bidirectional\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_4 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "bidirectional (Bidirectional (None, 128)               98816     \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 1)                 129       \n",
            "=================================================================\n",
            "Total params: 1,378,945\n",
            "Trainable params: 1,378,945\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QW9DHcVhJTtf"
      },
      "source": [
        "\n",
        "Notice the increased number of trainable parameters in model_4 (bidirectional LSTM) compared to model_2 (regular LSTM). This is due to the bidirectionality we added to our RNN.\n",
        "\n",
        "Time to fit our bidirectional model and track its performance."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AOrQxFSMJShg",
        "outputId": "52b5d33b-1f39-4646-b552-e707d4862566"
      },
      "source": [
        "# Fit the model (takes longer because of the bidirectional layers)\n",
        "model_4_history = model_4.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"bidirectional_RNN\")])"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/bidirectional_RNN/20210923-070908\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 11s 36ms/step - loss: 0.1051 - accuracy: 0.9682 - val_loss: 1.0647 - val_accuracy: 0.7625\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 7s 31ms/step - loss: 0.0544 - accuracy: 0.9772 - val_loss: 1.0872 - val_accuracy: 0.7795\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 7s 30ms/step - loss: 0.0458 - accuracy: 0.9793 - val_loss: 1.3188 - val_accuracy: 0.7559\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 7s 31ms/step - loss: 0.0431 - accuracy: 0.9803 - val_loss: 1.2611 - val_accuracy: 0.7441\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 7s 31ms/step - loss: 0.0424 - accuracy: 0.9801 - val_loss: 1.3971 - val_accuracy: 0.7612\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WCMz8I9-KIi8"
      },
      "source": [
        "Due to the bidirectionality of our model we see a slight increase in training time.\n",
        "\n",
        "Not to worry, it's not too dramatic of an increase.\n",
        "\n",
        "Let's make some predictions with it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CgoV5-_SJVsN",
        "outputId": "baeec806-ee0e-4df1-d203-b7a358e17c28"
      },
      "source": [
        "# Make predictions with bidirectional RNN on the validation data\n",
        "model_4_pred_probs = model_4.predict(val_sentences)\n",
        "model_4_pred_probs[:10]"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[6.7263842e-04],\n",
              "       [7.1435118e-01],\n",
              "       [9.9983978e-01],\n",
              "       [1.1493069e-01],\n",
              "       [4.5405777e-05],\n",
              "       [9.9948549e-01],\n",
              "       [8.9417624e-01],\n",
              "       [9.9996048e-01],\n",
              "       [9.9989235e-01],\n",
              "       [9.9900758e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 63
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2mS_wRNeKMlH",
        "outputId": "44af206c-b3f7-4c86-d889-42cc8d26cbf0"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_4_preds = tf.squeeze(tf.round(model_4_pred_probs))\n",
        "model_4_preds[:10]"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZWaZpJxvKO0s",
        "outputId": "4830e67c-9b4f-4d70-cc17-aabd8a73fea5"
      },
      "source": [
        "# Calculate bidirectional RNN model results\n",
        "model_4_results = calculate_results(val_labels, model_4_preds)\n",
        "model_4_results"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 76.11548556430446,\n",
              " 'f1_score': 0.7600554649035874,\n",
              " 'precision': 0.7610421046068336,\n",
              " 'recall': 0.7611548556430446}"
            ]
          },
          "metadata": {},
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BwjBz8pDKQMt",
        "outputId": "1720a589-6815-4ced-da3c-b4e690878ee7"
      },
      "source": [
        "# Check to see how the bidirectional model performs against the baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_4_results)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 76.12, Difference: -3.15\n",
            "Baseline precision: 0.81, New precision: 0.76, Difference: -0.05\n",
            "Baseline recall: 0.79, New recall: 0.76, Difference: -0.03\n",
            "Baseline f1_score: 0.79, New f1_score: 0.76, Difference: -0.03\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T9ZHFHDyKl4W"
      },
      "source": [
        "## Convolutional Neural Networks for Text\n",
        "You might've used convolutional neural networks (CNNs) for images before but they can also be used for sequences.\n",
        "\n",
        "The main difference between using CNNs for images and sequences is the shape of the data. Images come in 2-dimensions (height x width) where as sequences are often 1-dimensional (a string of text).\n",
        "\n",
        "So to use CNNs with sequences, we use a 1-dimensional convolution instead of a 2-dimensional convolution.\n",
        "\n",
        "A typical CNN architecture for sequences will look like the following:\n",
        "\n",
        "> Inputs (text) -> Tokenization -> Embedding -> Layers -> Outputs (class probabilities)\n",
        "\n",
        "You might be thinking \"that just looks like the architecture layout we've been using for the other models...\"\n",
        "\n",
        "And you'd be right.\n",
        "\n",
        "The difference again is in the layers component. Instead of using an LSTM or GRU cell, we're going to use a tensorflow.keras.layers.Conv1D() layer followed by a tensorflow.keras.layers.GlobablMaxPool1D() layer.\n",
        "\n",
        "> 📖 Resource: The intuition here is explained succinctly in the paper[Understanding Convolutional Neural Networks for Text Classification, ](https://aclanthology.org/W18-5408.pdf) where they state that CNNs classify text through the following steps:\n",
        "\n",
        "- 1-dimensional convolving filters are used as ngram detectors, each filter specializing in a closely-related family of ngrams (an ngram is a collection of n-words, for example, an ngram of 5 might result in \"hello, my name is Daniel\").\n",
        "- Max-pooling over time extracts the relevant ngrams for making a decision.\n",
        "- The rest of the network classifies the text based on this information.\n",
        "\n",
        "## Model 5: Conv1D\n",
        "Before we build a full 1-dimensional CNN model, let's see a 1-dimensional convolutional layer (also called a temporal convolution) in action.\n",
        "\n",
        "We'll first create an embedding of a sample of text and experiment passing it through a Conv1D() layer and GlobalMaxPool1D() layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BFz5T8rTKSCi",
        "outputId": "d70b5d53-898d-4b5a-a99a-c5cd167c0d26"
      },
      "source": [
        "# Test out the embedding, 1D convolutional and max pooling\n",
        "embedding_test = embedding(text_vectorizer([\"this is a test sentence\"])) # turn target sentence into embedding\n",
        "conv_1d = layers.Conv1D(filters=32, kernel_size=5, activation=\"relu\") # convolve over target sequence 5 words at a time\n",
        "conv_1d_output = conv_1d(embedding_test) # pass embedding through 1D convolutional layer\n",
        "max_pool = layers.GlobalMaxPool1D() \n",
        "max_pool_output = max_pool(conv_1d_output) # get the most important features\n",
        "embedding_test.shape, conv_1d_output.shape, max_pool_output.shape"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([1, 15, 128]), TensorShape([1, 11, 32]), TensorShape([1, 32]))"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pf_RC91UMAzb"
      },
      "source": [
        "\n",
        "Notice the output shapes of each layer.\n",
        "\n",
        "The embedding has an output shape dimension of the parameters we set it to (input_length=15 and output_dim=128).\n",
        "\n",
        "The 1-dimensional convolutional layer has an output which has been compressed inline with its parameters. And the same goes for the max pooling layer output.\n",
        "\n",
        "Our text starts out as a string but gets converted to a feature vector of length 64 through various transformation steps (from tokenization to embedding to 1-dimensional convolution to max pool).\n",
        "\n",
        "Let's take a peak at what each of these transformations looks like."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZrgjEl8YLpEE",
        "outputId": "2e84e3f6-6938-4785-fe99-cb888702163b"
      },
      "source": [
        "# See the outputs of each layer\n",
        "embedding_test[:1], conv_1d_output[:1], max_pool_output[:1]"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(1, 15, 128), dtype=float32, numpy=\n",
              " array([[[ 0.02780653,  0.00203244, -0.00530461, ..., -0.01946628,\n",
              "          -0.00015379,  0.03849815],\n",
              "         [ 0.05453511, -0.03749858, -0.06332359, ..., -0.01304879,\n",
              "          -0.04560092, -0.04129295],\n",
              "         [ 0.02811794, -0.0194961 ,  0.068604  , ...,  0.0094632 ,\n",
              "           0.00520931,  0.03739037],\n",
              "         ...,\n",
              "         [ 0.04146522,  0.04649357, -0.01223966, ..., -0.03891463,\n",
              "           0.03524785, -0.04111269],\n",
              "         [ 0.04146522,  0.04649357, -0.01223966, ..., -0.03891463,\n",
              "           0.03524785, -0.04111269],\n",
              "         [ 0.04146522,  0.04649357, -0.01223966, ..., -0.03891463,\n",
              "           0.03524785, -0.04111269]]], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1, 11, 32), dtype=float32, numpy=\n",
              " array([[[4.14357483e-02, 4.58854288e-02, 5.08416630e-02, 5.89552522e-03,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 7.96140209e-02,\n",
              "          1.12115052e-02, 1.98621713e-02, 1.81279797e-02, 0.00000000e+00,\n",
              "          3.70511971e-02, 0.00000000e+00, 3.16382758e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 2.92041022e-02,\n",
              "          0.00000000e+00, 0.00000000e+00, 3.53439339e-02, 1.04230851e-01,\n",
              "          9.51274112e-03, 0.00000000e+00, 5.23977056e-02, 0.00000000e+00,\n",
              "          9.69986618e-03, 7.09086210e-02, 2.71464530e-02, 0.00000000e+00],\n",
              "         [1.00393638e-01, 0.00000000e+00, 4.12566960e-02, 0.00000000e+00,\n",
              "          1.17755495e-04, 0.00000000e+00, 1.71719417e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          1.12861227e-02, 5.91278076e-04, 1.41468225e-03, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 1.12121791e-01, 0.00000000e+00,\n",
              "          0.00000000e+00, 1.41569078e-02, 0.00000000e+00, 8.28551725e-02,\n",
              "          6.02121130e-02, 4.59452793e-02, 2.58136522e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 5.52502386e-02],\n",
              "         [6.66107908e-02, 9.65830311e-03, 1.64456666e-02, 2.16342416e-02,\n",
              "          3.28405425e-02, 2.76187845e-02, 6.17115386e-03, 0.00000000e+00,\n",
              "          0.00000000e+00, 4.07338887e-02, 4.32704613e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 1.22092953e-02, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 5.06688915e-02,\n",
              "          3.83568108e-02, 2.44185943e-02, 8.13566148e-05, 1.34836398e-02,\n",
              "          2.90011037e-02, 0.00000000e+00, 6.16568103e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],\n",
              "         [1.26537263e-01, 0.00000000e+00, 6.62415149e-03, 5.64846396e-02,\n",
              "          5.00886329e-03, 0.00000000e+00, 8.77037831e-03, 7.28486627e-02,\n",
              "          0.00000000e+00, 9.66673903e-03, 2.63417251e-02, 0.00000000e+00,\n",
              "          5.20854443e-02, 0.00000000e+00, 4.15207930e-02, 2.57075578e-02,\n",
              "          2.63671968e-02, 1.73762254e-02, 2.43628547e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 8.11193734e-02, 0.00000000e+00, 5.87894581e-02,\n",
              "          0.00000000e+00, 1.42482463e-02, 1.09783020e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 6.77329674e-03, 0.00000000e+00],\n",
              "         [4.35046703e-02, 0.00000000e+00, 0.00000000e+00, 5.59407957e-02,\n",
              "          1.09894741e-02, 0.00000000e+00, 3.93665396e-03, 0.00000000e+00,\n",
              "          4.22157794e-02, 1.26345735e-02, 4.36789282e-02, 6.53194403e-03,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 1.68601498e-02, 0.00000000e+00, 7.15586729e-03,\n",
              "          0.00000000e+00, 7.33783282e-03, 1.29808635e-02, 3.75542603e-03,\n",
              "          0.00000000e+00, 7.44825453e-02, 2.04651840e-02, 0.00000000e+00,\n",
              "          2.38700025e-02, 1.90587640e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00],\n",
              "         [3.57347727e-02, 0.00000000e+00, 1.84348393e-02, 5.43548428e-02,\n",
              "          1.30102485e-02, 5.02031762e-03, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.51408666e-03, 1.63588636e-02, 0.00000000e+00,\n",
              "          0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00,\n",
              "          0.00000000e+00, 2.20529679e-02, 1.48724606e-02, 2.07543895e-02,\n",
              "          0.00000000e+00, 1.95472818e-02, 3.08901444e-03, 4.54204157e-02,\n",
              "          1.11136753e-02, 5.71373217e-02, 0.00000000e+00, 4.10768017e-03,\n",
              "          3.70520987e-02, 7.37374723e-02, 0.00000000e+00, 0.00000000e+00]]],\n",
              "       dtype=float32)>,\n",
              " <tf.Tensor: shape=(1, 32), dtype=float32, numpy=\n",
              " array([[0.12653726, 0.04588543, 0.05084166, 0.05648464, 0.03284054,\n",
              "         0.02761878, 0.01717194, 0.07961402, 0.04221578, 0.04073389,\n",
              "         0.04367893, 0.00653194, 0.05208544, 0.0122093 , 0.04152079,\n",
              "         0.02570756, 0.0263672 , 0.02205297, 0.11212179, 0.05066889,\n",
              "         0.03835681, 0.08111937, 0.03534393, 0.10423085, 0.06021211,\n",
              "         0.07448255, 0.06165681, 0.00410768, 0.0370521 , 0.07373747,\n",
              "         0.02714645, 0.05525024]], dtype=float32)>)"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DIGK5A-bMLme"
      },
      "source": [
        "\n",
        "Alright, we've seen the outputs of several components of a CNN for sequences, let's put them together and construct a full model, compile it (just as we've done with our other models) and get a summary."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zj9xNBvfMDct",
        "outputId": "7519316a-3d60-4023-a388-344e3af59403"
      },
      "source": [
        "# Create 1-dimensional convolutional layer to model sequences\n",
        "from tensorflow.keras import layers\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "x = text_vectorizer(inputs)\n",
        "x = embedding(x)\n",
        "x = layers.Conv1D(filters=32, kernel_size=5, activation=\"relu\")(x)\n",
        "x = layers.GlobalMaxPool1D()(x)\n",
        "# x = layers.Dense(64, activation=\"relu\")(x) # optional dense layer\n",
        "outputs = layers.Dense(1, activation=\"sigmoid\")(x)\n",
        "model_5 = tf.keras.Model(inputs, outputs, name=\"model_5_Conv1D\")\n",
        "\n",
        "# Compile Conv1D model\n",
        "model_5.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get a summary of our 1D convolution model\n",
        "model_5.summary()"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_5_Conv1D\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_5 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization_1 (TextVe (None, 15)                0         \n",
            "_________________________________________________________________\n",
            "embedding (Embedding)        (None, 15, 128)           1280000   \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 11, 32)            20512     \n",
            "_________________________________________________________________\n",
            "global_max_pooling1d_1 (Glob (None, 32)                0         \n",
            "_________________________________________________________________\n",
            "dense_4 (Dense)              (None, 1)                 33        \n",
            "=================================================================\n",
            "Total params: 1,300,545\n",
            "Trainable params: 1,300,545\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q25l_br9MT0n"
      },
      "source": [
        "\n",
        "Woohoo! Looking great! Notice how the number of trainable parameters for the 1-dimensional convolutional layer is similar to that of the LSTM layer in model_2.\n",
        "\n",
        "Let's fit our 1D CNN model to our text data. In line with previous experiments, we'll save its results using our create_tensorboard_callback() function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "y1WFY6BNMPMX",
        "outputId": "5d4bd4c6-d285-4a5a-92e7-9626c9b78971"
      },
      "source": [
        "# Fit the model\n",
        "model_5_history = model_5.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \n",
        "                                                                     \"Conv1D\")])"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/Conv1D/20210923-070946\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 4s 15ms/step - loss: 0.1296 - accuracy: 0.9645 - val_loss: 0.8615 - val_accuracy: 0.7677\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.0752 - accuracy: 0.9712 - val_loss: 1.0109 - val_accuracy: 0.7612\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.0613 - accuracy: 0.9775 - val_loss: 1.0987 - val_accuracy: 0.7480\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.0533 - accuracy: 0.9772 - val_loss: 1.1811 - val_accuracy: 0.7533\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 3s 14ms/step - loss: 0.0506 - accuracy: 0.9799 - val_loss: 1.2106 - val_accuracy: 0.7428\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hq1y2JZ4MaDh"
      },
      "source": [
        "Nice! Thanks to GPU acceleration, our 1D convolutional model trains nice and fast. Let's make some predictions with it and evaluate them just as before."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IaaP9mchMVqK",
        "outputId": "7651c0fe-f32f-4510-d26f-6db14d94b76f"
      },
      "source": [
        "# Make predictions with model_5\n",
        "model_5_pred_probs = model_5.predict(val_sentences)\n",
        "model_5_pred_probs[:10]"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[3.8595200e-02],\n",
              "       [8.2051539e-01],\n",
              "       [9.9980509e-01],\n",
              "       [4.8800677e-02],\n",
              "       [7.0517743e-08],\n",
              "       [9.9484587e-01],\n",
              "       [9.6185786e-01],\n",
              "       [9.9998152e-01],\n",
              "       [9.9999881e-01],\n",
              "       [9.6218956e-01]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RspMbu6oMehu",
        "outputId": "f74b5299-48e8-4551-a2fc-dde8638a0a2f"
      },
      "source": [
        "# Convert model_5 prediction probabilities to labels\n",
        "model_5_preds = tf.squeeze(tf.round(model_5_pred_probs))\n",
        "model_5_preds[:10]"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 0., 1., 1., 1., 1., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rY0x8uKRMgBJ",
        "outputId": "42fd4335-62a6-4129-d9a6-6077521f0695"
      },
      "source": [
        "#  Calculate model_5 evaluation metrics \n",
        "model_5_results = calculate_results(y_true=val_labels, \n",
        "                                    y_pred=model_5_preds)\n",
        "model_5_results"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 74.2782152230971,\n",
              " 'f1_score': 0.7417068523395672,\n",
              " 'precision': 0.7423828567871703,\n",
              " 'recall': 0.7427821522309711}"
            ]
          },
          "metadata": {},
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4lL5nJ_oMhse",
        "outputId": "7c8485b6-8268-4708-f4be-792feb1b1dfd"
      },
      "source": [
        "# Compare model_5 results to baseline \n",
        "compare_baseline_to_new_results(baseline_results, model_5_results)"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 74.28, Difference: -4.99\n",
            "Baseline precision: 0.81, New precision: 0.74, Difference: -0.07\n",
            "Baseline recall: 0.79, New recall: 0.74, Difference: -0.05\n",
            "Baseline f1_score: 0.79, New f1_score: 0.74, Difference: -0.04\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jHxlABueOMPc"
      },
      "source": [
        "## Using Pretrained Embeddings (transfer learning for NLP)\n",
        "For all of the previous deep learning models we've built and trained, we've created and used our own embeddings from scratch each time.\n",
        "\n",
        "However, a common practice is to leverage pretrained embeddings through transfer learning. This is one of the main benefits of using deep models: being able to take what one (often larger) model has learned (often on a large amount of data) and adjust it for our own use case.\n",
        "\n",
        "For our next model, instead of using our own embedding layer, we're going to replace it with a pretrained embedding layer.\n",
        "\n",
        "More specifically, we're going to be using the Universal Sentence Encoder from TensorFlow Hub (a great resource containing a plethora of pretrained model resources for a variety of tasks).\n",
        "\n",
        "> 🔑 Note: There are many different pretrained text embedding options on TensorFlow Hub, however, some require different levels of text preprocessing than others. Best to experiment with a few and see which best suits your use case.\n",
        "\n",
        "## Model 6: TensorFlow Hub Pretrained Sentence Encoder\n",
        "The main difference between the embedding layer we created and the Universal Sentence Encoder is that rather than create a word-level embedding, the Universal Sentence Encoder, as you might've guessed, creates a whole sentence-level embedding.\n",
        "\n",
        "Our embedding layer also outputs an a 128 dimensional vector for each word, where as, the Universal Sentence Encoder outputs a 512 dimensional vector for each sentence.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6U-3r2AOYA_"
      },
      "source": [
        "The feature extractor model we're building through the eyes of an encoder/decoder model.\n",
        "\n",
        "> 🔑 Note: An encoder is the name for a model which converts raw data such as text into a numerical representation (feature vector), a decoder converts the numerical representation to a desired output.\n",
        "\n",
        "As usual, this is best demonstrated with an example.\n",
        "\n",
        "We can load in a TensorFlow Hub module using the hub.load() method and passing it the target URL of the module we'd like to use, in our case, it's \"https://tfhub.dev/google/universal-sentence-encoder/4\".\n",
        "\n",
        "Let's load the Universal Sentence Encoder model and test it on a couple of sentences."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CP8UYTF0MjTj",
        "outputId": "63ba6782-2a10-4389-85a8-5b35ef5f971e"
      },
      "source": [
        "# Example of pretrained embedding with universal sentence encoder - https://tfhub.dev/google/universal-sentence-encoder/4\n",
        "import tensorflow_hub as hub\n",
        "embed = hub.load(\"https://tfhub.dev/google/universal-sentence-encoder/4\") # load Universal Sentence Encoder\n",
        "embed_samples = embed([sample_sentence,\"When you call the universal sentence encoder on a sentence, it turns it into numbers.\"])\n",
        "\n",
        "print(embed_samples[0][:50])"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tf.Tensor(\n",
            "[-0.01157029  0.02485909  0.0287805  -0.01271502  0.03971539  0.0882776\n",
            "  0.02680985  0.05589837 -0.01068732 -0.00597293  0.00639323 -0.01819516\n",
            "  0.00030816  0.09105889  0.05874642 -0.03180626  0.01512472 -0.05162928\n",
            "  0.00991366 -0.06865346 -0.04209306  0.0267898   0.03011009  0.00321067\n",
            " -0.00337971 -0.04787361  0.02266721 -0.00985926 -0.04063615 -0.01292092\n",
            " -0.04666384  0.05630298 -0.03949255  0.00517684  0.02495827 -0.07014441\n",
            "  0.0287151   0.0494768  -0.00633974 -0.08960193  0.02807117 -0.00808364\n",
            " -0.013606    0.05998649 -0.10361787 -0.05195374  0.00232957 -0.02332528\n",
            " -0.03758107  0.03327732], shape=(50,), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sZJMrVSRO6PN",
        "outputId": "ec487955-b31f-4238-ce3f-df4dbb9376db"
      },
      "source": [
        "# Each sentence has been encoded into a 512 dimension vector\n",
        "embed_samples[0].shape"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TensorShape([512])"
            ]
          },
          "metadata": {},
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tq4iotuWPT81"
      },
      "source": [
        "\n",
        "Passing our sentences to the Universal Sentence Encoder (USE) encodes them from strings to 512 dimensional vectors, which make no sense to us but hopefully make sense to our machine learning models.\n",
        "\n",
        "Speaking of models, let's build one with the USE as our embedding layer.\n",
        "\n",
        "We can convert the TensorFlow Hub USE module into a Keras layer using the hub.KerasLayer class.\n",
        "\n",
        "> 🔑 Note: Due to the size of the USE TensorFlow Hub module, it may take a little while to download. Once it's downloaded though, it'll be cached and ready to use. And as with many TensorFlow Hub modules, there is a \"lite\" version of the USE which takes up less space but sacrifices some performance and requires more preprocessing steps. However, depending on your available compute power, the lite version may be better for your application use case."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cLoibgsCPFNn"
      },
      "source": [
        "# We can use this encoding layer in place of our text_vectorizer and embedding layer\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[], # shape of inputs to our model\n",
        "                                        dtype=tf.string, # data type of inputs coming to the USE layer\n",
        "                                        trainable=False, # keep the pretrained weights (we'll create a feature extractor)\n",
        "                                        name=\"USE\")"
      ],
      "execution_count": 77,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5FZwChXgP8T4"
      },
      "source": [
        "\n",
        "Beautiful! Now we've got the USE as a Keras layer, we can use it in a Keras Sequential model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FO7_GukbP5lh",
        "outputId": "72005f2b-7535-4ae4-e23c-b7c9b3e08515"
      },
      "source": [
        "# Create model using the Sequential API\n",
        "# Create model using the Sequential API\n",
        "model_6 = tf.keras.Sequential([\n",
        "  sentence_encoder_layer, # take in sentences and then encode them into an embedding\n",
        "  layers.Dense(64, activation=\"relu\"),\n",
        "  layers.Dense(1, activation=\"sigmoid\")\n",
        "], name=\"model_6_USE\")\n",
        "\n",
        "# Compile model\n",
        "model_6.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "model_6.summary()"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "91o4gkwlQlx1"
      },
      "source": [
        "Notice the number of paramters in the USE layer, these are the pretrained weights its learned on various text sources (Wikipedia, web news, web question-answer forums, etc, see the Universal Sentence Encoder paper for more).\n",
        "\n",
        "The trainable parameters are only in our output layers, in other words, we're keeping the USE weights frozen and using it as a feature-extractor. We could fine-tune these weights by setting trainable=True when creating the hub.KerasLayer instance.\n",
        "\n",
        "Now we've got a feature extractor model ready, let's train it and track its results to TensorBoard using our create_tensorboard_callback() function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ebU9FfncQa6g",
        "outputId": "f1e62421-bc8e-4f4d-89aa-6494d3726e80"
      },
      "source": [
        "# Train a classifier on top of pretrained embeddings\n",
        "model_6_history = model_6.fit(train_sentences,\n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \n",
        "                                                                     \"tf_hub_sentence_encoder\")])"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/tf_hub_sentence_encoder/20210923-071030\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 7s 22ms/step - loss: 0.5110 - accuracy: 0.7800 - val_loss: 0.4458 - val_accuracy: 0.8045\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.4146 - accuracy: 0.8174 - val_loss: 0.4399 - val_accuracy: 0.8084\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.4008 - accuracy: 0.8235 - val_loss: 0.4299 - val_accuracy: 0.8189\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.3920 - accuracy: 0.8272 - val_loss: 0.4273 - val_accuracy: 0.8136\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.3841 - accuracy: 0.8321 - val_loss: 0.4242 - val_accuracy: 0.8136\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1r8d--E_QwBr"
      },
      "source": [
        "\n",
        "USE model trained! Let's make some predictions with it an evaluate them as we've done with our other models."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8Alc_h0RQoDD",
        "outputId": "836b8fa0-30c9-48a3-c4f4-0cc8450a621e"
      },
      "source": [
        "# Make predictions with USE TF Hub model\n",
        "model_6_pred_probs = model_6.predict(val_sentences)\n",
        "model_6_pred_probs[:10]"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.17200479],\n",
              "       [0.799361  ],\n",
              "       [0.9916382 ],\n",
              "       [0.21459398],\n",
              "       [0.7476342 ],\n",
              "       [0.7995194 ],\n",
              "       [0.9846957 ],\n",
              "       [0.98550594],\n",
              "       [0.95728445],\n",
              "       [0.11530343]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qmrOIEcjQxv7",
        "outputId": "02d6e6e4-155d-47f8-fe90-655b23521d2a"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_6_preds = tf.squeeze(tf.round(model_6_pred_probs))\n",
        "model_6_preds[:10]"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 81
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dNJZFBI2QzRv",
        "outputId": "03e1ba17-a3ca-4db7-8a65-be20dff67b86"
      },
      "source": [
        "# Calculate model 6 performance metrics\n",
        "model_6_results = calculate_results(val_labels, model_6_preds)\n",
        "model_6_results"
      ],
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 81.36482939632546,\n",
              " 'f1_score': 0.8127905275621398,\n",
              " 'precision': 0.8141694427619625,\n",
              " 'recall': 0.8136482939632546}"
            ]
          },
          "metadata": {},
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0CK4GpjqQ0a7",
        "outputId": "65f4c7dc-1336-4744-a99e-63c631e3a492"
      },
      "source": [
        "# Compare TF Hub model to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_6_results)"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 81.36, Difference: 2.10\n",
            "Baseline precision: 0.81, New precision: 0.81, Difference: 0.00\n",
            "Baseline recall: 0.79, New recall: 0.81, Difference: 0.02\n",
            "Baseline f1_score: 0.79, New f1_score: 0.81, Difference: 0.03\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MErydEMWRPzf"
      },
      "source": [
        "## Model 7: TensorFlow Hub Pretrained Sentence Encoder 10% of the training data\n",
        "One of the benefits of using transfer learning methods, such as, the pretrained embeddings within the USE is the ability to get great results on a small amount of data (the USE paper even mentions this in the abstract).\n",
        "\n",
        "To put this to the test, we're going to make a small subset of the training data (10%), train a model and evaluate it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "grHYxWYVQ16g"
      },
      "source": [
        "# One kind of correct way (there are more) to make data subset\n",
        "# (split the already split train_sentences/train_labels)\n",
        "train_sentences_90_percent, train_sentences_10_percent, train_labels_90_percent, train_labels_10_percent = train_test_split(np.array(train_sentences),\n",
        "                                                                                                                            train_labels,\n",
        "                                                                                                                            test_size=0.1,\n",
        "                                                                                                                            random_state=42)"
      ],
      "execution_count": 84,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PcdioBOTRggX",
        "outputId": "49b93a6d-4072-4a79-b35f-85fec37b32f6"
      },
      "source": [
        "# Check length of 10 percent datasets\n",
        "print(f\"Total training examples: {len(train_sentences)}\")\n",
        "print(f\"Length of 10% training examples: {len(train_sentences_10_percent)}\")\n"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total training examples: 6851\n",
            "Length of 10% training examples: 686\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dh4pCHYnRnge"
      },
      "source": [
        "\n",
        "Because we've selected a random subset of the training samples, the classes should be roughly balanced (as they are in the full training dataset)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LryRYmrNRiTB",
        "outputId": "d1f5865e-8682-4dcd-f8fd-661445d25c2b"
      },
      "source": [
        "# Check the number of targets in our subset of data \n",
        "# (this should be close to the distribution of labels in the original train_labels)\n",
        "pd.Series(train_labels_10_percent).value_counts()"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0    415\n",
              "1    271\n",
              "dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uCA6QOcuRx5h"
      },
      "source": [
        "\n",
        "To make sure we're making an appropriate comparison between our model's ability to learn from the full training set and 10% subset, we'll clone our USE model (model_6) using the tf.keras.models.clone_model() method.\n",
        "\n",
        "Doing this will create the same architecture but reset the learned weights of the clone target (pretrained weights from the USE will remain but all others will be reset).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Lqp1N7ygRqoS",
        "outputId": "cb132feb-c8cf-4d56-ef56-620a7d6ddee0"
      },
      "source": [
        "# Clone model_6 but reset weights\n",
        "model_7 = tf.keras.models.clone_model(model_6)\n",
        "\n",
        "# Compile model\n",
        "model_7.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Get a summary (will be same as model_6)\n",
        "model_7.summary()"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model_6_USE\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "USE (KerasLayer)             (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_5 (Dense)              (None, 64)                32832     \n",
            "_________________________________________________________________\n",
            "dense_6 (Dense)              (None, 1)                 65        \n",
            "=================================================================\n",
            "Total params: 256,830,721\n",
            "Trainable params: 32,897\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CBBxJgvMSO30"
      },
      "source": [
        "Notice the layout of model_7 is the same as model_6. Now let's train the newly created model on our 10% training data subset.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tFH8ofW0SJoQ",
        "outputId": "ef6d37dd-0166-4fd2-8f7b-475b5c3e48c8"
      },
      "source": [
        "# Fit the model to 10% of the training data\n",
        "model_7_history = model_7.fit(x=train_sentences_10_percent,\n",
        "                              y=train_labels_10_percent,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels),\n",
        "                              callbacks=[create_tensorboard_callback(SAVE_DIR, \"10_percent_tf_hub_sentence_encoder\")])"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving TensorBoard log files to: model_logs/10_percent_tf_hub_sentence_encoder/20210923-071058\n",
            "Epoch 1/5\n",
            "22/22 [==============================] - 5s 108ms/step - loss: 0.6661 - accuracy: 0.6939 - val_loss: 0.6453 - val_accuracy: 0.6903\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 0s 23ms/step - loss: 0.5901 - accuracy: 0.8149 - val_loss: 0.5874 - val_accuracy: 0.7454\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 0s 16ms/step - loss: 0.5116 - accuracy: 0.8280 - val_loss: 0.5333 - val_accuracy: 0.7703\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 0s 17ms/step - loss: 0.4519 - accuracy: 0.8309 - val_loss: 0.5063 - val_accuracy: 0.7651\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 0s 16ms/step - loss: 0.4094 - accuracy: 0.8367 - val_loss: 0.4911 - val_accuracy: 0.7756\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yno9qR9XSb7f"
      },
      "source": [
        "\n",
        "Due to the smaller amount of training data, training happens even quicker than before.\n",
        "\n",
        "Let's evaluate our model's performance after learning on 10% of the training data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fgUrcD3rSVw5",
        "outputId": "e404362a-92cf-4aa4-e2ec-cbaad6070bc3"
      },
      "source": [
        "# Make predictions with the model trained on 10% of the data\n",
        "model_7_pred_probs = model_7.predict(val_sentences)\n",
        "model_7_pred_probs[:10]"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.24727035],\n",
              "       [0.8012104 ],\n",
              "       [0.9155853 ],\n",
              "       [0.32642192],\n",
              "       [0.5177269 ],\n",
              "       [0.81429243],\n",
              "       [0.82760113],\n",
              "       [0.8489264 ],\n",
              "       [0.83172596],\n",
              "       [0.11420089]], dtype=float32)"
            ]
          },
          "metadata": {},
          "execution_count": 89
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KJA6q_zDSdkn",
        "outputId": "25ee7fde-2974-46d0-b86e-0c958f761627"
      },
      "source": [
        "# Convert prediction probabilities to labels\n",
        "model_7_preds = tf.squeeze(tf.round(model_7_pred_probs))\n",
        "model_7_preds[:10]"
      ],
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=float32, numpy=array([0., 1., 1., 0., 1., 1., 1., 1., 1., 0.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cWFCaxreSfAR",
        "outputId": "43e919ff-df59-4006-d70f-16d6964408dc"
      },
      "source": [
        "# Calculate model results\n",
        "model_7_results = calculate_results(val_labels, model_7_preds)\n",
        "model_7_results"
      ],
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 77.55905511811024,\n",
              " 'f1_score': 0.7718737954958428,\n",
              " 'precision': 0.7815353725945549,\n",
              " 'recall': 0.7755905511811023}"
            ]
          },
          "metadata": {},
          "execution_count": 91
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2trlMJSMSgfY",
        "outputId": "0009baba-4747-4d50-903b-97bac029556f"
      },
      "source": [
        "#  Compare to baseline\n",
        "compare_baseline_to_new_results(baseline_results, model_7_results)"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Baseline accuracy: 79.27, New accuracy: 77.56, Difference: -1.71\n",
            "Baseline precision: 0.81, New precision: 0.78, Difference: -0.03\n",
            "Baseline recall: 0.79, New recall: 0.78, Difference: -0.02\n",
            "Baseline f1_score: 0.79, New f1_score: 0.77, Difference: -0.01\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NjtEMzGtSrCY"
      },
      "source": [
        "# Comparing the performance of each of our models¶\n",
        "Woah. We've come a long way! From training a baseline to several deep models.\n",
        "\n",
        "Now it's time to compare our model's results.\n",
        "\n",
        "But just before we do, it's worthwhile mentioning, this type of practice is a standard deep learning workflow. Training various different models, then comparing them to see which one performed best and continuing to train it if necessary.\n",
        "\n",
        "The important thing to note is that for all of our modelling experiments we used the same training data (except for model_7 where we used 10% of the training data).\n",
        "\n",
        "To visualize our model's performances, let's create a pandas DataFrame we our results dictionaries and then plot it."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299
        },
        "id": "VY3ZbqpESiOb",
        "outputId": "8fcb2bbc-69dd-4599-8bcc-0d5a1957ba04"
      },
      "source": [
        "# Combine model results into a DataFrame\n",
        "all_model_results = pd.DataFrame({\"baseline\": baseline_results,\n",
        "                                  \"simple_dense\": model_1_results,\n",
        "                                  \"lstm\": model_2_results,\n",
        "                                  \"gru\": model_3_results,\n",
        "                                  \"bidirectional\": model_4_results,\n",
        "                                  \"conv1d\": model_5_results,\n",
        "                                  \"tf_hub_sentence_encoder\": model_6_results,\n",
        "                                  \"tf_hub_10_percent_data\": model_7_results})\n",
        "all_model_results = all_model_results.transpose()\n",
        "all_model_results"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>79.265092</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>simple_dense</th>\n",
              "      <td>78.346457</td>\n",
              "      <td>0.788873</td>\n",
              "      <td>0.783465</td>\n",
              "      <td>0.780192</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lstm</th>\n",
              "      <td>77.165354</td>\n",
              "      <td>0.775137</td>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.768748</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gru</th>\n",
              "      <td>77.690289</td>\n",
              "      <td>0.778683</td>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.774880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bidirectional</th>\n",
              "      <td>76.115486</td>\n",
              "      <td>0.761042</td>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.760055</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conv1d</th>\n",
              "      <td>74.278215</td>\n",
              "      <td>0.742383</td>\n",
              "      <td>0.742782</td>\n",
              "      <td>0.741707</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_sentence_encoder</th>\n",
              "      <td>81.364829</td>\n",
              "      <td>0.814169</td>\n",
              "      <td>0.813648</td>\n",
              "      <td>0.812791</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_10_percent_data</th>\n",
              "      <td>77.559055</td>\n",
              "      <td>0.781535</td>\n",
              "      <td>0.775591</td>\n",
              "      <td>0.771874</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                          accuracy  precision    recall  f1_score\n",
              "baseline                 79.265092   0.811139  0.792651  0.786219\n",
              "simple_dense             78.346457   0.788873  0.783465  0.780192\n",
              "lstm                     77.165354   0.775137  0.771654  0.768748\n",
              "gru                      77.690289   0.778683  0.776903  0.774880\n",
              "bidirectional            76.115486   0.761042  0.761155  0.760055\n",
              "conv1d                   74.278215   0.742383  0.742782  0.741707\n",
              "tf_hub_sentence_encoder  81.364829   0.814169  0.813648  0.812791\n",
              "tf_hub_10_percent_data   77.559055   0.781535  0.775591  0.771874"
            ]
          },
          "metadata": {},
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RNTi1_jfSvab"
      },
      "source": [
        "# Reduce the accuracy to same scale as other metrics\n",
        "all_model_results[\"accuracy\"] = all_model_results[\"accuracy\"]/100"
      ],
      "execution_count": 94,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 546
        },
        "id": "qFiOZMLgS3kK",
        "outputId": "e4044dbb-a744-477b-9008-1e09b94aba7d"
      },
      "source": [
        "# Plot and compare all of the model results\n",
        "all_model_results.plot(kind=\"bar\", figsize=(10, 7)).legend(bbox_to_anchor=(1.0, 1.0));"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAIRCAYAAABpvyTfAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzde5iVdb3//9drOAgIqOAAKiConEZFUUTzEJ5PJZan1DzkrvhpKWW5yzLNzA5a6m+TtjeeLXGbmimmaVoK7awUUEAQEJVQBEFEQAk5zPv7x7rHFsPADDLM/Rnu5+O61sW6P+uetd6zLmbmtT6n2xEhAAAAICUVeRcAAAAA1EZIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOS3zeuHtt98+evXqldfLAwAANNiECRPeiYjKvOsoktxCaq9evTR+/Pi8Xh4AAKDBbP8z7xqKhuF+AAAAJIeQCgAAgOQQUgEAAJCc3OakAgAANGcTJkzo0rJly1sl7SE6/jZWtaSXVq9e/aV99913QV0nEFIBAAA+hpYtW97arVu3AZWVlYsrKioi73qak+rqai9cuLBq/vz5t0oaVtc5pH4AAICPZ4/KysqlBNSNV1FREZWVlUtU6oWu+5wmrAcAAGBLUkFA/fiy9269WZSQCgAAgOQwJxUAAKAR9Lr00X0b8/lm//RTExrz+ZobelIBAACwQatWrWry1ySkAgAANGNHHnnkrrvvvvuA3Xbbbfef//zn20vSAw880LGqqmpAv379qj7xiU/0laQlS5ZUnHLKKb369u1b1bdv36o777xzW0lq167doJrnuuOOO7Y7+eSTe0nSySef3OvMM8/sOXDgwP4XXHBB96effrrd3nvv3X/AgAFVgwYN6j9p0qStJGn16tUaPnx49z59+uzet2/fqh/96EddxowZ0+HII4/cteZ5f/e733U86qijdtVGYLgfAACgGRs9evTsrl27rnn//fc9aNCgqs997nPvXXjhhb2eeeaZ6f3791/59ttvt5CkSy+9dIeOHTuumTlz5jRJWrhwYYv6nnvevHmtJ06cOL1ly5Z69913K55//vnprVq10kMPPdThW9/6Vvcnnnji1euuu65yzpw5radNmza1VatWevvtt1tUVlau+drXvtbzrbfearnjjjuuvv322zufd95572zM90VIBQAAaMauueaaro8++ui2kjR//vxWI0eOrBwyZMiy/v37r5Skrl27rpGkcePGdbz33ntfq/m6ysrKNfU990knnbS4ZctSXHz33XdbfO5zn+s9e/bsNrZj1apVlqQ///nPHc8///yFrVq1UvnrnXbaaYtuueWWTl/96lcXTZw4sf2DDz74+sZ8X4RUAACAZur3v/99h7Fjx3YYP3789A4dOlQPGTKk36BBg5bPmDGjTUOfw/ZH9//1r3+5/LH27dtX19z/9re/vdPQoUOXPfnkk6/OmDGj9eGHH95vQ897wQUXLPrUpz61W5s2beKEE05YXBNiG4o5qQAAAM3Ue++912KbbbZZ06FDh+oXXnihzaRJk7ZesWJFxXPPPddh+vTprSWpZrh/6NChS2+44YYuNV9bM9zfuXPnVRMnTmyzZs0aPfzww9ut77WWLl3aonv37isladSoUdvXtB9xxBFLR40atX3N4qqa1+vVq9eqrl27rrruuut2GD58+EYN9Uv0pAIAADSKPLaMOvnkk5fcfPPNlbvsssvuu+yyy4q99trrgy5duqweOXLk7M9+9rO7VVdXq3PnzqueffbZV37yk5/MO++883r26dNn94qKivjud7/71rnnnvveD37wg7knnnjibp06dVq91157Lf/ggw/q7MT89re/Pf9LX/pS72uuuWbHo4466r2a9osvvnjhzJkzt+rfv//uLVu2jHPPPXfhd7/73YWSdPrppy+66aabWu6zzz4rNvZ7c0Q+F0oYPHhwjB8/PpfXBgAA2Bi2J0TE4PK2SZMmzd5rr702uoewSM4555yegwYNWn7xxRfX+T5NmjRp+7322qtXXY/RkwoAwCbodemj9Z4zu82ZG3x8z949632O+36yut5zBkx/ud5zgKay++67D2jbtm31qFGj3vg4X7/lh9Qrt2nAOUs2fx0AAAAFMnXq1E361MTCKQAAACSnQSHV9rG2Z9ieZfvSOh7vaftp2y/Ynmz7+MYvFQAAAEVRb0i13ULSTZKOk1Ql6QzbVbVO+56k+yJikKTTJf2ysQsFAABAcTSkJ3WIpFkR8VpErJR0r6QTa50Tkjpm97eR9FbjlQgAAICiacjCqZ0kla/KelPS/rXOuVLSH21fJGlrSUfW9US2h0saLkk9e9a/khEAAKDZuHKbfRv3+ZY0+b6rkjRu3Lh2t99+e+c777yzzlX5s2fPbnX++ef3ePzxx1+r6/HG0lgLp86QdGdEdJd0vKRf217nuSPi5ogYHBGDKysrG+mlAQAAsD6rV9e/fVm5T37yk8vXF1Cl0pWkNndAlRoWUudK6lF23D1rK/dFSfdJUkT8TVIbSdsLAAAAm82MGTNa9+7de/dhw4b13mWXXXY/9thjd1m2bFnFTjvttOcFF1ywU1VV1YDbb799uwcffLDj3nvv3b+qqmrAcccdt8uSJUsqJGns2LHtBg0a1L9fv35Ve+6554DFixdX/P73v+9w2GGH7SZJjz76aPv+/ftX9e/fv2rAgAFVixcvrpgxY0brPn367C5Jy5cv9ymnnNKrb9++VQMGDKh65JFHOkjSyJEjOx999NG7HnLIIX123nnnPc4///zuG/u9NWS4/3lJfWz3Vimcni6p9q7EcyQdIelO2wNUCqkLN7aYj6O+TZRnt6n/Ofa8a896z5ly7pSGlgQAANBkZs+e3WbUqFGzjz766A9OPfXUXj/72c8qJalz586rp02b9vK8efNannDCCbuOGzduZseOHasvu+yybj/84Q+7Xn311fM///nP7zp69OhXhw4duvzdd9+taN++fXX5c1933XXdRo4c+c+jjz76gyVLllS0a9euesGCBR89fs0113SxrZkzZ0574YUX2hx//PF9Xn311Zckadq0ae0mTZo0rW3bttW77bbbHpdccsnbu+2226qGfl/19qRGxGpJF0p6QtLLKq3in2r7KtvDstO+KenLtidJ+l9JX4i8rrcKAABQIN26dVt59NFHfyBJZ5999qJnn322vSSdc845iyXpmWee2frVV19tM2TIkP79+/evuvfeezvPmTOn9eTJk9t06dJl1dChQ5dLUqdOnapbtWq11nMfcMAB719yySU9rr766i7vvPNOi9qPP/vss+3PPvvsRZI0aNCgFTvuuOPKKVOmtJGkgw8+eGnnzp3XtGvXLnbbbbcVr7766lYb83016IpTEfGYpMdqtV1Rdn+apIM25oWRM67EBQDAFsF2nccdOnSolqSI0MEHH7z0kUceeb38vOeee65tfc/94x//eP5nPvOZJQ8//PA2hxxySP9HH330lXbt2lXX93WS1Lp16486LFu0aBGrVq3yhs6vjStONdDL/Qds8AYAAJCHefPmtX7qqae2lqTRo0d3OvDAA98vf/zQQw/9YPz48e1feumlrSRp6dKlFZMnT95q4MCBKxYsWNBq7Nix7SRp8eLFFatWrT0aP3Xq1K2GDBnyrx/96EfzBw4c+MFLL7201kTKgw466P277767kyRNnjx5q3nz5rUeOHDgisb4vhrUk4rmpb55ulLjzNVlni4AAGVy2jKqV69eK37xi190GT58eLs+ffqsuOSSSxbeeuutXWoe33HHHVePGjVq9umnn77LypUrLUnf//735w4cOPDD0aNHvzpixIieK1asqGjTpk31uHHjZpY/97XXXtvl2Wef7Wg7+vXr969TTjllyZw5cz4a8//Wt7614Jxzztm5b9++VS1atNCoUaNmt23btlGmfBJSAQBA02PaWaNp2bKlHn744bWG8ufOnbtWT9KwYcOWDRs27OXaXzt06NDlkyZNml7e9ulPf3rZpz/96WWSdNddd62zFVW/fv1WvvLKK1MlqV27dvHAAw/Mrn3OiBEjFklaVHP89NNPz9q474qQik3QkGkOA6av8/OA5oY/JACAHBBSgYJjGzcAaL7KezW3NIRUoFx9vYb0GH5s9LwDxcIHYGwqQioKozEWlPELEwDSwgfgLRdbUAEAACA59KQCjYxP9QAAbDpCKgAAQCPY8649923M55ty7pRc9l0dOXJk5/Hjx2/9q1/9as43vvGNHdu3b7/mqquuerup62C4HwAAYAtQXV2tNWvW5F1Go6EnFQDqUP/K5DPrfY49e/es9xwW2gHYFDNmzGh9zDHH9B00aND7U6ZM2frEE09894knnth25cqV/tSnPvXeDTfc8JYk3XjjjZ1HjhzZ1bYGDBjwr4ceeuj1e+65Z5uf/vSnO6xatapiu+22W/2b3/zmtR49eqzO+3uqQUgFgBzVN4eZ+csA6jNnzpytbrvttteXLFny7v3337/d5MmTX44IHXnkkbv94Q9/aF9ZWbn65z//+Q5/+9vfpu+www6r33777RaSdNRRR71/+umnT6+oqND111+//VVXXdXtlltueTPv76cGIRUAAKAZ22GHHVYeccQRHwwfPrz7uHHjOlZVVVVJ0vLlyyumT5/eZuLEiRUnnHDC4h122GG1JHXt2nWNJL3++uutP/OZz3RfuHBhq5UrV1b06NHjwzy/j9qYkwoAANCMtWvXrlqSIkJf//rX502fPn3a9OnTp82ZM+eliy+++J31fd2FF17Y8ytf+cqCmTNnTrvxxhv/+eGHHyaVC5MqBgAAAB/Pcccdt/TXv/719kuWLKmQpNdff73V3LlzWx5zzDFLH3nkke3mz5/fQpJqhvuXLVvWomfPnqsk6c477+ycX+V1Y7gfAACgEeS1ZVSNk046aenUqVPb7Lfffv2lUg/r6NGjXx88ePCKb37zm/MOOeSQ/hUVFbHHHnss/+1vfzv7sssue+uMM87YdZtttll98MEHL5szZ85WedZfGyEVAACgmerXr9/KV155ZWrN8eWXX77g8ssvX1D7vIsuumjRRRddtKi87ayzznrvrLPOeq/2uSNGjFgkaZEkXX/99W9thrIbhOF+AAAAJIeeVABAg9S3d6zUOPvHsncsAImQCgBITH17x0rsHwsUAcP9AAAASA4hFQAAAMkhpAIAACA5zEkFAABoBC/3H7BvYz7fgOkv57rvat7oSQUAAGimrr766i677LLL7sccc8yue++9d//WrVvvc8UVV3TNu67GQE8qAABAM3XbbbdVPvXUUzPbtGkTs2bNav3AAw9s19Q1rFq1Sq1atWr056UnFQAAoBk688wze7755ptbHXfccX1uvfXWTkOHDl3eqlWrqO/rli5dWnHooYfu1q9fv6o+ffrsfsstt2wnSWPHjm03aNCg/v369avac889ByxevLhi+fLlPuWUU3r17du3asCAAVWPPPJIB0kaOXJk58MPP3y3Aw44oO+BBx7Yb+nSpRWnnnpqrz333HPAgAEDqu6+++5tN/X7oycVAACgGbrnnnvmjB07dpuxY8fO3GGHHVY39OsefPDBjt26dVv1zDPPzJKkRYsWtVixYoU///nP7zp69OhXhw4duvzdd9+taN++ffXVV1/d1bZmzpw57YUXXmhz/PHH93n11VdfkqSpU6e2mzx58tSuXbuuufDCC3c67LDDlt5///2z33nnnRaDBw8eMGzYsKUdO3as/rjfHz2pAAAABbLPPvv86y9/+UvHCy64YKfHH3+8fefOnddMnjy5TZcuXVYNHTp0uSR16tSpulWrVnr22Wfbn3322YskadCgQSt23HHHlVOmTGkjSYcccsjSrl27rpGkZ555puMNN9ywQ//+/asOPvjgfh9++KFnzZrVelPqpCcVAACgQAYOHPjhxIkTp/32t7/d5vLLL9/pqaeeWnraaae9t7HP065du496SSNCDzzwwKy99trrw8aqk5AKAADQCJrLllGzZ89u1aVLl9Vf+cpX3t1uu+3W3HbbbdtfffXV8xcsWNBq7Nix7YYOHbp88eLFFe3bt68+6KCD3r/77rs7DRs2bNnkyZO3mjdvXuuBAweu+Mc//tGu/DkPO+ywpdddd13XO++8c05FRYX++te/tj3ooIP+tSl1ElIBAACauTlz5rTcb7/9qj744IMWtmPUqFFdX3755Zc6deq0zpzQCRMmtP3Od77TvaKiQi1btoxf/vKX/2zTpk2MHj361REjRvRcsWJFRZs2barHjRs381vf+taCc845Z+e+fftWtWjRQqNGjZrdtm3bdRZn/fSnP31r+PDhPfv3719VXV3tHj16fPj000/P2pTviZAKAADQTM2dO3dKzf233357ckO+5uSTT1568sknT6vdPnTo0OWTJk2aXrv9gQcemF27bcSIEYskLao5bt++fdxzzz3/bGjdDcHCKQAAACSnQT2pto+V9F+SWki6NSJ+WuvxGyQdlh22k9QlIjZ5fywAAAB8PPPnz29x6KGH9qvd/swzz8zo1q3bmjxq2hj1hlTbLSTdJOkoSW9Ket72mIj4qJs4Ii4uO/8iSYM2Q60AAAApqa6urnZFRUW9G+jnoVu3bmumT5++zrB+Kqqrqy1pvfuoNmS4f4ikWRHxWkSslHSvpBM3cP4Zkv53o6oEAABofl5auHDhNlnYwkaorq72woULt5H00vrOachw/06S3ig7flPS/nWdaHtnSb0l/Xk9jw+XNFySevbs2YCXBgAASNPq1au/NH/+/Fvnz5+/h1jns7GqJb20evXqL63vhMZe3X+6pAcios55DhFxs6SbJWnw4MFJdo0DAAA0xL777rtA0rC869hSNST1z5XUo+y4e9ZWl9PFUD8AAAA2UUNC6vOS+tjubbu1SkF0TO2TbPeXtJ2kvzVuiQAAACiaekNqRKyWdKGkJyS9LOm+iJhq+yrb5V3cp0u6NyIYxgcAAMAmadCc1Ih4TNJjtdquqHV8ZeOVBQAAgCJjJRoAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAyWlQSLV9rO0ZtmfZvnQ955xme5rtqbbvadwyAQAAUCQt6zvBdgtJN0k6StKbkp63PSYippWd00fSdyQdFBGLbXfZXAUDAABgy9eQntQhkmZFxGsRsVLSvZJOrHXOlyXdFBGLJSkiFjRumQAAACiShoTUnSS9UXb8ZtZWrq+kvrb/avvvto+t64lsD7c93vb4hQsXfryKAQAAsMVrrIVTLSX1kXSopDMk3WJ729onRcTNETE4IgZXVlY20ksDAABgS9OQkDpXUo+y4+5ZW7k3JY2JiFUR8bqkmSqFVgAAAGCjNSSkPi+pj+3etltLOl3SmFrnPKRSL6psb6/S8P9rjVgnAAAACqTekBoRqyVdKOkJSS9Lui8iptq+yvaw7LQnJC2yPU3S05L+MyIWba6iAQAAsGWrdwsqSYqIxyQ9VqvtirL7Iekb2Q0AAADYJFxxCgAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSnQSHV9rG2Z9ieZfvSOh7/gu2Ftl/Mbl9q/FIBAABQFC3rO8F2C0k3STpK0puSnrc9JiKm1Tr1NxFx4WaoEQAAAAXTkJ7UIZJmRcRrEbFS0r2STty8ZQEAAKDIGhJSd5L0Rtnxm1lbbSfbnmz7Ads96noi28Ntj7c9fuHChR+jXAAAABRBYy2cekRSr4gYKOlJSXfVdVJE3BwRgyNicGVlZSO9NAAAALY0DQmpcyWV94x2z9o+EhGLIuLD7PBWSfs2TnkAAAAoooaE1Ocl9bHd23ZrSadLGlN+gu0dyg6HSXq58UoEAABA0dS7uj8iVtu+UNITklpIuj0iptq+StL4iBgjaYTtYZJWS3pX0hc2Y80AAADYwtUbUiUpIh6T9FittivK7n9H0ncatzQAAAAUFVecAgAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5DQqpto+1PcP2LNuXbuC8k22H7cGNVyIAAACKpt6QaruFpJskHSepStIZtqvqOK+DpK9J+kdjFwkAAIBiaUhP6hBJsyLitYhYKeleSSfWcd4PJV0jaUUj1gcAAIACakhI3UnSG2XHb2ZtH7G9j6QeEfHohp7I9nDb422PX7hw4UYXCwAAgGLY5IVTtiskXS/pm/WdGxE3R8TgiBhcWVm5qS8NAACALVRDQupcST3KjrtnbTU6SNpD0jO2Z0s6QNIYFk8BAADg42pISH1eUh/bvW23lnS6pDE1D0bEkojYPiJ6RUQvSX+XNCwixm+WigEAALDFqzekRsRqSRdKekLSy5Lui4iptq+yPWxzFwgAAIDiadmQkyLiMUmP1Wq7Yj3nHrrpZQEAAKDIuOIUAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASE6DQqrtY23PsD3L9qV1PH6+7Sm2X7T9f7arGr9UAAAAFEW9IdV2C0k3STpOUpWkM+oIofdExJ4RsbekayVd3+iVAgAAoDAa0pM6RNKsiHgtIlZKulfSieUnRMTSssOtJUXjlQgAAICiadmAc3aS9EbZ8ZuS9q99ku2vSvqGpNaSDq/riWwPlzRcknr27LmxtQIAAKAgGm3hVETcFBG7Svq2pO+t55ybI2JwRAyurKxsrJcGAADAFqYhIXWupB5lx92ztvW5V9JnNqUoAAAAFFtDQurzkvrY7m27taTTJY0pP8F2n7LDT0l6pfFKBAAAQNHUOyc1IlbbvlDSE5JaSLo9IqbavkrS+IgYI+lC20dKWiVpsaRzN2fRAAAA2LI1ZOGUIuIxSY/Varui7P7XGrkuAAAAFBhXnAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOQ0KqbaPtT3D9izbl9bx+DdsT7M92fafbO/c+KUCAACgKOoNqbZbSLpJ0nGSqiSdYbuq1mkvSBocEQMlPSDp2sYuFAAAAMXRkJ7UIZJmRcRrEbFS0r2STiw/ISKejojl2eHfJXVv3DIBAABQJA0JqTtJeqPs+M2sbX2+KOkPdT1ge7jt8bbHL1y4sOFVAgAAoFAadeGU7bMkDZb0s7oej4ibI2JwRAyurKxszJcGAADAFqRlA86ZK6lH2XH3rG0tto+UdJmkoRHxYeOUBwAAgCJqSE/q85L62O5tu7Wk0yWNKT/B9iBJoyQNi4gFjV8mAAAAiqTekBoRqyVdKOkJSS9Lui8iptq+yvaw7LSfSWov6X7bL9oes56nAwAAAOrVkOF+RcRjkh6r1XZF2f0jG7kuAAAAFBhXnAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJaVBItX2s7Rm2Z9m+tI7HP2l7ou3Vtk9p/DIBAABQJPWGVNstJN0k6ThJVZLOsF1V67Q5kr4g6Z7GLhAAAADF07IB5wyRNCsiXpMk2/dKOlHStJoTInycWtoAAB34SURBVGJ29lj1ZqgRAAAABdOQ4f6dJL1Rdvxm1rbRbA+3Pd72+IULF36cpwAAAEABNOnCqYi4OSIGR8TgysrKpnxpAAAANCMNCalzJfUoO+6etQEAAACbRUNC6vOS+tjubbu1pNMljdm8ZQEAAKDI6g2pEbFa0oWSnpD0sqT7ImKq7atsD5Mk2/vZflPSqZJG2Z66OYsGAADAlq0hq/sVEY9JeqxW2xVl959XaRoAAAAAsMm44hQAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAyWlQSLV9rO0ZtmfZvrSOx7ey/Zvs8X/Y7tXYhQIAAKA46g2ptltIuknScZKqJJ1hu6rWaV+UtDgidpN0g6RrGrtQAAAAFEdDelKHSJoVEa9FxEpJ90o6sdY5J0q6K7v/gKQjbLvxygQAAECROCI2fIJ9iqRjI+JL2fHZkvaPiAvLznkpO+fN7PjV7Jx3aj3XcEnDs8N+kmY01jeyibaX9E69ZxUP78u6eE/qxvtSN96XuvG+rIv3pG4pvS87R0Rl3kUUScumfLGIuFnSzU35mg1he3xEDM67jtTwvqyL96RuvC91432pG+/LunhP6sb7UmwNGe6fK6lH2XH3rK3Oc2y3lLSNpEWNUSAAAACKpyEh9XlJfWz3tt1a0umSxtQ6Z4ykc7P7p0j6c9Q3jwAAAABYj3qH+yNite0LJT0hqYWk2yNiqu2rJI2PiDGSbpP0a9uzJL2rUpBtTpKbgpAI3pd18Z7UjfelbrwvdeN9WRfvSd14Xwqs3oVTAAAAQFPjilMAAABIDiEVAAAAySGkAgAAIDmEVAAAACSnSTfzT43tgyX1iYg7bFdKah8Rr+ddV55st5P0TUk9I+LLtvtI6hcRv8+5tNzYHizpMkk7q/QzY0kREQNzLQxJsd1pQ49HxLtNVUsqbP9C0npX50bEiCYsJym2W0h6KiIOy7uW1GR/d34iqUpSm5r2iNglt6KQi8KGVNvflzRYpcuz3iGplaS7JR2UZ10JuEPSBEmfyI7nSrpfUmFDqqTRkv5T0hRJ1TnXkgzby/TvANJapZ+hDyKiY35V5WqCSu+H63gsJBXxD+z47N+DVAocv8mOT5U0LZeKEhERa2xX294mIpbkXU9i7pD0fUk3SDpM0nli5LeQChtSJX1W0iBJEyUpIt6y3SHfkpKwa0R8zvYZkhQRy23X9Ue3SBZm+wGjTER89POS/R85UdIB+VWUr4jonXcNqYmIuyTJ9gWSDo6I1dnx/0j6S561JeJ9SVNsPynpg5rGIvcwZ9pGxJ9sOyL+KelK2xMkXZF3YWhaRQ6pKyMibIck2d4674ISsdJ2W2U9ZLZ3lfRhviXl7vu2b5X0J5W9FxHxYH4lpSW7wtxD2QjFpXnXkzfb20nqo7WHKsflV1HutpPUUaWLvUhS+6yt6B7Mbljbh7YrJL2SXUxorkr/Z1AwRQ6p99keJWlb21+W9B+Sbsm5phR8X9LjknrYHq3SMN0Xcq0of+dJ6q/ScHbNcH+o4H9cbJ9Udlih0vSZFTmVkwzbX5L0NUndJb2oUu/y3yQdnmddOfuppBdsP63SdIhPSroy14oSEBF3ZZ0CPSNiRt71JORrktpJGiHphyoN+Z+Ta0XIRaGvOGX7KElHq/RL84mIeDLnkpJgu7NKf1gt6e8R8U7OJeXK9oyI6Jd3HamxfUfZ4WpJsyXdEhEL8qkoDbanSNpPpZ+dvW33l/TjiDipni/dotnuJmn/7PAfETE/z3pSYPsEST+X1DoietveW9JVETEs59JyZfvUiLi/vjZs+QodUrEu2wdJejEiPrB9lqR9JP1XNi+okLIw9rOIKPRCj3LZyuQREXFD3rWkxvbzEbGf7Rcl7R8RH9qeGhG7511bSmz3j4jpedeRp2ye5eGSnomIQVnbSxGxR76V5cv2xIjYp742bPkKO9yfDVVeI6mLSj2GNdsKFXVlco3/lrSX7b0kfUPSbZJ+JWlorlXl6wBJL9p+XaU5qYXfgipbmXyGSqtvsbY3bW8r6SFJT9peLKmwH/I24I+SeuZdRM5WRcSSWmtTC7uDiO3jJB0vaSfbI8se6qjSaA0KprAhVdK1kk6IiJfzLiQxq7MFZSdKuikibrP9xbyLytmxeReQqL/avlGlbYXKVyZPzK+k/EXEZ7O7V2ZzMLdRaZ534dQKGms9JGnbpqwlUVNtnympRbY36AhJz+ZcU57eUmnbsmEqbelWY5mki3OpCLkq7HC/7b9GRNH3RF2H7bEq/UE9T6XFDQskTYqIPXMtLEe2fx0RZ9fXVjRZAJP+vVdqTQ9zkRcISfpoOkRXlXUERMSc/CrKR7aX7jdV9w4h10XE9k1cUlKyi6dcprK1EZJ+GBGFXoBou1VErMq7DuSvyCH1vyR1U2lIjm2FMtnihjMlPR8Rf7HdU9KhEfGrnEvLTe25UFkAmRIRVTmWlTvb39Tam9eHpKWSxkfEi7kVljPbF6m0S8bbKtsNoojTQ2z/WdL3ImKd3kHbr7O3LOrCFadQo8gh9Y46miMi/qPJi0GSbH9H0ncltZW0vKZZ0kpJN0fEd/KqLQW271Fp26kxKr0vn5Y0WVIvSfdHxLX5VZcf27NUWjC1KO9a8pZdKnZFRCyv9+QCsf2INny52KKv7v8//fuKUycou+JURLCZf8EUNqSibiwoW5ftnxQ9kNbF9jhJx0fE+9lxe0mPqjSHd0JRe5qzaRBH1VxdCR/9Xnk0Iop+YRBJku2ahagnqTSid3d2fIaktyOi0PMvbU+IiH1tT6mZalbTlndtaFqFWzhl+1sRca3tX6iOT7Jcjo4FZXX4ve2t2ZZrHV209lzDVZK6RsS/bBc5jLwm6Rnbj2rtqUTX51dS7k6QdEP2weY3kh4vcoiPiLGSZPu6iBhc9tAjtsfnVFZKuOIUJBUwpEqqCV/8Iqjb2wTUdZRvy/VNSbeKbbkkabSkf9h+ODs+QdI92SWGi7yn7Jzs1jq7FV5EnGe7laTjVOotvMn2kxHxpZxLy9vWtneJiNckyXZvSVyie90rTh0u6dxcK0IuGO7HWlhQtq6ahVO2r5A0N9uWi42lJdkerNKlcyXprxHBh79MNv1BNdMhUFq1rdJ0kPMkfZLV/T5W0s0q9b5b0s6ShkfEH3MtDEhE4UIqE9Y3jAVl62JbLmwM23tI+rWkTlnTO5LOiYip+VWVr2yT9s9JOlTSM5Luk/THIg/517C9laT+2eH0Is/b5e8zaitiSN3gEG3NXCGgBttyYWPYflbSZRHxdHZ8qKQfR8SBuRaWI9v/q9Jc1D8UOYTVlvUsX6DSh1+pFOBHFXWPUBaUobbChdRytttK6hkRM/KuJRW2+6o0B7NrROxhe6CkYRFxdc6lAc2C7UkRsVd9bYDtWyW1knRX1nS2pDVFn6tre3ytBWV1tmHLV5F3AXmxfYKkF5VdrtD23rbH5FtVEm6R9B2VVmorIiZLOj3XinJie5ntpXXcltlemnd9SNZrti+33Su7fU+lOYeFZfsk26/YXsLP0Fr2i4hzI+LP2e08SfvlXVQCtrb90cb9LCgrriKu7q9xpaQhKg2vKCJezH4Qiq5dRDxnu7ytkPPGIqJD3jWgWfoPST+QVLPY8C9ZW5GxtV3d1tjeNSJelaQsmK3JuaYUXKzSNm5rLSjLtyTkocghdVVELKkVxoo79+Hf3rG9q7L3wvYpkublWxLQfETEYpW2zsG/sbVd3f5T0tO1wth5+ZaUv4h4PLs0ap0LymwfFRFP5lMdmlJh56Tavk3SnyRdKulklf6otIqI83MtLGfZJ/mbJR0oabGk1yWdFRGz86wLSJ3t/z8ivr6+FcpFXpnM1nbrl63u75cdzmBhWf3YArA4ihxS20m6TNLRKn2CfULSDyNiRa6FJSLbkL0iIpblXQvQHNjeNyImrG8HkSLvHMLWdnWz/VVJoyPivex4O0lnRMQv860sbbZfiIhBedeBza+wIbWc7RaSto6Iwk7kt/2NDT1e8Es6Ag1m+2sR8V/1tQG2X4yIvWu1EcDqQU9qcRR5df89tjtmPYZTJE2z/Z9515WjDtltsEr79u2U3c5X6Vr1ABqmrss3fqGpi0iJ7e62f2d7QXb7re3uedeVgBYuWxiRdZhwKV0gU+SFU1URsdT25yX9QaW5qRMk/SzfsvIRET+QJNvjJO1TM8xv+0pJj+ZYGtAs2D5DpYs+9K61nV0HSe/mU1Uy7pB0j6RTs+OzsrajcqsoDY9L+o3tUdnx/5e1FZrtrWrPza3VNrvpq0IeihxSW2VX+/iMpBsjYpVt5j5IXSWtLDtembUB2LBnVdoJY3tJ15W1L5M0OZeK0lEZEeXzUu+0/fXcqknHt1UKphdkx09KujW/cpLxN607gvdRW0Sc1OQVIRdFDqmjVPo0NknSONs7SyrsnNQyv5L0nO3fZcefkXRnfuUAzUNE/FPSP7PRmbdqFmFmV7brrmL3/iyyfZak/82Oz5C0KMd6khAR1Spd4e+/864lBdklqHeS1Nb2IJUWNUtSR0ntcisMuWHhVBnbLSOikBvXl7O9j6RDssNxEfFC2WPbZftAAqiD7fGSDoyIldlxa0l/jYjCXkko6wT4haRPqLQ917OSLoqIN3ItLGe2D1LpwjI7q9RpZJV2PdhlQ1+3pbJ9rkrztwdLGl/20DJJd7JlWfEUOqTa/pSk3SW1qWmLiKvyqyh9rKoENmw9K7YnRcReedWUN9t3Sfp6zQdc250k/ZwtqDxdpasrTVDZlaYiotC9zLZPjojf5l0H8lfY4X7b/6PS8MFhKs0BOkXSc7kW1Ty4/lOAQltoe1hEjJEk2ydKeifnmvI2sHwEJiLezYZzi25JRPwh7yIS9HvbZ0rqpbKcQidS8RQ2pKo0HDfQ9uSI+IHt61Ra5Y8NK27XO9Aw50sabfsmlX5e3pR0Tr4l5a6ifKpQ1pNa5L8/NZ62/TNJD2rtK3FNzK+kJDwsaYlKPcxcgavAivxL4l/Zv8tt76jSJP4dcqwHwBYgIl6VdIDt9tnx+zmXlILrJP3N9v3Z8amSfpRjPanYP/t3cFlbSDo8h1pS0j0ijs27COSvyCH197a3lXStSp/WJLb+aAiG+4ENsN1V0o8l7RgRx9mukvSJiLgt59JyExG/yhaU1YSvkyJiWp41pSAiDsu7hkQ9a3vPiJiSdyHIV2EXTmXbwlyg0ir2kPQXSf9ds21Mkdk+WFKfiLjDdqWk9hHxevZYp4go+sbkwHrZ/oNKG9VfFhF72W4p6YWI2DPn0pAYPtDUzfY0SbtJel2l4f6aXQ8G5loYmlyRQ+p9Km1rcXfWdKakbSLitPyqyp/t76s09NQvIvpmUyHuj4iDci4NaBZsPx8R+5Vfg72uFf8AH2jqlm1Zto5sL2IUSEXeBeRoj4j4YkQ8nd2+LGmPvItKwGclDZP0gSRFxFsqXdYRQMN8YLuzskWGtg9QaREIUNv2EXGfpGpJyvbpXrPhL9nyZWG0h6TDs/vLVey8UlhFnpM60fYBEfF3SbK9v9bePLioVkZE1Fwi1vbWeRcENDPfkDRG0q62/yqpUqUt7oDa+EBTh/IRPZV6mlupNOrJiF7BFC6k2p6i0i+EVipNzp6THe8saXqetSXiPtujJG1r+8uS/kPSLTnXBDQLtltIGprd+qk0l25GRKzKtTCkig80dfuspEGSJkqlET3bjOgVUOHmpK5vrksN5rxIto+SdLRKf2CfiIgncy4JaDZsPxcRQ/KuA81DNg+1zg80to8q4u/fmp+hmiscZiN6f2PhVPEULqQCwOZk+waVRmp+o2xut8QG7dh4Rb0Mte1LJPWRdJSkn6g0ondPRPwi18LQ5AipkCTZXqa6ryZVs/VHxyYuCWiWbD9dR3NERNE3aMdGKt8homgY0YNESAUAIEkF7kntLWlezb7l2b7mXSNidq6FockVbuEU6md7H0kHq9Sz+n8R8ULOJQHJs31WRNxt+xt1PR4R1zd1TUAzdb+kA8uO12Rt++VTDvLCvmNYi+0rJN0lqbOk7SXdaft7+VYFNAs127V1WM8N2Fiz8y4gJy0jYmXNQXa/dY71ICcM92MttmdI2qvWMMuLEdEv38oAYMtiu52kb0rqGRFftt1Hpav9/T7n0nJl+0lJv4iIMdnxiZJGRMQR+VaGpsZwP2p7S1IbSSuy460kzc2vHKB5sD1yQ49HxIimqgXNxh2SJkj6RHY8V6Vh7UKHVEnnSxpt+8bs+E1JZ+dYD3JCSEVtSyRNzT7JhkpbgDxX8weYP7TAek3I/j1IUpVKW1BJ0qmSpuVSEVK3a0R8zvYZkhQRy20776LylF0Q44KIOMB2e0mKiPdzLgs5IaSitt9ltxrP5FQH0KxExF2SZPsCSQdn12GX7f+R9Jc8a0OyVmZTqmoui7qrpA/zLSlfEbHG9sHZfcJpwRFSsZaaP7QAPrbtJHWU9G523D5rA2r7vqTHJfWwPVqlXvgv5FpRGl6wPUalqQ/lF8R4ML+SkAdCKtZi+9OSfihpZ5X+f7CZP7BxfqrSH9mnVfr5+aSkK3OtCEmKiCdtT5R0gEr/V74WEe/kXFYK2khaJKn8AhghiZBaMKzux1psz5J0kqQpwX8O4GOx3U3S/tnhPyJifp71IE22PyvpzxGxJDveVtKhEfFQvpUBaWCfVNT2hqSXCKjAxrHdP/t3H0k7qvSz9IakHbM2oLbv1wRUSYqI91SaAlBotvva/pPtl7LjgezXXUz0pGIttvdTabh/rMom8HO1HGDDbN8cEcOzYf7yX6w1U2YOX8+XoqBsT46IgbXapkTEnnnVlALbYyX9p6RRETEoa3spIvbItzI0NXpSUduPJC1XaU4QV8sBGigihmd3j5f0qErbub0naUzWBtQ23vb1tnfNbtfr31uZFVm7iHiuVtvqXCpBrlg4hdp25NMqsEnukrRUUs3m/mdK+pWk03KrCKm6SNLl+veeuk9K+mp+5STjnWw7rpqtuU6RNC/fkpAHhvuxFtvXSnoqIv6Ydy1Ac2R7WkRU1dcGoG62d5F0s6QDJS2W9Lqkz0fEP3MtDE2OkIq12F4maWuV5qOuEltQARvF9t2SboyIv2fH+0v6akSck29lSI3tvpIukdRLZSObzF8usb21pIqIWJZ3LcgHIRUAGoHtKSoNT7aS1E/SnOx4Z0nT6UlFbbYnSfofleahrqlpj4hCz0u13VmlXQ4OVuln6P8kXRURi3ItDE2OkApJpe1zImL6+rbKiYiJTV0T0JzY3nlDjzNUidpsT4iIffOuIzW2n5Q0TtLdWdPnVdo/9sj8qkIeCKmQtM72OTU++s/B8BMANC7bV0paIOl3WnvLv3fX9zVFUNd2U2zNVUyEVKzF9mmSHo+IpbYvl7SPpB/SkwoAjcv263U0R0Ts0uTFJCTbius5SfdlTadIGhIRl+RXFfJASMVaajaXtn2wSpv6/1zSFRGxfz1fCgDAJitbwFszT7eFpA+y+yzkLRA280dtNb8UPiXploh4VFLrHOsBgC2S7Xa2v2f75uy4j+1P511X3iKiQ0RURESr7FaRtXWIiI62d8+7RjQNQipqm2t7lKTPSXrM9lbi/wkAbA53SFqp0n6gkjRX0tX5ldNs/DrvAtA0CB+o7TRJT0g6JiLek9RJpWsoAwAa164Rca1Ke1IrIpartDc1Noz3qCC4LCrWkv2SfLDseJ64HB0AbA4rbbfVvy//uavKVvljvVhMUxCEVAAA8nGlpMcl9bA9WtJBks7LtSIgIazuBwAgJ9nVlQ5QaQj77xHxTs4lJc/23yPigLzrwOZHSAUAIAe2/xQRR9TXViS2t5F0rKSdsqa5kp7I1kigYFg4BQBAE7LdxnYnSdvb3s52p+zWS/8OZ4Vj+xxJEyUdKqlddjtM0oTsMRQMPakAADQh21+T9HVJO6rUU1izWn2pSvtT35hXbXmyPUPS/rV7TW1vJ+kfEdE3n8qQF0IqAAA5sH1RRPwi7zpSYXumpP0iYkmt9m0kjY+IPvlUhrywuh8AgBxExC9sHyipl8r+HkfEr3IrKl8/kjTR9h8lvZG19ZR0lEqX6UbB0JMKAEAObP9a0q6SXtS/L0kdETEiv6rylQ3tH6N1F04tzq8q5IWQCgBADmy/LKkq+EMM1InV/QAA5OMlSd3yLqI5sD0l7xrQ9JiTCgBAPraXNM32cyq7HGpEDMuvpPzYPml9D4kwX0iEVAAA8nFl3gUk5jeSRkuqa/pDmyauBQlgTioAADmxvbOkPhHxlO12klpExLK868qD7QmSzo2Il+p47I2I6JFDWcgRc1IBAMiB7S9LekDSqKxpJ0kP5VdR7r6u0gUN6vLZpiwEaSCkAgCQj69KOkhZMIuIVyR1ybWiHEXEXyJiznoeG19z3/Z3mq4q5ImQCgBAPj6MiJU1B7Zbqu75mFjbqXkXgKZBSAUAIB9jbX9XUlvbR0m6X9IjOdfUHDjvAtA0WDgFAEAObFdI+qKko1UKXk9IupXN/TfM9sSI2CfvOrD5EVIBAMiZ7U6SukfE5LxrSZ3tFyJiUN51YPNjuB8AgBzYfsZ2xyygTpB0i+0b8q6rGbg/7wLQNAipAADkY5uIWCrpJEm/ioj9JR2Rc025s72L7Udsv2N7ge2Hbe9S83hE/DjP+tB0CKkAAOSjpe0dJJ2m/9fe/YPYUcVRHD8nKIi4mwRExM40AYsVA2JhkU7SKBgMsbKwi2CpMa1NSLATwU5Qif9ABC3EQrDQJhpxI1gkEAQbmwQTLITASTFvdViem4fg/d3s/X7gsW9mtji86sedO2ekL6rDdOScpI81vQr1IU0rpx+UJkIJhlQAAGq8rulhqctJzi9WCy8VZ+rBvUneS3Jz8XlfvBZ1SDw4BQBAh2yfSnK6Okcri725knRS0jVJH2rqjT0uaX8SSvwHw5AKAECHRqtasn1F01C6rAc1SQ4sOY9d7K7qAAAAYKmhSuuTPFydAX1hSAUAoE9D3uq0/cKy80nebZ0FtRhSAQDo01ArqTOPz77fo6mW64IkhtTBMKQCANCnIUvrk7w8P7a9T9NDVBgMFVQAABSgtH5lf0piv+qAWEkFAKDGOUlvSXp2cfy8ptL6J8oSdcD25/pnP+4eSY9oKvfHYKigAgCggO3NJBvbzv2U5NGqTD2wfXh2eFPSr0l+q8qDOgypAAA0RGk9sBqGVAAAGqK0fme2j0o6I+kBTb+RNf0u66XB0BxDKgAA6Ibty5KeTvJLdRbU4sEpAAAKUFr/r35nQIXESioAACVsvzk7/Lu0PslzRZFKLW7zS9JhSQ9K+kzSX1vXk3xakQt1GFIBAOjAVml9kiPVWSrYfmeHy0nyYrMw6AJDKgAAHbB9t6SfkxysztIz26eSnK7Ogf8fe1IBAChAaf1/dkwSQ+oAGFIBAKjxxuw7pfWrW1bdhV2IIRUAgAJJvqnOcIdin+Ig9lQHAABgRLaP2r5k+w/b123fsH29OtcdgJXUQTCkAgBQ46ykZ5LsTbKeZG3ktyrZPrP4e+w2//pJgzjoAE/3AwBQwPa3SZ6sztEL2xclbUj6Icmh6jyox55UAAAampXWf2/7I1Fav+VLSdck3bdt24M19aQOu8o8KlZSAQBoiNL6ndn+KslT286dTfJqVSbUYEgFAKBDo5bW276w/Xa/7c0kG1WZUIMHpwAA6NPtHiDaVWyfWOxLPWh7c/a5IulidT60x0oqAAAdsv1jkseqc7Rie6+k/ZreJvXa7NKNJFdrUqESQyoAAB1adtsbGAm3+wEA6BOl9RgaQyoAAA1RWg+shtv9AAA0RGk9sBrK/AEAaIvSemAF3O4HAKChJK8k2Sfp6yTrs8+apLer8wG9YEgFAKDG/UvOHWmeAugUt/sBAGjI9glJL0k6YHtzdmlN0nc1qYD+8OAUAAANUVoPrIYhFQAAAN1hTyoAAAC6w5AKAACA7jCkAgAAoDsMqQAAAOjOLVvwPuj6SIXSAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CSmfcqUKTCv7"
      },
      "source": [
        "\n",
        "Looks like our pretrained USE TensorFlow Hub models have the best performance, even the one with only 10% of the training data seems to outperform the other models. This goes to show the power of transfer learning.\n",
        "\n",
        "How about we drill down and get the F1-score's of each model?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 546
        },
        "id": "8WdiPrpUTQyE",
        "outputId": "3abec9f5-5a9a-4ddb-f15c-1cbd7e950829"
      },
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(\"f1_score\", ascending=False)[\"f1_score\"].plot(kind=\"bar\", figsize=(10, 7));"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAIRCAYAAABu0TiPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5wlZX3v+88XEBUFxDBq5C571EMULxnBiCfeghvNFiLxAsZ4jRw94iUad3BrkOBOjBr1ZCs7iibeFcGdmFFRJN7jlQERBCVOQAXMjoMaILojYn7nj6qGRdNNNzw9XdVTn/frtV6sqlUz/XXZvebbVU89T6oKSZIk3TLbDR1AkiRpLbNMSZIkNbBMSZIkNbBMSZIkNbBMSZIkNdhhqC+8++6717777jvUl5ckSVq2s88++4qqWrfQa4OVqX333ZdNmzYN9eUlSZKWLcl3F3vNy3ySJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNdhg6QKt9j/vo0BEA+M6f/ebQESRJ0gA8MyVJktRgWWUqyWFJLkqyOclxC7y+d5JPJ/lakvOSPHrlo0qSJI3PkmUqyfbAScCjgAOAo5McMO+wlwOnVtX9gKOA/7nSQSVJksZoOWemDgI2V9XFVXUNcApwxLxjCtilf74r8P2ViyhJkjReyylTewCXzmxf1u+bdQLw5CSXAacDz1voL0pyTJJNSTZt2bLlFsSVJEkal5UagH408I6q2hN4NPDuJDf6u6vq5KraUFUb1q1bt0JfWpIkaTjLKVOXA3vNbO/Z75v1TOBUgKr6EnAbYPeVCChJkjRmyylTZwHrk+yXZEe6AeYb5x3zPeARAEn+L7oy5XU8SZK0zVuyTFXVtcCxwBnAN+nu2rsgyYlJDu8PezHwrCRfB94PPK2qamuFliRJGotlzYBeVafTDSyf3Xf8zPMLgUNWNpokSdL4rfnlZHRjY1liB1xmR5K07XM5GUmSpAaWKUmSpAZe5tNkePlTkrQ1eGZKkiSpgWempInzjJ0ktfHMlCRJUgPPTEnSAsZyxs6zddL4WaYkScsyloIJlkyNi5f5JEmSGlimJEmSGlimJEmSGjhmSpKkBo4lk2emJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGlimJEmSGjjPlCRJWnFTmn/LM1OSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNLFOSJEkNllWmkhyW5KIkm5Mct8Drb0hybv/4xyT/uvJRJUmSxmeHpQ5Isj1wEnAocBlwVpKNVXXh3DFV9fszxz8PuN9WyCpJkjQ6yzkzdRCwuaourqprgFOAI27i+KOB969EOEmSpLFbTpnaA7h0Zvuyft+NJNkH2A/41CKvH5NkU5JNW7ZsublZJUmSRmelB6AfBXywqn6x0ItVdXJVbaiqDevWrVvhLy1JkrT6llOmLgf2mtnes9+3kKPwEp8kSZqQ5ZSps4D1SfZLsiNdYdo4/6Ak9wR2A760shElSZLGa8kyVVXXAscCZwDfBE6tqguSnJjk8JlDjwJOqaraOlElSZLGZ8mpEQCq6nTg9Hn7jp+3fcLKxZIkSVobnAFdkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpwbLKVJLDklyUZHOS4xY55glJLkxyQZL3rWxMSZKkcdphqQOSbA+cBBwKXAaclWRjVV04c8x64KXAIVX14yR32lqBJUmSxmQ5Z6YOAjZX1cVVdQ1wCnDEvGOeBZxUVT8GqKofrGxMSZKkcVpOmdoDuHRm+7J+36y7A3dP8oUkX05y2EJ/UZJjkmxKsmnLli23LLEkSdKIrNQA9B2A9cBDgaOBtya5w/yDqurkqtpQVRvWrVu3Ql9akiRpOMspU5cDe81s79nvm3UZsLGqfl5VlwD/SFeuJEmStmnLKVNnAeuT7JdkR+AoYOO8Yz5Ed1aKJLvTXfa7eAVzSpIkjdKSZaqqrgWOBc4AvgmcWlUXJDkxyeH9YWcAP0xyIfBp4CVV9cOtFVqSJGkslpwaAaCqTgdOn7fv+JnnBbyof0iSJE2GM6BLkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1WFaZSnJYkouSbE5y3AKvPy3JliTn9o/fW/mokiRJ47PDUgck2R44CTgUuAw4K8nGqrpw3qEfqKpjt0JGSZKk0VrOmamDgM1VdXFVXQOcAhyxdWNJkiStDcspU3sAl85sX9bvm++3k5yX5INJ9lroL0pyTJJNSTZt2bLlFsSVJEkal5UagP5hYN+qOhA4E3jnQgdV1clVtaGqNqxbt26FvrQkSdJwllOmLgdmzzTt2e+7TlX9sKp+1m++DfjVlYknSZI0bsspU2cB65Psl2RH4Chg4+wBSX55ZvNw4JsrF1GSJGm8lrybr6quTXIscAawPfDXVXVBkhOBTVW1EXh+ksOBa4EfAU/bipklSZJGY8kyBVBVpwOnz9t3/MzzlwIvXdlokiRJ4+cM6JIkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ0sU5IkSQ2WVaaSHJbkoiSbkxx3E8f9dpJKsmHlIkqSJI3XkmUqyfbAScCjgAOAo5McsMBxOwMvAL6y0iElSZLGajlnpg4CNlfVxVV1DXAKcMQCx70SeDXw7yuYT5IkadSWU6b2AC6d2b6s33edJPcH9qqqj65gNkmSpNFrHoCeZDvg9cCLl3HsMUk2Jdm0ZcuW1i8tSZI0uOWUqcuBvWa29+z3zdkZuBfwmSTfAR4IbFxoEHpVnVxVG6pqw7p16255akmSpJFYTpk6C1ifZL8kOwJHARvnXqyqK6tq96rat6r2Bb4MHF5Vm7ZKYkmSpBFZskxV1bXAscAZwDeBU6vqgiQnJjl8aweUJEkasx2Wc1BVnQ6cPm/f8Ysc+9D2WJIkSWuDM6BLkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1sExJkiQ1WFaZSnJYkouSbE5y3AKvPzvJ+UnOTfIPSQ5Y+aiSJEnjs2SZSrI9cBLwKOAA4OgFytL7qureVXVf4DXA61c8qSRJ0ggt58zUQcDmqrq4qq4BTgGOmD2gqq6a2bwdUCsXUZIkabx2WMYxewCXzmxfBhw8/6AkzwVeBOwIPHyhvyjJMcAxAHvvvffNzSpJkjQ6KzYAvapOqqr9gT8EXr7IMSdX1Yaq2rBu3bqV+tKSJEmDWU6ZuhzYa2Z7z37fYk4BfqsllCRJ0lqxnDJ1FrA+yX5JdgSOAjbOHpBk/czmbwLfXrmIkiRJ47XkmKmqujbJscAZwPbAX1fVBUlOBDZV1Ubg2CS/Afwc+DHw1K0ZWpIkaSyWMwCdqjodOH3evuNnnr9ghXNJkiStCc6ALkmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1MAyJUmS1GBZZSrJYUkuSrI5yXELvP6iJBcmOS/JJ5Pss/JRJUmSxmfJMpVke+Ak4FHAAcDRSQ6Yd9jXgA1VdSDwQeA1Kx1UkiRpjJZzZuogYHNVXVxV1wCnAEfMHlBVn66qn/abXwb2XNmYkiRJ47ScMrUHcOnM9mX9vsU8E/jYQi8kOSbJpiSbtmzZsvyUkiRJI7WiA9CTPBnYALx2oder6uSq2lBVG9atW7eSX1qSJGkQOyzjmMuBvWa29+z33UCS3wBeBjykqn62MvEkSZLGbTlnps4C1ifZL8mOwFHAxtkDktwPeAtweFX9YOVjSpIkjdOSZaqqrgWOBc4AvgmcWlUXJDkxyeH9Ya8Fbg+cluTcJBsX+eskSZK2Kcu5zEdVnQ6cPm/f8TPPf2OFc0mSJK0JzoAuSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUwDIlSZLUYFllKslhSS5KsjnJcQu8/utJzklybZLHrXxMSZKkcVqyTCXZHjgJeBRwAHB0kgPmHfY94GnA+1Y6oCRJ0pjtsIxjDgI2V9XFAElOAY4ALpw7oKq+07/2H1shoyRJ0mgt5zLfHsClM9uX9ftutiTHJNmUZNOWLVtuyV8hSZI0Kqs6AL2qTq6qDVW1Yd26dav5pSVJkraK5ZSpy4G9Zrb37PdJkiRN3nLK1FnA+iT7JdkROArYuHVjSZIkrQ1LlqmquhY4FjgD+CZwalVdkOTEJIcDJHlAksuAxwNvSXLB1gwtSZI0Fsu5m4+qOh04fd6+42een0V3+U+SJGlSnAFdkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpgWVKkiSpwbLKVJLDklyUZHOS4xZ4/dZJPtC//pUk+650UEmSpDFaskwl2R44CXgUcABwdJID5h32TODHVfWfgDcAr17poJIkSWO0nDNTBwGbq+riqroGOAU4Yt4xRwDv7J9/EHhEkqxcTEmSpHFKVd30AcnjgMOq6vf67d8FDq6qY2eO+UZ/zGX99j/1x1wx7+86Bjim37wHcNFK/Q9ptDtwxZJHTY/vy435nizM92Vhvi8L8325Md+ThY3pfdmnqtYt9MIOq5miqk4GTl7Nr7kcSTZV1Yahc4yN78uN+Z4szPdlYb4vC/N9uTHfk4WtlfdlOZf5Lgf2mtnes9+34DFJdgB2BX64EgElSZLGbDll6ixgfZL9kuwIHAVsnHfMRuCp/fPHAZ+qpa4fSpIkbQOWvMxXVdcmORY4A9ge+OuquiDJicCmqtoI/BXw7iSbgR/RFa61ZHSXHkfC9+XGfE8W5vuyMN+Xhfm+3JjvycLWxPuy5AB0SZIkLc4Z0CVJkhpYpiRJkhpYpiRJkhpYpiRJkhqs6qSdY9GvN/j3VfWwobOMUZIHA+ur6u1J1gG3r6pLhs41lCQ7AS8G9q6qZyVZD9yjqj4ycDSNUP/98Sq6tUxvM7e/qu42WKgRSLIBeBmwD92/PQGqqg4cNNgAktzxpl6vqh+tVpYxSfJGYNG74qrq+asY52aZZJmqql8k+Y8ku1bVlUPnGZMkrwA20C3383bgVsB7gEOGzDWwtwNnA7/Wb18OnAZMukwluZrrP/h2pPte+UlV7TJcqlF4O/AKukXfHwY8Ha8CALwXeAlwPvAfA2cZ2tl0PzsLrWFbwFSL96b+v4fQ/TLygX778cCFgyRapkmWqd6/AecnORP4ydzOMTffVfJY4H7AOQBV9f0kOw8baXD7V9UTkxwNUFU/dSFvqKrrvi/69+MI4IHDJRqN21bVJ5Okqr4LnJDkbOD4oYMNbEs/L+HkVdV+Q2cYo6p6J0CS5wAPrqpr++03A58fMttSplym/qZ/6IauqapKUgBJbjd0oBG4Jslt6c/CJNkf+NmwkcalX/HgQ/2ZzeOGzjOwnyXZDvh2P+Hx5cDtB840Bq9I8jbgk8z8/FTVpD+Hk+wGrOeGl4Q/N1yiUdgN2IVuEnDofn52Gy7O0iZbpqrqnf0/kHtX1UVD5xmRU5O8BbhDkmcBzwDeOnCmob0C+DiwV5L30p2CftqgiUYgyZEzm9vRXR7+94HijMkLgJ2A5wOvpLvU95RBE43D04F70l0OnrvMV0z4l9okv0f3/bIncC7dmd0vAQ8fMtcI/BnwtSSfprsU+uvACYMmWsJkZ0BP8hjgz4Edq2q/JPcFTqyqwweONrgkhwKPpPsmPqOqzhw40uCS/BLdB12AL1fVFQNHGlySt89sXgt8B3hrVf1gmETjkOTxVXXaUvumJslFVXWPoXOMSZLzgQfQfabcN8k9gT+tqiOX+KPbvCR3AQ7uN79SVf97yDxLmXKZOpuu/X+mqu7X7/tGVd1r2GQamySHAOdW1U+SPBm4P/AX/XiYServiH1+Vb1h6Cxjk+Scqrr/Uvumpi/fr62qUQ8kXk1JzqqqByQ5Fzi4qn6W5IKq+pWhs41NkntW1beGzrGYyV7mA35eVVfOG0c89TtM5i7dvBq4E91ZmLnbl6d8h9ZfAvdJch/gRXQLe78LeMigqQbU3xF7NN0dawKSPAp4NLBHkv8x89IudGfupu6BwLlJLqEbMzXZqRFmXJbkDsCHgDOT/BiY7C9pS/gEsPfQIRYz5TJ1QZInAdv388I8H/jiwJnG4DXAY6rqm0MHGZFr+0H5RwAnVdVfJXnm0KFG4AtJ3kR3+/LsHbHnDBdpUN+nu7X7cLpb3+dcDfz+IInG5bChA4xNVT22f3pCPz5oV7rxmZM075eQG7wE3GE1s9xcU77MtxPdBHLXjQ0CXllVkx5Am+QLVTXlOaVuJMln6T7gnk43EPIHwNer6t6DBhtY/+EP1881NXemYdKDZ5Pcqqp+PnSOsUny7qr63aX2TU1/yfzOzJzcqKrvDZdoOP3cdS9m4bulX1dVu69ypGWbbJnSwpL8BXAXutPO3r7MdQMhnwScVVWfT7I38NCqetfA0QaV5MXccOLBAq4CNlXVuYMFG5gzoC9s/rixvkScX1UHDBhrUEmeR3e38L8wc4fjVC99JvkU8PKqutFVoiSXjHl+rsmVqSQf5qanq5/03Xzz7tCaU1X1jFUPo1FL8j666RA20hWq/wKcB+wLnFZVrxku3XCS/APXz4D+GPoZ0KtqkpN2Jnkp8N+A2wI/ndsNXAOcXFUvHSrb0JJspht4/sOhs4xBv8zOv1fVT5c8eGSmWKbmBg0fSXcG5j399tHAv1SVYxt0Aw7KX1iSzwGPrqp/67dvD3yUbmzM2VM945Dk7Kr61STnz10Knts3dLYhJXnVlIvTQvpL5YfOzfStTv+Z+9GqWjOTI09uAHpVfRYgyeuqasPMSx9OsmmRP7bNS/Jfq+o1iy00OfFldhyUv7A7ccOxDT8H7lxV/yfJmvkQ3AqcAX1hH0lyO6cYuYGLgc8k+Sg3HFbx+uEijcJjgDf0v7B9APj42Avn5MrUjNsluVtVXQyQZD9gykunzBWFyRbKm/AvFqkFvRf4SpK/67cfA7yvX4JoynMJzZ8B/eHAUwdNNA6zU4y8GHgbE59iBPhe/9ixfwioqqcnuRXwKLqrRiclObOqfm/gaIua3GW+OUkOA06m+80gwD7AMVX1iUGDaXQclL+4JBvoltcB+EJVWca1oLkB6EmOBy7vpxiZ/GSmcN0lcuYumavTF6rD6O+k9m6+kUpya7q1ogC+tZauz640B+YvzkH5Wg5/hm6aU4zcWJJ7Ae8G7tjvugJ4SlVdMFyq4fUT4D4ReCjwGeBU4BNjvtQ32TLVN97n0P1QQ/d/2FumOj/MzMD8Bc2NNZO0MG9uuWlOMXJjSb4IvKyqPt1vP5Rubb4HDRpsYEneTzdW6mNr5STHlMvU2+hWL39nv+t3gV+M+ZrsaklyW2Dvqrpo6CxjkOTudOM97lxV90pyIHB4Vf33gaNphJJsmndzy4L7pCRfr6r7LLVP47fd0AEG9ICqempVfap/PJ1u9e5JS/IY4Fz6JQ2S3DfJxmFTDe6twEvp7lajqs4Djho0kcbsdkmum6Bz6je3JLk6yVULPK5OctXQ+QZ2cZI/SrJv/3g53TjeSUtyZJJvJ7lyrXyvTPluvl8k2b+q/gmg//D7xcCZxuAE4CC6y55U1bn9PwZTtlNVfXXeotijvXavwf0+3e3uN7i5ZdhIw6mqnYfOMGLPAP4YmLuZ5fP9vqlbc9PRTLlMvQT49LwPvKcPG2kUfl5VV84rDtO8Fny9K5LsT/8+JHkc8M/DRtJYVdXH+yVlFry5JcmhVXXmMOk0JlX1Y7opNHRDa246msmOmYLr7ua7R7950VoZ6LY1Jfkr4JPAccBv0/2g36qqnj1osAH1Zy1PBh4E/Bi4BHhyVX1nyFxam5wOQEn+v6p64WJ3gHrn59qbjmayZSrJc4H3VtW/9tu7AUdX1f8cNtmwkuwEvAx4JN0ZuzOAV1bVvw8abAT6ySi3q6qrh86itSvJ16rqfkPn0HCS/GpVnb3YXdRTv3t6LU5HM+UydW5V3XfePj/kZvSrut+uqkY98G9rSfKim3rdJR90S3hmSnOSvKCq/mKpfRq/Kd/Nt31mBgb1xWHy0/kneV+SXfqzMOcDFyZ5ydC5BrJz/9hANyfZHv3j2XTriklSi4WWGXraaocYmyR7JvnbJD/oH/8ryZ5D57opUx6A/nHgA0ne0m//P/2+qTugqq5K8jvAx+jGTp0NvHbYWKuvqv4YoF9s8/5zl/eSnAB8dMBoGrEkt54//nLevu+sfiqNSZKj6SYw3W/e1DM7Az8aJtWovB14H/D4fvvJ/b5DB0u0hCmXqT+kK1DP6bfPpFt4c+pu1c8O/1vAm6rq50mmeS34encGrpnZvqbfJy3kS9z4zOV1+6rqyFVPpLH5It0dwbsDr5vZfzVw3iCJxmVdVc2Om3pHkhcOlmYZJlumquo/6Ga1/suhs4zMW+h+c/468Lkk+wCTHDM1413AV5P8bb/9W8A7houjMeqXS9kDuG2S+9HdwAGwC7DTYME0OlX1XeC7/RWA78/d4NOvPrEnnr38YZInA+/vt48GfjhgniVNeQD6IXQTVO5DVypDd7fA3W7qz01Rkh3GvMDkakhyf+D/7jc/V1Vfm3ltt36+GE1YkqfSjXfZAGyaeelq4B1jvq1bw0iyCXhQVV3Tb+8IfKGqJr0aR/9L/BuBX6ObOuKLwPOq6tJBg92EKZepb9HNVHw2MzOfV9Wo2+9qSPKbwK8At5nbV1UnDpdo3Lw7S7OS/HZV/a+hc2j8FrmrfPJr8yV5J/DCuV9Sk9wR+PMxT40w2ct8wJVV9bGhQ4xNkjfTXZJ4GN0YsscBXx001Phl6UM0IR9J8iRgX2Y+Y/2FRAvYkuTwqtoIkOQI4IqBM43BgbNn+6vqR/2l89Gacpn6dJLX0q2JNDvD6jnDRRqFB1XVgUnOq6o/TvI6urv6tLhpnt7VYv4OuJLurPfkV1XQTXo28N4kJ9F9jlwGPGXYSKOw3ezwif7M1Kj7yqjDbWUH9//dMLOvgIcPkGVM/k//358muSvdoL9fHjCPtNbsWVWHDR1C41dV/wQ8MMnt++1/GzjSWLwO+FKS0/rtxwN/MmCeJU22TFXVw4bOMFIfSXIHulW7z+73OWXETfMyn2Z9Mcm9q+r8oYNo3JLcGfhT4K5V9agkBwC/VlV/NXC0QVXVu/rB+XMnN46sqguHzLSUKQ9A95t4Af2tuc+hu3OtgM8Dfzn1tfmSPBhYX1VvT7IOuH1VXdK/dseqcqI9AZDkQuA/0S2I/TOuv1P4wEGDaXSSfIxuMsqXVdV9kuwAfK2q7j1wNN1MUy5TfhMvIMmpdLdyv6ff9SRg16p6wnCphpXkFXSXg+9RVTtxJycAAA0hSURBVHfvL3+eVlWHDBxNI9Tf1n0j/dxC0nWSnFVVD5hdF3ahO/w0flNem2/3qjoV+A+Afh6lX9z0H5mEe1XVM6vq0/3jWcC9hg41sMcChwM/Aaiq79Mt+yDdSF+a9gIe3j//KdP+rNXifpLkl+hvYknyQLqbF7TGTHbMFH4TL+acJA+sqi8DJDmYG05AOEXXVFXNLavTLwItLWj2TCbd2e9b0Z3p9Uym5nsRsBHYP8kXgHV009FojZlymfKbeEaS8+mK5a3oBtB+r9/eB/jWkNlG4NR+Qew7JHkW8AzgrQNn0ng9FrgfcA50ZzKTeCZTN5Bke+Ah/eMedGPrLqqqnw8aTLfIZMdMQbdMCot8Eyc5tKrOHCzcKltsnMecqY/3SHIo8Ei675UzpvS9oZsnyVer6qC5mfH7M5lfcgC65pv7Xhk6h9pNukzdFJcIkXRLJPkDYD1wKPAqujOZ76uqNw4aTKOT5A10VwM+QD8mE5w8ei2yTC1i9u4KTVOSq1l4dvO5W913WeVIWiM8k6nlSPLpBXZXVU198ug1xzK1CM9MSbolkuwH/PPc3Gz93G13rqrvDBpM0lYz5QHo0rIluT/wYLozVf9QVV8bOJLG6zTgQTPbv+j3PWCYOBqbJE+uqvckedFCr1fV61c7k9o498nivjN0AI1DkuOBdwK/BOwOvCPJy4dNpRHboaqumdvon+84YB6Nz9z0Kjsv8tAaM9nLfEl2Al4M7F1Vz0qynm6G648MHE0jk+Qi4D7zLtucW1X3GDaZxijJmcAbq2pjv30E8PyqesSwySRtLVO+zPd2uoV8f63fvpzuVLxlSvN9H7gNMLc+4a3pvl+khTwbeG+SN/XblwG/O2AejUyS/3FTr1fV81cri1bGlMvU/lX1xCRHA1TVT5Nk6FAapSuBC/ozDkV3y/tX5z4Q/eDTnH4ixudU1QOT3B6gqv5t4Fgan7P7/x4CHEA3NQLA44ELB0mkJlMuU9f0l2vmlgjZn26Fd2m+v+0fcz4zUA6NXFX9IsmD++eWKC2oqt4JkOQ5wIP7tWFJ8mbg80Nm0y0z5TL1CuDjwF5J3kv3G8LTBk2kUZr74JOW6WtJNtING5idiPFvhoukkdoN2AX4Ub99+36f1pjJlqmqOjPJOcAD6SbWe0FVXTFwLI1Qkv8CvJJuncIdcNJO3bTbAD8EZideLMAypfn+jK58f5ruc+XXgRMGTaRbZMp38z0W+FRVXdlv3wF4aFV9aNhkGpskm4EjgfNrqj8wkraKJHcBDu43v1JV/3vIPLplpjzP1CvmihRAVf0r3aU/ab5LgW9YpLQcSe6e5JNJvtFvH+i8ZJqV5J79f+8P3JXuM+ZS4K79Pq0xUz4zdd78VdyTnF9V9x4qk8YpyQPoLvN9lpmbFJylWAtJ8lngJcBb5tb3TPKNqrrXsMk0FklOrqpj+st7s/8Izw0hcG2+NWbKZ6Y2JXl9kv37x+u5/nZVadafAD+lGwvjLMVayk5V9dV5+64dJIlGqaqO6Z8+Gvgo3fQr/wps7PdpjZnsAHTgecAfcf38HmcCzx0ujkbsrp5V0M1wRT/Vyty0K48D/nnYSBqpdwJXAXOTeD4JeBfwhMES6RaZ7GU+abmSvAb4+6r6xNBZNH5J7gacTLfY8Y+BS4DfqarvDhpMo5Pkwqo6YKl9Gr/Jlqkkdwf+ANiXmTN0XqvWfEmupluY9GfAz3FqBC1DktsB21XV1UNn0TgleQ/wpqr6cr99MPDcqnrKsMl0c025TH0deDPdOKlfzO2vKsdNSbrFkvwS3Z3BD6a71PcPwIlV9cNBg2k0kpxP971xK+AewPf67X2Ab3lmau2Zcpk6u6p+degcGq8k96yqby12q3JVnbPamTR+/RqOnwPe0+/6Hbo57H5juFQakyT73NTrXhJee6Zcpk4AfkC35trs7e4/WuzPaFrm3b4857ofGC8JayELTYPgtCvStm3KZeqSBXZXVd1t1cNo1JI8Afh4VV2V5I+A+wOv9MyUFtJPs/JV4NR+1+OAg6rqD4ZLJWlrmmyZkpZrboLXJA+mm7zzz4Hjq+rgJf6oJmjmhoW5sZjbc/2Cx964IG2DJjtpZ5Kdkrw8ycn99vp+QVtpvrl/FH8TeGtVfRTYccA8GrGq2rmqtquqW/WP7fp9O1fVLkl+ZeiMklbWZMsU8HbgGrq5YAAuB/77cHE0YpcneQvwROD0JLdm2j87avPuoQNIWllT/gdh/6p6Dd28QVTVT+nmD5LmewJwBvCf+wWx70i39pp0S/g5I21jpryczDVJbsv1Sz7sz8xdfdKcvmj/zcz2P+PyILrlHKgqbWOmXKZOAD4O7JXkvcAhwNMHTSRJktacSd/N189U/EC60+5frqorBo4kaRuX5MtV9cChc0haOZMtU0k+WVWPWGqfJC1Xkl2Bw4A9+l2XA2f0Y+0kbaMmNwA9yW2S3BHYPcluSe7YP/bl+g9ASbpZkjwFOAd4KLBT/3gYcHb/mqRt1OTOTCV5AfBC4K50vzXO3VlzFd0cQm8aKpuktSvJRcDB889CJdkN+EpV3X2YZJK2tsmVqTlJnldVbxw6h6RtQ5J/BB5QVVfO278rsKmq1g+TTNLWNtm7+arqjUkeBOzLzPtQVe8aLJSktexPgHOSfAK4tN+3N3Ao3TJEkrZRUz4z9W5gf+Bcrl8upKrq+cOlkrSW9Zf0/jM3HoD+4+FSSdraplymvgkcUFN9AyRJ0oqY3N18M74B3GXoEJK2fUnOHzqDpK1nsmOmgN2BC5N8lZllZKrq8OEiSVqrkhy52Ev4i5u0TZtymTph6ACStikfAN7Lwmvv3WaVs0haRZMdMwWQZB9gfVX9fZKdgO2r6uqhc0lae5KcDTy1qr6xwGuXVtVeA8SStAomO2YqybOADwJv6XftAXxouESS1rgX0k3+u5DHrmYQSatrsmUKeC5wCP2HX1V9G7jToIkkrVlV9fmq+t4ir22ae57kpauXStJqmHKZ+llVXTO3kWQHFh7rIEkr6fFDB5C0sqZcpj6b5L8Bt01yKHAa8OGBM0na9mXpQyStJZMdgJ5kO+CZwCPpPtzOAN7mJJ6StqYk51TV/YfOIWnlTLZMzUpyR2DPqjpv6CyStm1JvlZV9xs6h6SVM9nLfEk+k2SXvkidDbw1yRuGziVpm3fa0AEkrazJlilg16q6CjgSeFdVHQw8YuBMkta4JHdL8uEkVyT5QZK/S3K3uder6k+HzCdp5U25TO2Q5JeBJwAfGTqMpG3G+4BT6ZaQuSvdmaj3D5pI0lY15TJ1It2g881VdVb/m+O3B84kae3bqareXVXX9o/34HIy0jbNAeiLSPLSqnrV0DkkrQ39+EuAPwR+DJxCN3fdE4HdqsrJOqVtlGVqEd6+LOnmSHIJXXlaaB6pqqq7LbBf0jZgh6EDjJgT60latqrab+gMkoZhmVqcp+wk3WxJnrLQ/qp612pnkbQ6LFOL88yUpFviATPPb0M35co5gGVK2kZZphbnxHqSbraqet7sdpI70A1Gl7SNmuzUCE6sJ2mV/ARwPJW0DZvyman3AScBj+23j6KbWO/gwRJJWvOSfJjrx1xuBxxAN4mnpG3UZKdGSHJeVR04b9/Xq+o+Q2WStPYlecjM5rXAd6vqsqHySNr6JlemnFhPkiStpCmWKSfWk7TVJDkSeDVwJ7rPmdB9tuwyaDBJW83kypQkbU1JNgOPqapvDp1F0uqY7AB0J9aTtJX8i0VKmpbJnplK8saZzesm1quqxw0USdIa1l/eA3gIcBfgQ8DP5l6vqr8ZIpekrW+yZWq+uYn1quqwobNIWnuSvP0mXq6qesaqhZG0qixTvSS3Ar5RVfcYOoukbVeSl1bVq4bOIWnlTHnMlBPrSRrC4wHLlLQNmWyZAv585rkT60laLS6iLm1jJlumquqzQ2eQNEmOrZC2MVNe6PjIJN9OcmWSq5JcneSqoXNJ2uZ5Zkraxky2TAGvAQ6vql2rapeq2tkZiiXdUkle3f/38UscetoqxJG0iiZ7N1+SL1TVIUPnkLRtSHI+cCBwdlXdf+g8klbP5MZMzUystynJB3BiPUkr4+N0i6ffft6QAdfmk7Zxkzsz5cR6kramJJ+oqkfO2/eaqvqvQ2WStHVNrkwtlxPrSbolkpwz/zJfkvOq6sChMknauqY8AH0pSw0ilaTrJHlOP27qHknOm3lcApw/dD5JW49nphaR5GtVdb+hc0haG5LsCuxGN7v5cTMvXV1VPxomlaTVYJlaxEKn6iVJkubzMt/inFhPkiQtaXJlyon1JEnSSprcZT4n1pMkSStpcpN24sR6kiRpBU3uMl9VvaSq7gB8ql+Tb+6xM/DmofNJkqS1ZXJlasbuC+w7bNVTSJKkNW1yl/mSPAf4f4G7JTlv5qWdgS8Ok0qSJK1VUxyA7sR6kiRpxUyuTEmSJK2kKY+ZkiRJamaZkiRJamCZkiRJamCZkiRJavD/AxVQpwaPl9AqAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K8GnCQvSTe4g"
      },
      "source": [
        "\n",
        "Drilling down into a single metric we see our USE TensorFlow Hub models performing better than all of the other models. Interestingly, the baseline's F1-score isn't too far off the rest of the deeper models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UgBTP8YVVmqF"
      },
      "source": [
        "## Combining our models (model ensembling/stacking)\n",
        "Many production systems use an ensemble (multiple different models combined) of models to make a prediction.\n",
        "\n",
        "The idea behind model stacking is that if several uncorrelated models agree on a prediction, then the prediction must be more robust than a prediction made by a singular model.\n",
        "\n",
        "The keyword in the sentence above is uncorrelated, which is another way of saying, different types of models. For example, in our case, we might combine our baseline, our bidirectional model and our TensorFlow Hub USE model.\n",
        "\n",
        "Although these models are all trained on the same data, they all have a different way of finding patterns.\n",
        "\n",
        "If we were to use three similarly trained models, such as three LSTM models, the predictions they output will likely be very similar.\n",
        "\n",
        "Think of it as trying to decide where to eat with your friends. If you all have similar tastes, you'll probably all pick the same restaurant. But if you've all got different tastes and still end up picking the same restaurant, the restaurant must be good.\n",
        "\n",
        "Since we're working with a classification problem, there are a few of ways we can combine our models:\n",
        "\n",
        "- Averaging - Take the output prediction probabilities of each model for each sample, combine them and then average them.\n",
        "- Majority vote (mode) - Make class predictions with each of your models on all samples, the predicted class is the one in majority. For example, if three different models predict [1, 0, 1] respectively, the majority class is 1, therefore, that would be the predicted label.\n",
        "- Model stacking - Take the outputs of each of your chosen models and use them as inputs to another model.\n",
        "\n",
        "> 📖 Resource: The above methods for model stacking/ensembling were adapted from Chapter 6 of the Machine Learning Engineering Book by Andriy Burkov. If you're looking to enter the field of machine learning engineering, not only building models but production-scale machine learning systems, I'd highly recommend reading it in its entirety.\n",
        "\n",
        "Again, the concept of model stacking is best seen in action.\n",
        "\n",
        "We're going to combine our baseline model (model_0), LSTM model (model_2) and our USE model trained on the full training data (model_6) by averaging the combined prediction probabilities of each."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6AP6bRQ8TUeP",
        "outputId": "a505b0be-d143-4fa6-b2a9-6c90cf71e1c1"
      },
      "source": [
        "# Get mean pred probs for 3 models\n",
        "baseline_pred_probs = np.max(model_0.predict_proba(val_sentences), axis=1) # Get the prediction probabilities from baseline model\n",
        "\n",
        "combined_pred_probs = baseline_pred_probs + tf.squeeze(model_2_pred_probs, axis=1) + tf.squeeze(model_6_pred_probs)\n",
        "combined_preds = tf.round(combined_pred_probs/3) # average and round the prediction probabilities to get prediction classes\n",
        "combined_preds[:20]"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(20,), dtype=float32, numpy=\n",
              "array([0., 1., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 0., 0.,\n",
              "       1., 0., 1.], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoUFERAuWrRo"
      },
      "source": [
        "Wonderful! We've got a combined predictions array of different classes, let's evaluate them against the true labels and add our stacked model's results to our all_model_results DataFrame."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "grdakeO8WkHb",
        "outputId": "2f2e0954-e086-4852-e152-59faefed867b"
      },
      "source": [
        "# Calculate results from averaging the prediction probabilities\n",
        "ensemble_results = calculate_results(val_labels, combined_preds)\n",
        "ensemble_results"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 79.00262467191601,\n",
              " 'f1_score': 0.7895462398117437,\n",
              " 'precision': 0.789769736334093,\n",
              " 'recall': 0.7900262467191601}"
            ]
          },
          "metadata": {},
          "execution_count": 98
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Mx7jft-W7HP"
      },
      "source": [
        "# Add our combined model's results to the results DataFrame\n",
        "all_model_results.loc[\"ensemble_results\"] = ensemble_results"
      ],
      "execution_count": 99,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LEU6OZySW_kW"
      },
      "source": [
        "# Convert the accuracy to the same scale as the rest of the results\n",
        "all_model_results.loc[\"ensemble_results\"][\"accuracy\"] = all_model_results.loc[\"ensemble_results\"][\"accuracy\"]/100"
      ],
      "execution_count": 100,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ws3ANJGeXBdk",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 331
        },
        "outputId": "74125795-ed86-4a32-be75-0446bd3187a3"
      },
      "source": [
        "all_model_results"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.811139</td>\n",
              "      <td>0.792651</td>\n",
              "      <td>0.786219</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>simple_dense</th>\n",
              "      <td>0.783465</td>\n",
              "      <td>0.788873</td>\n",
              "      <td>0.783465</td>\n",
              "      <td>0.780192</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>lstm</th>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.775137</td>\n",
              "      <td>0.771654</td>\n",
              "      <td>0.768748</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>gru</th>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.778683</td>\n",
              "      <td>0.776903</td>\n",
              "      <td>0.774880</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>bidirectional</th>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.761042</td>\n",
              "      <td>0.761155</td>\n",
              "      <td>0.760055</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>conv1d</th>\n",
              "      <td>0.742782</td>\n",
              "      <td>0.742383</td>\n",
              "      <td>0.742782</td>\n",
              "      <td>0.741707</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_sentence_encoder</th>\n",
              "      <td>0.813648</td>\n",
              "      <td>0.814169</td>\n",
              "      <td>0.813648</td>\n",
              "      <td>0.812791</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tf_hub_10_percent_data</th>\n",
              "      <td>0.775591</td>\n",
              "      <td>0.781535</td>\n",
              "      <td>0.775591</td>\n",
              "      <td>0.771874</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>ensemble_results</th>\n",
              "      <td>0.790026</td>\n",
              "      <td>0.789770</td>\n",
              "      <td>0.790026</td>\n",
              "      <td>0.789546</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                         accuracy  precision    recall  f1_score\n",
              "baseline                 0.792651   0.811139  0.792651  0.786219\n",
              "simple_dense             0.783465   0.788873  0.783465  0.780192\n",
              "lstm                     0.771654   0.775137  0.771654  0.768748\n",
              "gru                      0.776903   0.778683  0.776903  0.774880\n",
              "bidirectional            0.761155   0.761042  0.761155  0.760055\n",
              "conv1d                   0.742782   0.742383  0.742782  0.741707\n",
              "tf_hub_sentence_encoder  0.813648   0.814169  0.813648  0.812791\n",
              "tf_hub_10_percent_data   0.775591   0.781535  0.775591  0.771874\n",
              "ensemble_results         0.790026   0.789770  0.790026  0.789546"
            ]
          },
          "metadata": {},
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2iuikrARfqfx"
      },
      "source": [
        "\n",
        "How did the stacked model go against the other models?\n",
        "\n",
        "> 🔑 Note: It seems many of our model's results are similar. This may mean there are some limitations to what can be learned from our data. When many of your modelling experiments return similar results, it's a good idea to revisit your data, we'll do this shortly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YDZe2DASfv0N"
      },
      "source": [
        "## Saving and loading a trained model\n",
        "Although training time didn't take very long, it's good practice to save your trained models to avoid having to retrain them.\n",
        "\n",
        "Saving your models also enables you to export them for use elsewhere outside of your notebooks, such as in a web application.\n",
        "\n",
        "There are two main ways of saving a model in TensorFlow:\n",
        "\n",
        "- The HDF5 format.\n",
        "- The SavedModel format (default).\n",
        "\n",
        "Let's take a look at both."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RhlwNvvpXFli"
      },
      "source": [
        "# Save tf hub sentemce Encoder model to HDF5 format\n",
        "model_6.save(\"model_6.h5\")"
      ],
      "execution_count": 103,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y5WOlEnIiTo_"
      },
      "source": [
        "\n",
        "If you save a model as a HDF5, when loading it back in, you need to let TensorFlow know about any custom objects you've used (e.g. components which aren't built from pure TensorFlow, such as TensorFlow Hub components)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ea09qrTFf-Vd"
      },
      "source": [
        "# load model with custom hub layer (required with HDF5 format)\n",
        "loaded_model_6 = tf.keras.models.load_model(\"model_6.h5\", custom_objects={\"KerasLayer\":hub.KerasLayer})"
      ],
      "execution_count": 104,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SklhIlTwiWsg",
        "outputId": "a917dba2-0197-4897-955b-c7f3a59ddb15"
      },
      "source": [
        "# how does out loaded model perform?\n",
        "loaded_model_6.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 19ms/step - loss: 0.4242 - accuracy: 0.8136\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4242488443851471, 0.8136482834815979]"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5z-VXLC4jBeq"
      },
      "source": [
        "Calling the save() method on our target model and passing it a filepath allows us to save our model in the SavedModel format."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pGiSCYQtjB67",
        "outputId": "44279f35-f70c-4004-fd9f-86414d13d3f7"
      },
      "source": [
        "# Save TF Hub Sentence Encoder model to SavedModel format (default)\n",
        "model_6.save(\"model_6_SavedModel_format\")"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:Function `_wrapped_model` contains input name(s) USE_input with unsupported characters which will be renamed to use_input in the SavedModel.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "INFO:tensorflow:Assets written to: model_6_SavedModel_format/assets\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DNo3vtPljLUF"
      },
      "source": [
        "\n",
        "If you use SavedModel format (default), you can reload your model without specifying custom objects using the tensorflow.keras.models.load_model() function."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8NFGRjrBjHc7"
      },
      "source": [
        "# Load TF Hub Sentence Encoder SavedModel\n",
        "loaded_model_6_SavedModel = tf.keras.models.load_model(\"model_6_SavedModel_format\")"
      ],
      "execution_count": 107,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zR-Yhf41jNT5",
        "outputId": "8d4bddb5-4347-45bf-b849-7eeb387cb12a"
      },
      "source": [
        "#  Evaluate loaded SavedModel format\n",
        "loaded_model_6_SavedModel.evaluate(val_sentences, val_labels)"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "24/24 [==============================] - 1s 8ms/step - loss: 0.4242 - accuracy: 0.8136\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.4242488443851471, 0.8136482834815979]"
            ]
          },
          "metadata": {},
          "execution_count": 108
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FcmH98nujOnr"
      },
      "source": [
        "\n",
        "As you can see saving and loading our model with either format results in the same performance.\n",
        "\n",
        "> 🤔 Question: Should you used the SavedModel format or HDF5 format?\n",
        "\n",
        "For most use cases, the SavedModel format will suffice. However, this is a TensorFlow specific standard. If you need a more general-purpose data standard, HDF5 might be better. For more, check out the TensorFlow documentation on saving and loading models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWEMPSnsju31"
      },
      "source": [
        "## Finding the most wrong examples\n",
        "We mentioned before that if many of our modelling experiments are returning similar results, despite using different kinds of models, it's a good idea to return to the data and inspect why this might be.\n",
        "\n",
        "One of the best ways to inspect your data is to sort your model's predictions and find the samples it got most wrong, meaning, what predictions had a high prediction probability but turned out to be wrong.\n",
        "\n",
        "Once again, visualization is your friend. Visualize, visualize, visualize.\n",
        "\n",
        "To make things visual, let's take our best performing model's prediction probabilities and classes along with the validation samples (text and ground truth labels) and combine them in a pandas DataFrame.\n",
        "\n",
        "- If our best model still isn't perfect, what examples is it getting wrong?\n",
        "- Which ones are the most wrong?\n",
        "- Are there some labels which are wrong? E.g. the model gets it right but the ground truth label doesn't reflect this"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 205
        },
        "id": "kqXpVQJhjWG0",
        "outputId": "c8775563-48ea-4015-e110-02cfaff53bea"
      },
      "source": [
        "# Create dataframe with validation sentences and best performing model predictions\n",
        "val_df = pd.DataFrame({\"text\":val_sentences,\n",
        "                       \"target\":val_labels,\n",
        "                       \"pred\":model_6_preds,\n",
        "                       \"pred_prob\":tf.squeeze(model_6_pred_probs)})\n",
        "\n",
        "val_df.head()"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>DFR EP016 Monthly Meltdown - On Dnbheaven 2015...</td>\n",
              "      <td>0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.172005</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>FedEx no longer to transport bioterror germs i...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.799361</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Gunmen kill four in El Salvador bus attack: Su...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.991638</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>@camilacabello97 Internally and externally scr...</td>\n",
              "      <td>1</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.214594</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Radiation emergency #preparedness starts with ...</td>\n",
              "      <td>1</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.747634</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                text  target  pred  pred_prob\n",
              "0  DFR EP016 Monthly Meltdown - On Dnbheaven 2015...       0   0.0   0.172005\n",
              "1  FedEx no longer to transport bioterror germs i...       0   1.0   0.799361\n",
              "2  Gunmen kill four in El Salvador bus attack: Su...       1   1.0   0.991638\n",
              "3  @camilacabello97 Internally and externally scr...       1   0.0   0.214594\n",
              "4  Radiation emergency #preparedness starts with ...       1   1.0   0.747634"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "99wiE3tbkLwh"
      },
      "source": [
        "\n",
        "Oh yeah! Now let's find our model's wrong predictions (where target != pred) and sort them by their prediction probability (the pred_prob column)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 362
        },
        "id": "whjXAzwFkEBs",
        "outputId": "51b4078d-c5c1-4736-fad1-81ebd025bc2e"
      },
      "source": [
        "# Find the wrong predictions and sort by prediction probabilities\n",
        "most_wrong = val_df[val_df[\"target\"] != val_df[\"pred\"]].sort_values(\"pred_prob\", ascending=False)\n",
        "most_wrong[:10]"
      ],
      "execution_count": 111,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>target</th>\n",
              "      <th>pred</th>\n",
              "      <th>pred_prob</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>? High Skies - Burning Buildings ? http://t.co...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.926577</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>759</th>\n",
              "      <td>FedEx will no longer transport bioterror patho...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.906641</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>628</th>\n",
              "      <td>@noah_anyname That's where the concentration c...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.896482</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>209</th>\n",
              "      <td>Ashes 2015: AustraliaÛªs collapse at Trent Br...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.858158</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>@madonnamking RSPCA site multiple 7 story high...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.832434</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>251</th>\n",
              "      <td>@AshGhebranious civil rights continued in the ...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.831336</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>698</th>\n",
              "      <td>åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Ten...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.823245</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>393</th>\n",
              "      <td>@SonofLiberty357 all illuminated by the bright...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.820716</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>144</th>\n",
              "      <td>The Sound of Arson</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.807447</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>109</th>\n",
              "      <td>[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...</td>\n",
              "      <td>0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.806171</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                  text  target  pred  pred_prob\n",
              "31   ? High Skies - Burning Buildings ? http://t.co...       0   1.0   0.926577\n",
              "759  FedEx will no longer transport bioterror patho...       0   1.0   0.906641\n",
              "628  @noah_anyname That's where the concentration c...       0   1.0   0.896482\n",
              "209  Ashes 2015: AustraliaÛªs collapse at Trent Br...       0   1.0   0.858158\n",
              "49   @madonnamking RSPCA site multiple 7 story high...       0   1.0   0.832434\n",
              "251  @AshGhebranious civil rights continued in the ...       0   1.0   0.831336\n",
              "698  åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Ten...       0   1.0   0.823245\n",
              "393  @SonofLiberty357 all illuminated by the bright...       0   1.0   0.820716\n",
              "144                                 The Sound of Arson       0   1.0   0.807447\n",
              "109  [55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES W...       0   1.0   0.806171"
            ]
          },
          "metadata": {},
          "execution_count": 111
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iRqSwlPnklwt"
      },
      "source": [
        "\n",
        "Finally, we can write some code to visualize the sample text, truth label, prediction class and prediction probability. Because we've sorted our samples by prediction probability, viewing samples from the head of our most_wrong DataFrame will show us false positives.\n",
        "\n",
        "A reminder:\n",
        "\n",
        "0 = Not a real diaster Tweet\n",
        "\n",
        "1 = Real diaster Tweet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2lSue4Aaka0w",
        "outputId": "5e52d876-4d07-42bb-9472-4435d7a01e38"
      },
      "source": [
        "# Check the false positives (model predicted 1 when should've been 0)\n",
        "for row in most_wrong[:10].itertuples(): # loop through the top 10 rows (change the index to view different rows)\n",
        "  _, text, target, pred, prob = row\n",
        "  print(f\"Target: {target}, Pred: {int(pred)}, Prob: {prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 0, Pred: 1, Prob: 0.926577091217041\n",
            "Text:\n",
            "? High Skies - Burning Buildings ? http://t.co/uVq41i3Kx2 #nowplaying\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.9066413044929504\n",
            "Text:\n",
            "FedEx will no longer transport bioterror pathogens in wake of anthrax lab mishaps http://t.co/lHpgxc4b8J\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8964816331863403\n",
            "Text:\n",
            "@noah_anyname That's where the concentration camps and mass murder come in. \n",
            " \n",
            "EVERY. FUCKING. TIME.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8581575155258179\n",
            "Text:\n",
            "Ashes 2015: AustraliaÛªs collapse at Trent Bridge among worst in history: England bundled out Australia for 60 ... http://t.co/t5TrhjUAU0\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.832433819770813\n",
            "Text:\n",
            "@madonnamking RSPCA site multiple 7 story high rise buildings next to low density character residential in an area that floods\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8313357830047607\n",
            "Text:\n",
            "@AshGhebranious civil rights continued in the 60s. And what about trans-generational trauma? if anything we should listen to the Americans.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8232451677322388\n",
            "Text:\n",
            "åÈMGN-AFRICAå¨ pin:263789F4 åÈ Correction: Tent Collapse Story: Correction: Tent Collapse story åÈ http://t.co/fDJUYvZMrv @wizkidayo\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.820716381072998\n",
            "Text:\n",
            "@SonofLiberty357 all illuminated by the brightly burning buildings all around the town!\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8074465394020081\n",
            "Text:\n",
            "The Sound of Arson\n",
            "\n",
            "----\n",
            "\n",
            "Target: 0, Pred: 1, Prob: 0.8061710596084595\n",
            "Text:\n",
            "[55436] 1950 LIONEL TRAINS SMOKE LOCOMOTIVES WITH MAGNE-TRACTION INSTRUCTIONS http://t.co/xEZBs3sq0y http://t.co/C2x0QoKGlY\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKRS2rKCk1l-"
      },
      "source": [
        "\n",
        "We can view the bottom end of our most_wrong DataFrame to inspect false negatives (model predicts 0, not a real diaster Tweet, when it should've predicted 1, real diaster Tweet)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KdogPD88kwBi",
        "outputId": "c58765fe-b0b3-469a-f7ce-5f870cbebd03"
      },
      "source": [
        "#  Check the most wrong false negatives (model predicted 0 when should've predict 1)\n",
        "for row in most_wrong[-10:].itertuples():\n",
        "  _, text, target, pred, prob = row\n",
        "  print(f\"Target: {target}, Pred: {int(pred)}, Prob: {prob}\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Target: 1, Pred: 0, Prob: 0.07150489091873169\n",
            "Text:\n",
            "Lucas Duda is Ghost Rider. Not the Nic Cage version but an actual 'engulfed in flames' badass. #Mets\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.06994360685348511\n",
            "Text:\n",
            "You can never escape me. Bullets don't harm me. Nothing harms me. But I know pain. I know pain. Sometimes I share it. With someone like you.\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.06841179728507996\n",
            "Text:\n",
            "going to redo my nails and watch behind the scenes of desolation of smaug ayyy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.06446188688278198\n",
            "Text:\n",
            "@DavidVonderhaar At least you were sincere ??\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.0594727098941803\n",
            "Text:\n",
            "@willienelson We need help! Horses will die!Please RT &amp; sign petition!Take a stand &amp; be a voice for them! #gilbert23 https://t.co/e8dl1lNCVu\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.05145040154457092\n",
            "Text:\n",
            "I get to smoke my shit in peace\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.050514280796051025\n",
            "Text:\n",
            "@SoonerMagic_ I mean I'm a fan but I don't need a girl sounding off like a damn siren\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.04625841975212097\n",
            "Text:\n",
            "Why are you deluged with low self-image? Take the quiz: http://t.co/XsPqdOrIqj http://t.co/CQYvFR4UCy\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.041441887617111206\n",
            "Text:\n",
            "Reddit Will Now QuarantineÛ_ http://t.co/pkUAMXw6pm #onlinecommunities #reddit #amageddon #freespeech #Business http://t.co/PAWvNJ4sAP\n",
            "\n",
            "----\n",
            "\n",
            "Target: 1, Pred: 0, Prob: 0.03949397802352905\n",
            "Text:\n",
            "Ron &amp; Fez - Dave's High School Crush https://t.co/aN3W16c8F6 via @YouTube\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MlcPgi7wlDJy"
      },
      "source": [
        "\n",
        "Do you notice anything interesting about the most wrong samples?\n",
        "\n",
        "Are the ground truth labels correct? What do you think would happen if we went back and corrected the labels which aren't?\n",
        "\n",
        "## Making predictions on the test dataset\n",
        "Alright we've seen how our model's perform on the validation set.\n",
        "\n",
        "But how about the test dataset?\n",
        "\n",
        "We don't have labels for the test dataset so we're going to have to make some predictions and inspect them for ourselves.\n",
        "\n",
        "Let's write some code to make predictions on random samples from the test dataset and visualize them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KvLgLj6yk8Eb",
        "outputId": "40fd0455-0713-4973-a215-20f418691bc9"
      },
      "source": [
        "# Making prediction on the test dataset\n",
        "test_sentences = test_df[\"text\"].to_list()\n",
        "test_samples = random.sample(test_sentences, 10)\n",
        "for test_sample in test_samples:\n",
        "  pred_prob = tf.squeeze(model_6.predict([test_sample])) # has to be list\n",
        "  pred = tf.round(pred_prob)\n",
        "  print(f\"Pred: {int(pred)}, Prob: {pred_prob}\")\n",
        "  print(f\"Text:\\n{test_sample}\\n\")\n",
        "  print(\"----\\n\")"
      ],
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 0, Prob: 0.029479175806045532\n",
            "Text:\n",
            "My new mashup of 'Bad Blood x Where Are U Now' comes out tomorrow!!! Make sure t... (Vine by @Shaylen_Carroll) https://t.co/zB6GZJLgkl\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.2707194983959198\n",
            "Text:\n",
            "PSA: IÛªm splitting my personalities.\n",
            "\n",
            "?? techies follow @ablaze_co\n",
            "?? Burners follow @ablaze\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.04286396503448486\n",
            "Text:\n",
            "' #OTRAMETLIFE ' I SWEAR TO GOD I DIDNT EVEN READ IT PROPERLY AND I THOUGH IT SAID 'OMLETTE LIFE' AND SCREAMED. I NEED TO SLEEP OMFG. BYE.\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.18227869272232056\n",
            "Text:\n",
            "That ball was crushed\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9576691389083862\n",
            "Text:\n",
            "Flash floods landslides possible as Hanna draws in more monsoon rains - Pagasa http://t.co/TzWmsEQPHW | via News5 #iBalita\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.8816193342208862\n",
            "Text:\n",
            "A sinkhole grows in Brooklyn: six-meter crater swallows street http://t.co/LTLh53aRN4\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.643398642539978\n",
            "Text:\n",
            "Migrants drowning - we will be erecting monuments 100 years from now and shed tears for the most powerful nations' greed of letting them die\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.07783225178718567\n",
            "Text:\n",
            "Yeah I hate cats I just tried to pet our own cat and now my hand has bite marks scratch marks and it's bleeding. #DogsAreBetterThanCats\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 1, Prob: 0.9828091263771057\n",
            "Text:\n",
            "[HIGH PRIORITY] SEVERE THUNDERSTORM WATCH ENDED Issued for Red Deer [Updated: Aug 05th 20:29 MDT] http://t.co/ZrIyNa6f6m\n",
            "\n",
            "----\n",
            "\n",
            "Pred: 0, Prob: 0.32228872179985046\n",
            "Text:\n",
            "@peregrinekiwi @boymonster I recall CP2020 Australia was a blighted desolate corporate\n",
            "\n",
            "----\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lrxP87tPl1u0"
      },
      "source": [
        "\n",
        "How do our model's predictions look on the test dataset?\n",
        "\n",
        "It's important to do these kind of visualization checks as often as possible to get a glance of how your model performs on unseen data and subsequently how it might perform on the real test: Tweets from the wild.\n",
        "\n",
        "## Predicting on Tweets from the wild\n",
        "How about we find some Tweets and use our model to predict whether or not they're about a diaster or not?\n",
        "\n",
        "To start, let's take one of my own Tweets on living life like an ensemble model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IHJ2AYNulsia"
      },
      "source": [
        "# Turn Tweet into string\n",
        "daniels_tweet = \"Life like an ensemble: take the best choices from others and make your own\"\n"
      ],
      "execution_count": 115,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eS4xdOqvl8Aa"
      },
      "source": [
        "\n",
        "Now we'll write a small function to take a model and an example sentence and return a prediction.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S88l92TPl55L"
      },
      "source": [
        "def predict_on_sentence(model, sentence):\n",
        "  \"\"\"\n",
        "  Uses model to make a prediction on sentence.\n",
        "\n",
        "  Returns the sentence, the predicted label and the prediction probability.\n",
        "  \"\"\"\n",
        "  pred_prob = model.predict([sentence])\n",
        "  pred_label = tf.squeeze(tf.round(pred_prob)).numpy()\n",
        "  print(f\"Pred: {pred_label}\", \"(real disaster)\" if pred_label > 0 else \"(not real disaster)\", f\"Prob: {pred_prob[0][0]}\")\n",
        "  print(f\"Text:\\n{sentence}\")"
      ],
      "execution_count": 116,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o1Yo0bR9mEFN",
        "outputId": "da166a2d-d088-41db-cbbe-ca0e2b3fb550"
      },
      "source": [
        "# Make a prediction on Tweet from the wild\n",
        "predict_on_sentence(model = model_6, sentence = daniels_tweet)"
      ],
      "execution_count": 117,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 0.0 (not real disaster) Prob: 0.0634278953075409\n",
            "Text:\n",
            "Life like an ensemble: take the best choices from others and make your own\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zS9ACuWtmRd-"
      },
      "source": [
        "Woohoo! Our model predicted correctly. My Tweet wasn't about a diaster.\n",
        "\n",
        "How about we find a few Tweets about actual diasters?\n",
        "\n",
        "Such as the following two Tweets about the 2020 Beirut explosions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HdmqTmBJmNdC"
      },
      "source": [
        "# Source - https://twitter.com/BeirutCityGuide/status/1290696551376007168\n",
        "beirut_tweet_1 = \"Reports that the smoke in Beirut sky contains nitric acid, which is toxic. Please share and refrain from stepping outside unless urgent. #Lebanon\"\n",
        "\n",
        "# Source - https://twitter.com/BeirutCityGuide/status/1290773498743476224\n",
        "beirut_tweet_2 = \"#Beirut declared a “devastated city”, two-week state of emergency officially declared. #Lebanon\""
      ],
      "execution_count": 118,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K4sVA_J7mUZJ",
        "outputId": "a0a6db7a-fb4d-4802-8e90-4a7100e72513"
      },
      "source": [
        "# Predict on diaster Tweet 1\n",
        "predict_on_sentence(model=model_6, \n",
        "                    sentence=beirut_tweet_1)\n"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Pred: 1.0 (real disaster) Prob: 0.9758996963500977\n",
            "Text:\n",
            "Reports that the smoke in Beirut sky contains nitric acid, which is toxic. Please share and refrain from stepping outside unless urgent. #Lebanon\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_ObgG99mmbMx"
      },
      "source": [
        "\n",
        "Looks like our model is performing as expected, predicting both of the diaster Tweets as actual diasters.\n",
        "\n",
        "> 🔑 Note: The above examples are cherry-picked and are cases where you'd expect a model to function at high performance. For actual production systems, you'll want to continaully perform tests to see how your model is performing."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "li9XPKm6n2f9"
      },
      "source": [
        "## The speed/score tradeoff\n",
        "One of the final tests we're going to do is to find the speed/score tradeoffs between our best model and baseline model.\n",
        "\n",
        "Why is this important?\n",
        "\n",
        "Although it can be tempting to just choose the best performing model you find through experimentation, this model might not actually work in a production setting.\n",
        "\n",
        "Put it this way, imagine you're Twitter and receive 1 million Tweets per hour (this is a made up number, the actual number is much higher). And you're trying to build a diaster detection system to read Tweets and alert authorities with details about a diaster in close to real-time.\n",
        "\n",
        "Compute power isn't free so you're limited to a single compute machine for the project. On that machine, one of your models makes 10,000 predictions per second at 80% accuracy where as another one of your models (a larger model) makes 100 predictions per second at 85% accuracy.\n",
        "\n",
        "Which model do you choose?\n",
        "\n",
        "Is the second model's performance boost worth missing out on the extra capacity?\n",
        "\n",
        "Of course, there are many options you could try here, such as sending as many Tweets as possible to the first model and then sending the ones which the model is least certain of to the second model.\n",
        "\n",
        "The point here is to illustrate the best model you find through experimentation, might not be the model you end up using in production.\n",
        "\n",
        "To make this more concrete, let's write a function to take a model and a number of samples and time how long the given model takes to make predictions on those samples."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bNJAlnVtmWAt"
      },
      "source": [
        "# Calculate the time of predictions\n",
        "import time\n",
        "def pred_timer(model, samples):\n",
        "  \"\"\"\n",
        "  Times how long a model takes to make predictions on samples.\n",
        "  \n",
        "  Args:\n",
        "  ----\n",
        "  model = a trained model\n",
        "  sample = a list of samples\n",
        "\n",
        "  Returns:\n",
        "  ----\n",
        "  total_time = total elapsed time for model to make predictions on samples\n",
        "  time_per_pred = time in seconds per single sample\n",
        "  \"\"\"\n",
        "  start_time = time.perf_counter() # Get start time\n",
        "  model.predict(samples) # make predictions\n",
        "  end_time = time.perf_counter() # get finish time\n",
        "  total_time = end_time - start_time # Calculate how long predictiosn took to make\n",
        "  time_per_pred = total_time / len(val_sentences) # find prediction time per sample\n",
        "  return total_time, time_per_pred"
      ],
      "execution_count": 120,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jbqdKP3XoYI1"
      },
      "source": [
        "Looking good!\n",
        "\n",
        "Now let's use our pred_timer() function to evaluate the prediction times of our best performing model (model_6) and our baseline model (model_0).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nfiwCgmyoWdk",
        "outputId": "e93844c9-f2b6-4052-d689-a1a20c7a6455"
      },
      "source": [
        "# Calculate TF Hub Sentence Encoder prediction times\n",
        "model_6_total_pred_time, model_6_time_per_pred = pred_timer(model_6, val_sentences)\n",
        "model_6_total_pred_time, model_6_time_per_pred"
      ],
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.21340802800023084, 0.00028006302887169404)"
            ]
          },
          "metadata": {},
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XSXwJz2iopiy",
        "outputId": "f14a590d-4b08-4bcb-9abb-2524a0670367"
      },
      "source": [
        "# Calculate Naive Bayes prediction times\n",
        "baseline_total_pred_time, baseline_time_per_pred = pred_timer(model_0, val_sentences)\n",
        "baseline_total_pred_time, baseline_time_per_pred"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(0.02141306000021359, 2.8101128609204186e-05)"
            ]
          },
          "metadata": {},
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X4Zgsqvwo2mV"
      },
      "source": [
        "\n",
        "It seems with our current hardware (in my case, I'm using a Google Colab notebook) our best performing model takes over 10x the time to make predictions as our baseline model.\n",
        "\n",
        "Is that extra prediction time worth it?\n",
        "\n",
        "Let's compare time per prediction versus our model's F1-scores."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 458
        },
        "id": "cYOs1XhCosFh",
        "outputId": "b8050392-b4df-4d74-e730-a24c7d78a959"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(10, 7))\n",
        "plt.scatter(baseline_time_per_pred, baseline_results[\"f1_score\"], label=\"baseline\")\n",
        "plt.scatter(model_6_time_per_pred, model_6_results[\"f1_score\"], label=\"tf_hub_sentence_encoder\")\n",
        "plt.legend()\n",
        "plt.title(\"F1-score versus time per prediction\")\n",
        "plt.xlabel(\"Time per prediction\")\n",
        "plt.ylabel(\"F1-Score\");"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm4AAAG5CAYAAAA3e7gZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3df7wWdZ3//8dLQDHLVKQthQRLUX4c+XEkf1SiZli6aqUurvbJH2WmZtsWpVtuZvlZzb5ZuphaH6W0RFMz0go2xbQ09bAoioqisgK6hgQaBMqP1/ePa87x4nh+IVyca+Bxv93mxlzvec973jNzjufpe2auicxEkiRJ9W+L7u6AJEmSusbgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTpHUQEf8WET/u7n7Uu4gYExHzqz7Piogxb6KdD0TE7A3aOanEDG5SHYqIuRGxPCKWVk07FcuuiojZEbEmIk7s5q5u0lqHD4DM/L+Z+enu6lNZZeaQzLyrs3oRkRHx3qr17snMQTXtnFQiBjepfv1jZr61anq+KH8YOB34727sGwAR0XNz3HbZbIhjFRE9NkRfJK0fg5tUMpk5ITPvAFZ0VjciekfEdRGxKCKWRMSDEfEPxbIdIuKaiHg+IhZHxK1V630mIuZExF8jYnLzaF+xLCPijIh4CniqKDs8Ih4qtnFvRDS0058fRsR3W5X9KiL+tZjfKSJujoiFEfFsRJxVVe+8iLip2J9XgBMjYnRENEXEKxHxYkR8r6j7hpGyYhTzQ8V8m+u1qr8N8Ftgp+pRz6If1xV1BhTH46SImFccx9MiYu+ImFkcj/9s1e7JEfF4UXdKROzSzrFqbvvU4hy9EBFfrlq+RUScHRFPF+f3xojYodW6p0TEc8CdbbQ/JiLmF5d+XyqOz/FVyycW5+s3EbEMOLCT87N1sc7iiHgM2LuD49+j2O7TEfG3iJgeEf0j4u6i+sPF8f6n1ucyIvaMiLuKYzsrIo5o1ecJEXF70e79EfGeto6vVFqZ6eTkVGcTMBf4UCd1/gic2EmdzwK/Bt4C9ABGAdsWy24HbgC2B3oBBxTlBwEvASOBrYDLgLur2kzgv4AdgK2BEcBfgPcV2/hU0f+t2ujPB4F5QBSftweWAztR+R/J6cC/A1sCuwLPAGOLuucBK4GjirpbA/cBnyyWvxXYp5gfA8xv75i2t14b/W2rnfOA64r5AcXxuALoDXyYSqC+FXgHsHNxbJqP7ZHAHGBPoCfwdeDedrbd3Pb1wDbAMGBh1T58Afgz0K84T1cC17da96fFulu3s2+rgO8V6x8ALAMGFcsnAi8D+xfH+y2dnJ8LgXuKn4v+wKPVx67V8R8PPAIMAgLYC+hT9fP13rbOAZWf0znAvxV9OAj4W6s+LwJGF8f3Z8Ck7v59dnLakJMjblL9urUYVVhSPRq2jlYCfaj8IVydmdMz85WIeBfwEeC0zFycmSsz8w/FOscDV2fmf2fmq8A5wL4RMaCq3f/IzL9m5nLgVODKzLy/2MZPgFeBfdrozz1U/jB/oPh8NHBfVi4D7w30zczzM/O1zHwG+BEwrmr9+zLz1sxcU2x7JfDeiNgxM5dm5p/X4bi8mfXa863MXJGZU6mEn+sz8y+ZuaDY5xFFvdOoHLvHM3MV8H+B4e2NuhW+mZnLMvMR4BrguKq2vpaZ84vzdB5wdKx9WfS8Yt3lHbR/bma+Wpz/24Fjq5b9KjP/lJlrqATHjs7PscAFxc/FPODSDrb5aeDrmTk7Kx7OzEUd1G+2D5WgfWHRhzuB26qOCcAvM/OB4vj+DBjehXal0jC4SfXrqMzcrpiO6soKsfbDDO8GrgWmAJOKy23fiYheVEZE/pqZi9toZifgf5o/ZOZSKqMYO1fVmVc1vwvwpaqQuaRofydaycwEJvH6H9p/pvLHtbmdnVq182/AP7SzXYBTgN2BJ6JyGfjw9o7NBlqvPS9WzS9v4/Nbi/ldgB9U7d9fqYw4VR/b1qr3+X94/bjuAvyyqq3HgdV0fLxaW5yZy9ppv/X6nZ2fndroa3v6A0930re27ATMK4Jk9Xaqj9//Vs3/ndePvbRJ8OZeaROSmW39kfom8M1ixOw3wOzi3x0iYrvMXNKq/vNU/kgDLfd69QEWVG+qan4elZGWC7rYzeuBqRFxIZXLqx+raufZzNytg3VzrQ+ZTwHHRcQWwMeBmyKiD5VRr7dU7UMPoG9n67UKMW/Y3gbQfKx+1mnN1/UHnijm303l/DS3dXJm/qn1ClWjo531f/uI2KZqv99N5RJns9bnuaPz80LR11lVbbVnHvCeVtvqiueB/hGxRVV4ezfw5Dq2I5WWI25SyUTElhHRm8pITa+oPIDQ5u9yRBwYEcOK4PIKlUuEazLzBSo33l8eEdtHRK+I+GCx2vXASRExPCK2onI57/7MnNtOl34EnBYR74uKbSLisIh4W1uVM3MGlXvofgxMqQqODwB/i4ivFje694iIoRGxd1vtFPt3QkT0Lf6IN7ezhsof8t5FP3pRuZdsqy6s19qLQJ+IeHt7fVhHVwDnRMSQoh9vj4hjOlnn3Ih4S7HOSVTuS2xu64Lmy6wR0TcijnwTffpm8TP1AeBw4Bft1Ovs/NxY7Nv2EdEP+HwH2/wx8K2I2K34mWkoAjdUjvmu7ax3P5VRtK8UP7NjgH+kMoorbRYMblL5TKVy+W0/4Kpi/oPt1H0ncBOV0PY48Acql08BPkklyD1B5Qb6fwHIzN8D5wI3UxlFeQ9r32e2lsxsAj4D/CewmMrN4yd2sg8/Bz5U/NvczmoqwWE48Cyvh7uOQtOhwKyIWAr8ABiXmcsz82UqX5nyYyojhcuA+Z2t18a+PUElyD5TXB58w+XfdZGZvwQuonLp+hUqI04f6WS1P1A5pncA3y3uo6Po92Qqo5d/o/KgwvvWsUv/S+WcPU/lkvVpxT631ffOzs83qVy2fJbKz+i1bTTT7HtUgt5UKj+b/4/KwyZQuVfvJ8Xxrr7fjsx8jUpQ+0ix/cuB/9Nen6VNUfOTXZKkOlJc7nwW6FXcaL+h2x9D5enYfhu6bUm144ibJElSSRjcJEmSSsJLpZIkSSXhiJskSVJJbBbf47bjjjvmgAEDursbkiRJnZo+ffpLmdm3rWWbRXAbMGAATU1N3d0NSZKkTkVEu28e8VKpJElSSRjcJEmSSsLgJkmSVBKbxT1ubVm5ciXz589nxYoV3d0VbeZ69+5Nv3796NWrV3d3RZJU5zbb4DZ//nze9ra3MWDAACKiu7ujzVRmsmjRIubPn8/AgQO7uzuSpDq32V4qXbFiBX369DG0qVtFBH369HHkV5LUJZttcAMMbaoL/hxKkrpqsw5ukiRJZWJw60Zz585l6NChNWn7rrvu4vDDDwdg8uTJXHjhhTXZjiRJ2ng224cTNidHHHEERxxxRHd3Q5IkrSdH3Lro1hkL2P/COxl49u3sf+Gd3DpjwQZpd9WqVRx//PHsueeeHH300fz973/n/PPPZ++992bo0KGceuqpZCYAl156KYMHD6ahoYFx48YBsGzZMk4++WRGjx7NiBEj+NWvfvWGbUycOJEzzzwTgBNPPJGzzjqL/fbbj1133ZWbbrqppd7FF1/M3nvvTUNDA9/4xjc2yP5JkqQNx+DWBbfOWMA5tzzCgiXLSWDBkuWcc8sjGyS8zZ49m9NPP53HH3+cbbfdlssvv5wzzzyTBx98kEcffZTly5dz2223AXDhhRcyY8YMZs6cyRVXXAHABRdcwEEHHcQDDzzAtGnTGD9+PMuWLetwmy+88AJ//OMfue222zj77LMBmDp1Kk899RQPPPAADz30ENOnT+fuu+9e7/2TJEkbjsGtCy6eMpvlK1evVbZ85WounjJ7vdvu378/+++/PwAnnHACf/zjH5k2bRrve9/7GDZsGHfeeSezZs0CoKGhgeOPP57rrruOnj0rV7mnTp3KhRdeyPDhwxkzZgwrVqzgueee63CbRx11FFtssQWDBw/mxRdfbGln6tSpjBgxgpEjR/LEE0/w1FNPrff+SZKkDcd73Lrg+SXL16l8XbT+KoiI4PTTT6epqYn+/ftz3nnntXzH1+23387dd9/Nr3/9ay644AIeeeQRMpObb76ZQYMGrdVOcyBry1ZbbdUy33wZNjM555xz+OxnP7ve+yRJ0iZl5o1wx/nw8nx4ez84+N+h4dhu6Yojbl2w03Zbr1P5unjuuee47777APj5z3/O+9//fgB23HFHli5d2nIP2po1a5g3bx4HHnggF110ES+//DJLly5l7NixXHbZZS0BbMaMGW+qH2PHjuXqq69m6dKlACxYsIC//OUv67t7kiSV28wb4ddnwcvzgKz8++uzKuXdwBG3Lhg/dhDn3PLIWpdLt+7Vg/FjB3WwVtcMGjSICRMmcPLJJzN48GA+97nPsXjxYoYOHco73/lO9t57bwBWr17NCSecwMsvv0xmctZZZ7Hddttx7rnn8i//8i80NDSwZs0aBg4c2HJP3Lr48Ic/zOOPP86+++4LwFvf+lauu+463vGOd6z3PkqSVFp3nA8rW11hW7m8Ut4No27RPFKzKWtsbMympqa1yh5//HH23HPPLrdx64wFXDxlNs8vWc5O223N+LGDOGrEzhu6q9pMrevPoyRpIzlvO6CtrBRw3pKabDIipmdmY1vLHHHroqNG7GxQkyRpc/P2fsVl0jbKu4H3uEmSJLXn4H+HXq3uae+1daW8GxjcJEmS2tNwLPzjpfD2/kBU/v3HS7vtqVIvlUqSJHWk4dhuC2qtOeImSZJUEgY3SZKkkjC4SZIklYTBrZssWbKEyy+/vOXz+PHjGTJkCOPHj2+z/oknntjyFoWuGjBgAC+99NJ69XNdff/73+fvf//7Rt1md7rrrrs4/PDDu7sbkqTNhMGtq2beCJcMrXwR3yVD1/tVF62D21VXXcXMmTO5+OKL17en3WpzC27ratWqVd3dBUlSiRncuqIG7yk7++yzefrppxk+fDiHHHIIS5cuZdSoUdxwww3trnP33Xez3377seuuu7aMvrUe8TnzzDOZOHFiy+fvfOc7DBs2jNGjRzNnzpx22/7FL37B0KFD2WuvvfjgBz8IVF6zNX78ePbee28aGhq48sorW7Y5ZswYjj76aPbYYw+OP/54MpNLL72U559/ngMPPJADDzwQgKlTp7LvvvsycuRIjjnmmJZ3oQ4YMIBvfOMbjBw5kmHDhvHEE08AsHTpUk466SSGDRtGQ0MDN998c4fttGX69OkccMABjBo1irFjx/LCCy8AMGbMGL761a8yevRodt99d+65556W/fzyl7/M0KFDaWho4LLLLgPgjjvuYMSIEQwbNoyTTz6ZV199FYDf/e537LHHHowcOZJbbrmlZbvLli3j5JNPZvTo0YwYMYJf/epXAEycOJEjjjiCgw46iIMPPrjdfkuS1KnM3OSnUaNGZWuPPfbYG8ra9b0hmd/Y9o3T94Z0vY1Wnn322Rwy5PX1t9lmmw7rf+pTn8qjjz46V69enbNmzcr3vOc9mZk5bdq0POyww1rqnXHGGXnNNddkZuYuu+yS3/72tzMz8yc/+cla9VobOnRozp8/PzMzFy9enJmZV155ZX7rW9/KzMwVK1bkqFGj8plnnslp06bltttum/PmzcvVq1fnPvvsk/fcc0/LNhcuXJiZmQsXLswPfOADuXTp0szMvPDCC/Ob3/xmS71LL700MzMnTJiQp5xySmZmfuUrX8kvfOELLf3661//2mE7rb322mu577775l/+8pfMzJw0aVKedNJJmZl5wAEH5L/+679mZubtt9+eBx98cGZmXn755fmJT3wiV65cmZmZixYtyuXLl2e/fv1y9uzZmZn5yU9+Mi+55JKW8ieffDLXrFmTxxxzTMtxPeecc/Laa69tOYa77bZbLl26NK+55prceeedc9GiRe0e/3X6eZQkbdKApmwn0/g9bl3x8vx1K6+Ro446ii222ILBgwfz4osvdmmd4447ruXfL37xi+3W23///TnxxBM59thj+fjHPw5URrlmzpzZMrr38ssv89RTT7HlllsyevRo+vWrvO5j+PDhzJ07l/e///1rtfnnP/+Zxx57jP333x+A1157reUl9kDLdkaNGtUycvX73/+eSZMmtdTZfvvtue222zpsp9rs2bN59NFHOeSQQ4DKaNq73vWuNrc5d+7clm2edtpp9OxZ+XXYYYcdePjhhxk4cCC77747AJ/61KeYMGECY8aMYeDAgey2224AnHDCCVx11VUtx2vy5Ml897vfBWDFihU899xzABxyyCHssMMO7R5/SZK6wuDWFXXynrKtttqqZb4SyKFnz56sWbOmpXzFihVrrRMRbc63dsUVV3D//fdz++23M2rUKKZPn05mctlllzF27Ni16t51111r9aVHjx5t3ruVmRxyyCFcf/31He5Pe+t3tZ3WdYcMGcJ99923Xtt8MzKTm2++mUGDBq1Vfv/997PNNtts0G1JkjZP3uPWFTV4T9nb3vY2/va3v61nx2CXXXbhscce49VXX2XJkiXccccday1vvmfuhhtuaHeUCuDpp5/mfe97H+effz59+/Zl3rx5jB07lh/+8IesXLkSgCeffJJly5Z12J/q/dpnn33405/+1HJv3bJly3jyySc7XP+QQw5hwoQJLZ8XL168Tu0MGjSIhQsXtgS3lStXMmvWrE63eeWVV7YEub/+9a8MGjSIuXPntmzz2muv5YADDmCPPfZg7ty5PP300wBrhcmxY8dy2WWXtYTqGTNmdLhdSZLWlcGtK2rwnrI+ffqw//77M3To0Ha/AqQr+vfvz7HHHsvQoUM59thjGTFixFrLFy9eTENDAz/4wQ+45JJL2m1n/PjxDBs2jKFDh7Lffvux11578elPf5rBgwczcuRIhg4dymc/+9lOR6lOPfVUDj30UA488ED69u3LxIkTOe6442hoaGDfffdteQihPV//+tdZvHhxy4MS06ZNW6d2ttxyS2666Sa++tWvstdeezF8+HDuvffeDrf56U9/mne/+900NDSw11578fOf/5zevXtzzTXXcMwxxzBs2DC22GILTjvtNHr37s1VV13FYYcdxsiRI3nHO97R0s65557LypUraWhoYMiQIZx77rkdbleSpHUVzaMDm7LGxsZsampaq+zxxx9nzz337KYeSWvz51GS1CwipmdmY1vLHHGTJEkqCR9OqDMXXHABv/jFL9YqO+aYY/ja175WivY3po997GM8++yza5VddNFFb3iYQpKkTcVmfal0jz326PBJS2ljyEyeeOIJL5VKkgAvlbapd+/eLFq0iM0huKp+ZSaLFi2id+/e3d0VSVIJbLaXSvv168f8+fNZuHBhd3dFm7nevXu3fJmxJEkd2WyDW69evRg4cGB3d0OSJKnLNttLpZIkSWVjcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSqJmga3iDg0ImZHxJyIOLuN5e+OiGkRMSMiZkbER4vyPkX50oj4z1brjIqIR4o2L42IqOU+SJIk1YuaBbeI6AFMAD4CDAaOi4jBrap9HbgxM0cA44DLi/IVwLnAl9to+ofAZ4DdiunQDd97SZKk+lPLEbfRwJzMfCYzXwMmAUe2qpPAtsX824HnATJzWWb+kUqAaxER7wK2zcw/Z2YCPwWOquE+SJIk1Y1aBredgXlVn+cXZdXOA06IiPnAb4DPd6HN+Z20CUBEnBoRTRHRtHDhwnXptyRJUl3q7ocTjgMmZmY/4KPAtRGxQfqUmVdlZmNmNvbt23dDNClJktStahncFgD9qz73K8qqnQLcCJCZ9wG9gR07abNfJ21KkiRtkmoZ3B4EdouIgRGxJZWHDya3qvMccDBAROxJJbi1e10zM18AXomIfYqnSf8P8KtadF6SJKne9KxVw5m5KiLOBKYAPYCrM3NWRJwPNGXmZOBLwI8i4otUHlQ4sXjogIiYS+XBhS0j4ijgw5n5GHA6MBHYGvhtMUmSJG3yoshJm7TGxsZsamrq7m5IkiR1KiKmZ2ZjW8u6++EESZIkdZHBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkqhpcIuIQyNidkTMiYiz21j+7oiYFhEzImJmRHy0atk5xXqzI2JsVfnciHgkIh6KiKZa9l+SJKme9KxVwxHRA5gAHALMBx6MiMmZ+VhVta8DN2bmDyNiMPAbYEAxPw4YAuwE/D4ids/M1cV6B2bmS7XquyRJUj2q5YjbaGBOZj6Tma8Bk4AjW9VJYNti/u3A88X8kcCkzHw1M58F5hTtSZIkbbZqGdx2BuZVfZ5flFU7DzghIuZTGW37fBfWTWBqREyPiFPb23hEnBoRTRHRtHDhwje/F5IkSXWiux9OOA6YmJn9gI8C10ZEZ316f2aOBD4CnBERH2yrUmZelZmNmdnYt2/fDdtrSZKkblDL4LYA6F/1uV9RVu0U4EaAzLwP6A3s2NG6mdn871+AX+IlVEmStJmoZXB7ENgtIgZGxJZUHjaY3KrOc8DBABGxJ5XgtrCoNy4itoqIgcBuwAMRsU1EvK2ovw3wYeDRGu6DJElS3ajZU6WZuSoizgSmAD2AqzNzVkScDzRl5mTgS8CPIuKLVO5dOzEzE5gVETcCjwGrgDMyc3VE/APwy4ho7vvPM/N3tdoHSZKkehKVnLRpa2xszKYmv/JNkiTVv4iYnpmNbS3r7ocTJEmS1EUGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSXQpuEXE7hFxR0Q8WnxuiIiv17ZrkiRJqtbVEbcfAecAKwEycyYwrladkiRJ0ht1Nbi9JTMfaFW2akN3RpIkSe3ranB7KSLeAyRARBwNvFCzXkmSJOkNenax3hnAVcAeEbEAeBY4vma9kiRJ0ht0GtwiogdwemZ+KCK2AbbIzL/VvmuSJEmq1mlwy8zVEfH+Yn5Z7bskSZKktnT1UumMiJgM/AJoCW+ZeUtNeiVJkqQ36Gpw6w0sAg6qKkvA4CZJkrSRdCm4ZeZJb6bxiDgU+AHQA/hxZl7Yavm7gZ8A2xV1zs7M3xTLzgFOAVYDZ2XmlK60KUmStKnq6psT+kXELyPiL8V0c0T062SdHsAE4CPAYOC4iBjcqtrXgRszcwSVL/S9vFh3cPF5CHAocHlE9Ohim5IkSZukrn6P2zXAZGCnYvp1UdaR0cCczHwmM18DJgFHtqqTwLbF/NuB54v5I4FJmflqZj4LzCna60qbkiRJm6SuBre+mXlNZq4qpolA307W2RmYV/V5flFW7TzghIiYD/wG+Hwn63alTQAi4tSIaIqIpoULF3bSVUmSpPrX1eC2KCJOaL5cGREnUHlYYX0dB0zMzH7AR4FrI6KrfepQZl6VmY2Z2di3b2cZU5Ikqf51NSSdDBwL/C+VV10dDXT2wMICoH/V535FWbVTgBsBMvM+Kk+v7tjBul1pU5IkaZPUpeCWmf+TmUdkZt/MfEdmHpWZz3Wy2oPAbhExMCK2pPKwweRWdZ4DDgaIiD2pBLeFRb1xEbFVRAwEdgMe6GKbkiRJm6SuPlX6k4jYrurz9hFxdUfrZOYq4ExgCvA4ladHZ0XE+RFxRFHtS8BnIuJh4HrgxKyYRWUk7jHgd8AZmbm6vTbXZYclSZLKKjKz80oRM4qv7OiwrF41NjZmU1NTd3dDkiSpUxExPTMb21rW1XvctoiI7asa3IGuv3VBkiRJG0BXw9f/B9wXEb8AgsrDCRfUrFeSJEl6g66+8uqnEdFE5V2lCXw8Mx+rac8kSZK0lg4vlUbEWyKiF0AR1P4L2BLYYyP0TZIkSVU6u8ftd8AAgIh4L3AfsCtwRkT4cndJkqSNqLPgtn1mPlXMfwq4PjM/T+Ul74fVtGeSJElaS2fBrfq7Qg6icqmU4gXva2rVKUmSJL1RZw8nzIyI71J5rdR7gakA1V/GK0mSpI2jsxG3zwAvUbnP7cOZ+feifDDw3Rr2S5IkSa10OOKWmcuBtR5CiIiRmXkvcG8tOyZJkqS1dfXNCdV+vMF7IUmSpE69meAWG7wXkiRJ6tSbCW7f3OC9kCRJUqfWObhl5q0AEeHbEyRJkjaiNzPi1mzqBuuFJEmSOtXhU6URcWl7iwC/y02SJGkj6uwLeE8CvgS82say4zZ8dyRJktSezoLbg8Cjxfe2rSUizqtJjyRJktSmzoLb0cCKthZk5sAN3x1JkiS1p7OHE95a9ZorSZIkdaPOgtutzTMRcTMhsJQAABFJSURBVHON+yJJkqQOdBbcqt+SsGstOyJJkqSOdRbcsp15SZIkbWSdPZywV0S8QmXkbetinuJzZua2Ne2dJEmSWnQY3DKzx8bqiCRJkjq2Pq+8kiRJ0kZkcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklURNg1tEHBoRsyNiTkSc3cbySyLioWJ6MiKWVC27KCIeLaZ/qiqfGBHPVq03vJb7IEmSVC961qrhiOgBTAAOAeYDD0bE5Mx8rLlOZn6xqv7ngRHF/GHASGA4sBVwV0T8NjNfKaqPz8ybatV3SZKkelTLEbfRwJzMfCYzXwMmAUd2UP844PpifjBwd2auysxlwEzg0Br2VZIkqe7VMrjtDMyr+jy/KHuDiNgFGAjcWRQ9DBwaEW+JiB2BA4H+VatcEBEzi0utW7XT5qkR0RQRTQsXLlzffZEkSep29fJwwjjgpsxcDZCZU4HfAPdSGYW7D1hd1D0H2APYG9gB+GpbDWbmVZnZmJmNffv2rXH3JUmSaq+WwW0Ba4+S9SvK2jKO1y+TApCZF2Tm8Mw8BAjgyaL8hax4FbiGyiVZSZKkTV4tg9uDwG4RMTAitqQSzia3rhQRewDbUxlVay7rERF9ivkGoAGYWnx+V/FvAEcBj9ZwHyRJkupGzZ4qzcxVEXEmMAXoAVydmbMi4nygKTObQ9w4YFJmZtXqvYB7KtmMV4ATMnNVsexnEdGXyijcQ8BptdoHSZKkehJr56VNU2NjYzY1NXV3NyRJkjoVEdMzs7GtZfXycIIkSZI6YXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkDG6SJEklYXCTJEkqCYObJElSSRjcJEmSSsLgJkmSVBIGN0mSpJIwuEmSJJWEwU2SJKkkahrcIuLQiJgdEXMi4uw2ll8SEQ8V05MRsaRq2UUR8Wgx/VNV+cCIuL9o84aI2LKW+yBJklQvahbcIqIHMAH4CDAYOC4iBlfXycwvZubwzBwOXAbcUqx7GDASGA68D/hyRGxbrHYRcElmvhdYDJxSq32QJEmqJ7UccRsNzMnMZzLzNWAScGQH9Y8Dri/mBwN3Z+aqzFwGzAQOjYgADgJuKur9BDiqJr2XJEmqM7UMbjsD86o+zy/K3iAidgEGAncWRQ9TCWpviYgdgQOB/kAfYElmrupCm6dGRFNENC1cuHC9d0aSJKm71cvDCeOAmzJzNUBmTgV+A9xLZRTuPmD1ujSYmVdlZmNmNvbt23dD91eSJGmjq2VwW0BllKxZv6KsLeN4/TIpAJl5QXH/2yFAAE8Ci4DtIqJnF9qUJEnapNQyuD0I7FY8BbollXA2uXWliNgD2J7KqFpzWY+I6FPMNwANwNTMTGAacHRR9VPAr2q4D5IkSXWjZ+dV3pzMXBURZwJTgB7A1Zk5KyLOB5oysznEjQMmFaGsWS/gnsqzCLwCnFB1X9tXgUkR8W1gBvD/arUPkiRJ9STWzkubpsbGxmxqaurubkiSJHUqIqZnZmNby+rl4QRJkiR1wuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKwuAmSZJUEgY3SZKkkjC4SZIklYTBTZIkqSQMbpIkSSVhcJMkSSoJg5skSVJJGNwkSZJKomd3d6Dsbp2xgIunzOb5JcvZabutGT92EEeN2Lm7uyVJkjZBBrf1cOuMBZxzyyMsX7kagAVLlnPOLY8AGN4kSdIG56XS9XDxlNktoa3Z8pWruXjK7G7qkSRJ2pQZ3NbD80uWr1O5JEnS+jC4rYedttt6ncolSZLWh8FtPYwfO4ite/VYq2zrXj0YP3ZQN/VIkiRtynw4YT00P4DgU6WSJGljMLitp6NG7GxQkyRJG4WXSiVJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwkyRJKomaBreIODQiZkfEnIg4u43ll0TEQ8X0ZEQsqVr2nYiYFRGPR8SlERFF+V1Fm83rvaOW+yBJklQvavYFvBHRA5gAHALMBx6MiMmZ+Vhzncz8YlX9zwMjivn9gP2BhmLxH4EDgLuKz8dnZlOt+i5JklSPajniNhqYk5nPZOZrwCTgyA7qHwdcX8wn0BvYEtgK6AW8WMO+SpIk1b1aBredgXlVn+cXZW8QEbsAA4E7ATLzPmAa8EIxTcnMx6tWuaa4THpu8yXUNto8NSKaIqJp4cKF6783kiRJ3axeHk4YB9yUmasBIuK9wJ5APyph76CI+EBR9/jMHAZ8oJg+2VaDmXlVZjZmZmPfvn1rvgOSJEm1VsuXzC8A+ld97leUtWUccEbV548Bf87MpQAR8VtgX+CezFwAkJl/i4ifU7kk+9OOOjJ9+vSXIuJ/3tRebLp2BF7q7k5oLZ6T+uM5qS+ej/rjOamNXdpbUMvg9iCwW0QMpBLYxgH/3LpSROwBbA/cV1X8HPCZiPgPIKg8mPD9iOgJbJeZL0VEL+Bw4PeddSQzHXJrJSKaMrOxu/uh13lO6o/npL54PuqP52Tjq9ml0sxcBZwJTAEeB27MzFkRcX5EHFFVdRwwKTOzquwm4GngEeBh4OHM/DWVBxWmRMRM4CEqgfBHtdoHSZKkehJr5yVtLvy/pPrjOak/npP64vmoP56Tja9eHk7QxndVd3dAb+A5qT+ek/ri+ag/npONzBE3SZKkknDETZIkqSQMbpIkSSVhcCuRiDg0ImZHxJyIOLuN5VtFxA3F8vsjYkDVsnOK8tkRMbazNiNiYNHGnKLNLYvyEyNiYfHmioci4tO13ev6tpHPyZlFWUbEjlXlERGXFstmRsTI2u1x/auTczImIl6u+j3599rtcX3byOfjZ0X5oxFxdfG1Uf6OtFIn58TfkTcrM51KMAE9qHxFyq5U3uH6MDC4VZ3TgSuK+XHADcX84KL+VlReLfZ00V67bQI3AuOK+SuAzxXzJwL/2d3Hox6mbjgnI4ABwFxgx6ptfBT4LZXvPNwHuL+7j43nhDHAbd19PLp76obz8dHi9yCovPv6c1Xl/o7U1znxd+RNTo64lcdoYE5mPpOZrwGTgCNb1TkS+EkxfxNwcEREUT4pM1/NzGeBOUV7bbZZrHNQ0QZFm0fVcN/KaqOdE4DMnJGZc9vox5HAT7Piz8B2EfGuDbqn5VEv50QVG/t8/Kb4PUjgASpv7Gnehr8jFfVyTvQmGdzKY2dgXtXn+UVZm3Wy8gXILwN9Oli3vfI+wJKijba29YnicsNNEVH9WrPNzcY8J+vbj81FvZwTgH0j4uGI+G1EDFmXndiEdMv5KC7HfRL43Tr0Y3NRL+cE/B15UwxuWle/BgZkZgPwX7z+f2WSXvffwC6ZuRdwGXBrN/dnc3M5cHdm3tPdHVGL1ufE35E3yeBWHguA6tGtfkVZm3Wi8l7XtwOLOli3vfJFVC4l9GxVTmYuysxXi/IfA6PWa6/KbWOek/Xtx+aiLs5JZr6SmUuL+d8AvaofXtiMbPTzERHfAPoC/7qO/dhc1MU58XdkPXT3TXZOXZuAnsAzVG4Ibb75c0irOmew9g2lNxbzQ1j7htJnqNxM2m6bwC9Y++GE04v5d1Vt72PAn7v72Gwu56SqzbmsfSP8Yax94/UD3X1sPCe8k9e/4Hw08Fzz581p6ob/bn0auBfYutU2/B2pv3Pi78ibPYfd3QGndThZladznqTy9M7XirLzgSOK+d5UAtccKjeB7lq17teK9WYDH+mozaJ816KNOUWbWxXl/wHMKn4xpwF7dPdx2YzOyVlU7h1ZBTwP/LgoD2BCUf8RoLG7j4vnhDOrfk/+DOzX3cdlMzkfq4qyh4rp34tyf0fq75z4O/ImJ195JUmSVBLe4yZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJAxukiRJJWFwk9QtIqJPRDxUTP8bEQuK+aURcXl3929jiogBEfFoMd8YEZd2Uv/fWn2+t5b9k1Q//DoQSd0uIs4Dlmbmd7u7L22JiJ75+rt7N/h6ETEAuC0zh3ax3aWZ+dZ17Y+k8nPETVJdiYgxEXFbMX9eRPwkIu6JiP+JiI9HxHci4pGI+F3x4moiYlRE/CEipkfElIh4VxvtToyIKyKiKSKejIjDi/IeEXFxRDwYETMj4rNV/bgnIiYDj7XR3tKIuCQiZkXEHRHRtyi/KyK+HxFNwBfa61tR/nBEPEzlm+rb2v+3RsQ1xf7OjIhPRMSFwNbF6OTPmvtS/BvFvjxarPNPVW3eFRE3RcQTEfGziIgNdc4kbTwGN0n17j3AQcARwHXAtMwcBiwHDivC22XA0Zk5CrgauKCdtgZQeb3OYcAVEdEbOAV4OTP3BvYGPhMRA4v6I4EvZObubbS1DdCUmUOAPwDfqFq2ZWY2Apd20LdrgM9n5SXb7Tm36NuwzGwA7szMs4HlmTk8M49vVf/jwHBgL+BDwMVVIXYE8C/AYCpvRtm/g+1KqlM9O68iSd3qt5m5MiIeofJexN8V5Y9QCWKDgKHAfxWDSD2AF9pp68bMXAM8FRHPAHsAHwYaIuLoos7bgd2A16i80/LZdtpaA9xQzF8H3FK1rLm8zb5FxHbAdpl5d1HvWuAjbWzjQ1TeFQlAZi5upy/N3g9cn5mrgRcj4g9Uwugrxb7MB4iIh6gcuz920p6kOmNwk1TvXgXIzDURsTJfvzF3DZX/hgUwKzP37UJbrW/qzWL9z2fmlOoFETEGWLYO/axuu3m9NvtWBLeN7dWq+dX433+plLxUKqnsZgN9I2JfgIjoFRFD2ql7TERsERHvoXK5cDYwBfhc1f1yu0fENl3Y7hZA8yjdP9P26FWbfcvMJcCSiHh/Ua/1Jc9m/8Xa979tX8yubO5vK/cA/1Tct9cX+CCVl4RL2kQY3CSVWma+RiVAXVTc6P8QsF871Z+jEmR+C5yWmSuAH1N5+OC/i6/kuJKujUYtA0YX6xwEnL+OfTsJmFBctmzvQYFvA9sXDxs8DBxYlF8FzGx+OKHKL4GZwMPAncBXMvN/u7AvkkrCrwORtFmIiIlUvnLjpg3Unl/JIWmjc8RNkiSpJBxxkyRJKglH3CRJkkrC4CZJklQSBjdJkqSSMLhJkiSVhMFNkiSpJP5/llDW8bA2thAAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xS7H2n5NpIAx"
      },
      "source": [
        "Of course, the ideal position for each of these dots is to be in the top left of the plot (low time per prediction, high F1-score).\n",
        "\n",
        "In our case, there's a clear tradeoff for time per prediction and performance. Our best performing model takes an order of magnitude longer per prediction but only results in a few F1-score point increase.\n",
        "\n",
        "This kind of tradeoff is something you'll need to keep in mind when incorporating machine learning models into your own applications."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y44zdBzopLQD"
      },
      "source": [
        "# 🛠 Exercises\n",
        "- Rebuild, compile and train \n",
        "model_1, model_2 and model_5 using the Keras Sequential API instead of the Functional API.\n",
        "- Retrain the baseline model with 10% of the training data. How does perform compared to the Universal Sentence Encoder model with 10% of the training data?\n",
        "- Try fine-tuning the TF Hub Universal Sentence Encoder model by setting training=True when instantiating it as a Keras layer.\n",
        "We can use this encoding layer in place of our text_vectorizer and embedding layer\n",
        "\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=True) # turn training on to fine-tune the TensorFlow Hub model\n",
        "- Retrain the best model you've got so far on the whole training set (no validation split). Then use this trained model to make predictions on the test dataset and format the predictions into the same format as the sample_submission.csv file from Kaggle (see the Files tab in Colab for what the sample_submission.csv file looks like). Once you've done this, make a submission to the Kaggle competition, how did your model perform?\n",
        "- Combine the ensemble predictions using the majority vote (mode), how does this perform compare to averaging the prediction probabilities of each model?\n",
        "- Make a confusion matrix with the best performing model's predictions on the validation set and the validation ground truth labels."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8Zn2SbeRo5ER"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}