09_SkimLit_nlp_milestone_project_2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "09_SkimLit_nlp_milestone_project_2.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyOO8nggscsZQtEtuMoYSgx4",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/prachuryanath/TF-Learning/blob/main/09_SkimLit_nlp_milestone_project_2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yc96wwcKO-El"
      },
      "source": [
        "# Milestone Project 2 : SkimLit \n",
        "\n",
        "In this project, we're going to be putting what we've learned into practice.\n",
        "\n",
        "More specificially, we're going to be replicating the deep learning model behind the 2017 paper `PubMed 200k RCT: a Dataset for Sequenctial Sentence Classification in Medical Abstracts`.\n",
        "\n",
        "When it was released, the paper presented a new dataset called `PubMed 200k RCT which consists of ~200,000 labelled Randomized Controlled Trial (RCT) abstracts`.\n",
        "\n",
        "The goal of the dataset was to explore the ability for NLP models to classify sentences which appear in sequential order.\n",
        "\n",
        "### **Problem in a sentence**\n",
        "*The number of RCT papers released is continuing to increase, those without structured abstracts can be hard to read and in turn slow down researchers moving through the literature.*\n",
        "\n",
        "### **Solution in a sentence**\n",
        "*Create an NLP model to classify abstract sentences into the role they play (e.g. objective, methods, results, etc) to enable researchers to skim through the literature (hence SkimLit 🤓🔥) and dive deeper when necessary.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49iOYPcIPeRI"
      },
      "source": [
        "## What we're going to cover\n",
        "Time to take what we've learned in the NLP fundmentals notebook and build our biggest NLP model yet:\n",
        "\n",
        "* Downloading a text dataset (PubMed RCT200k from GitHub)\n",
        "* Writing a preprocessing function to prepare our data for modelling\n",
        "* Setting up a series of modelling experiments\n",
        "  * Making a baseline (TF-IDF classifier)\n",
        "  * Deep models with different combinations of: token embeddings, character embeddings, pretrained embeddings, positional embeddings\n",
        "* Building our first multimodal model (taking multiple types of data inputs)\n",
        "  * Replicating the model architecture from https://arxiv.org/pdf/1612.05251.pdf\n",
        "* Find the most wrong predictions\n",
        "* Making predictions on PubMed abstracts from the wild"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vbHryq10QGlA",
        "outputId": "2d5c0ebb-2649-4de3-fe20-9b4b22d95fdc"
      },
      "source": [
        "# Check for GPU\n",
        "!nvidia-smi"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sun Jun 27 16:49:47 2021       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 465.27       Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   55C    P8    10W /  70W |      0MiB / 15109MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aXj_Ou67QU0U"
      },
      "source": [
        "## Get data\n",
        "\n",
        "Before we can start building a model, we've got to download the PubMed 200k RCT dataset.\n",
        "\n",
        "In a phenomenal act of kindness, the authors of the paper have made the data they used for their research availably publically and for free in the form of .txt files on GitHub.\n",
        "\n",
        "We can copy them to our local directory using git clone `https://github.com/Franck-Dernoncourt/pubmed-rct.`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rQ5b5_lMQuJV",
        "outputId": "1476113c-f23e-4105-ab86-3d83916e97be"
      },
      "source": [
        "!git clone https://github.com/Franck-Dernoncourt/pubmed-rct.git\n",
        "!ls pubmed-rct"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'pubmed-rct'...\n",
            "remote: Enumerating objects: 33, done.\u001b[K\n",
            "remote: Counting objects: 100% (3/3), done.\u001b[K\n",
            "remote: Compressing objects: 100% (3/3), done.\u001b[K\n",
            "remote: Total 33 (delta 0), reused 0 (delta 0), pack-reused 30\u001b[K\n",
            "Unpacking objects: 100% (33/33), done.\n",
            "PubMed_200k_RCT\n",
            "PubMed_200k_RCT_numbers_replaced_with_at_sign\n",
            "PubMed_20k_RCT\n",
            "PubMed_20k_RCT_numbers_replaced_with_at_sign\n",
            "README.md\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FSrJNVHRQ0aQ"
      },
      "source": [
        "Looking at the README file from the GitHub page, we get the following information:\n",
        "\n",
        "* `PubMed 20k` is a subset of `PubMed 200k`. I.e., any abstract present in `PubMed 20k` is also present in `PubMed 200k`.\n",
        "* `PubMed_200k_RCT` is the same as `PubMed_200k_RCT_numbers_replaced_with_at_sign`, except that in the latter all numbers had been replaced by @. (same for `PubMed_20k_RCT` vs. `PubMed_20k_RCT_numbers_replaced_with_at_sign`).\n",
        "* Since Github file size limit is 100 MiB, we had to compress `PubMed_200k_RCT\\train.7z` and `PubMed_200k_RCT_numbers_replaced_with_at_sign\\train.zip`. To uncompress train.7z, you may use 7-Zip on Windows, Keka on Mac OS X, or p7zip on Linux.\n",
        "\n",
        "To begin with, the dataset we're going to be focused on is `PubMed_20k_RCT_numbers_replaced_with_at_sign`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z1liAnC0R5Zc",
        "outputId": "c890bf4e-15d5-413c-8e68-cb93c2df4045"
      },
      "source": [
        "# Check what files are in the PubMed_20K dataset \n",
        "!ls pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "dev.txt  test.txt  train.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9v0_aP8SJWc"
      },
      "source": [
        "Beautiful, looks like we've got three separate text files:\n",
        "\n",
        "* **train.txt** - *training samples.*\n",
        "* **dev.txt** - *dev is short for development set, which is another name for validation set (in our case, we'll be using and referring to this file as our validation set).*\n",
        "* **test.txt** - *test samples.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z72jYlzJSATJ"
      },
      "source": [
        "# Start by using the 20k dataset\n",
        "data_dir = \"pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/\""
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AXbC3PBRSDHX",
        "outputId": "95eb8823-d829-42a6-da4c-abc24733c0c6"
      },
      "source": [
        "# Check all of the filenames in the target directory\n",
        "import os\n",
        "filenames = [data_dir + filename for filename in os.listdir(data_dir)]\n",
        "filenames"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/dev.txt',\n",
              " 'pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/train.txt',\n",
              " 'pubmed-rct/PubMed_20k_RCT_numbers_replaced_with_at_sign/test.txt']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pc8I6W8nSE75"
      },
      "source": [
        "## Preprocess data\n",
        "\n",
        "To get familiar and understand how we have to prepare our data for our deep learning models, we've got to visualize it.\n",
        "\n",
        "Because our data is in the form of text files, let's write some code to read each of the lines in a target file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SXXE_zSsShAx"
      },
      "source": [
        "# Create function to read the lines of a document\n",
        "def get_lines(filename):\n",
        "  \"\"\"\n",
        "  Reads filename (a text file) and returns the lines of text as a list\n",
        "\n",
        "  Args:\n",
        "    filename : a string containing the target filepath to read.\n",
        "\n",
        "  Returns:\n",
        "    A list of strings with one string per line from the target filename.\n",
        "    For example:\n",
        "    [\"this is the first line of filename\",\n",
        "    \"this is the second line of filename\",\n",
        "    ...]\n",
        "  \"\"\"\n",
        "  with open(filename, \"r\") as f:\n",
        "    return f.readlines()"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SLH534ogTwad",
        "outputId": "20dae020-fb9b-4cdb-ed19-c1080a7e2792"
      },
      "source": [
        "train_lines = get_lines(data_dir+\"train.txt\")\n",
        "train_lines[:10]"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['###24293578\\n',\n",
              " 'OBJECTIVE\\tTo investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( OA ) .\\n',\n",
              " 'METHODS\\tA total of @ patients with primary knee OA were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .\\n',\n",
              " 'METHODS\\tOutcome measures included pain reduction and improvement in function scores and systemic inflammation markers .\\n',\n",
              " 'METHODS\\tPain was assessed using the visual analog pain scale ( @-@ mm ) .\\n',\n",
              " 'METHODS\\tSecondary outcome measures included the Western Ontario and McMaster Universities Osteoarthritis Index scores , patient global assessment ( PGA ) of the severity of knee OA , and @-min walk distance ( @MWD ) .\\n',\n",
              " 'METHODS\\tSerum levels of interleukin @ ( IL-@ ) , IL-@ , tumor necrosis factor ( TNF ) - , and high-sensitivity C-reactive protein ( hsCRP ) were measured .\\n',\n",
              " 'RESULTS\\tThere was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , PGA , and @MWD at @ weeks .\\n',\n",
              " 'RESULTS\\tThe mean difference between treatment arms ( @ % CI ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .\\n',\n",
              " 'RESULTS\\tFurther , there was a clinically relevant reduction in the serum levels of IL-@ , IL-@ , TNF - , and hsCRP at @ weeks in the intervention group when compared to the placebo group .\\n']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwXHgushT7pY"
      },
      "source": [
        "Let's write a function to perform the following steps:\n",
        "\n",
        "* Take a target file of abstract samples.\n",
        "* Read the lines in the target file.\n",
        "* For each line in the target file:\n",
        "  * If the line begins with ### mark it as an abstract ID and the beginning of a new abstract.\n",
        "    * Keep count of the number of lines in a sample.\n",
        "  * If the line begins with \\n mark it as the end of an abstract sample.\n",
        "    * Keep count of the total lines in a sample.\n",
        "  * Record the text before the \\t as the label of the line.\n",
        "  * Record the text after the \\t as the text of the line.\n",
        "* Return all of the lines in the target text file as a list of dictionaries containing the key/value pairs:\n",
        "  * \"line_number\" - the position of the line in the abstract (e.g. 3).\n",
        "  * \"target\" - the role of the line in the abstract (e.g. OBJECTIVE).\n",
        "  * \"text\" - the text of the line in the abstract.\n",
        "  * \"total_lines\" - the total lines in an abstract sample (e.g. 14).\n",
        "* Abstract ID's and newlines should be omitted from the returned preprocessed data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S1deHBAEUhMe"
      },
      "source": [
        "def preprocess_text_with_line_numbers(filename):\n",
        "  \"\"\"\n",
        "  Returns a list of dictionaries of abstract line data.\n",
        "\n",
        "  Takes in filename, reads its contents and sorts through each line,\n",
        "  extracting things like the target label, the text of the sentence,\n",
        "  how many sentences are in the current abstract and what sentence\n",
        "  number the target line is.\n",
        "\n",
        "  Args:\n",
        "    filename : a string of the target text file to read and extract line data from.\n",
        "  \n",
        "  Returns:\n",
        "     A list of dictionaries each containing a line from an abstract,\n",
        "     the lines label, the lines position in the abstract and the total\n",
        "     number of lines in the abstract where the line is from. For example:\n",
        "\n",
        "     [{\"target\":'Conclusion',\n",
        "     \"text\": \"The study couldn't have gone better\",\n",
        "     \"line_number\":8,\n",
        "     \"total_lines\":9}]\n",
        "  \"\"\"\n",
        "  input_lines = get_lines(filename)     # get all lines from filename\n",
        "  abstract_lines = \"\"                   # create an empty abstract\n",
        "  abstract_samples = []                 # create an empty list of abstracts\n",
        "\n",
        "  # Loop through each line in target file\n",
        "  for line in input_lines:\n",
        "    if line.startswith('###'):          # check to see if line is an ID line\n",
        "      abstract_id = line\n",
        "      abstract_lines = \"\" # reset abstract string\n",
        "    elif line.isspace(): # check to see if line is a new line\n",
        "      abstract_line_split = abstract_lines.splitlines() # split abstract into separate lines\n",
        "\n",
        "      # Iterate through each line in abstract and count them at the same time\n",
        "      for abstract_line_number, abstract_line in enumerate(abstract_line_split):\n",
        "        line_data = {} # create empty dict to store data from line\n",
        "        target_text_split = abstract_line.split(\"\\t\") # split target label from text\n",
        "        line_data[\"target\"] = target_text_split[0] # get target label\n",
        "        line_data[\"text\"] = target_text_split[1].lower() # get target text and lower it\n",
        "        line_data[\"line_number\"] = abstract_line_number # what number line does the line appear in the abstract?\n",
        "        line_data[\"total_lines\"] = len(abstract_line_split) - 1 # how many total lines are in the abstract? (start from 0)\n",
        "        abstract_samples.append(line_data) # add line data to abstract samples list\n",
        "    \n",
        "    else: # if the above conditions aren't fulfilled, the line contains a labelled sentence\n",
        "      abstract_lines += line\n",
        "  \n",
        "  return abstract_samples"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WOZpSqPEaC0K",
        "outputId": "6f4e7274-c864-4463-8210-e3c49b7ffc4c"
      },
      "source": [
        "# Get data from file and preprocess it\n",
        "%%time\n",
        "train_samples = preprocess_text_with_line_numbers(data_dir + \"train.txt\")\n",
        "val_samples = preprocess_text_with_line_numbers(data_dir + \"dev.txt\") # dev is another name for validation set\n",
        "test_samples = preprocess_text_with_line_numbers(data_dir + \"test.txt\")\n",
        "len(train_samples), len(val_samples), len(test_samples)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 424 ms, sys: 88.9 ms, total: 513 ms\n",
            "Wall time: 513 ms\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EgZfGCaiaHR0",
        "outputId": "2ad7b170-da9a-4776-d16f-2879137e9ed9"
      },
      "source": [
        "# Check the first abstract of our training data\n",
        "train_samples[:10]"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'line_number': 0,\n",
              "  'target': 'OBJECTIVE',\n",
              "  'text': 'to investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( oa ) .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 1,\n",
              "  'target': 'METHODS',\n",
              "  'text': 'a total of @ patients with primary knee oa were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 2,\n",
              "  'target': 'METHODS',\n",
              "  'text': 'outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 3,\n",
              "  'target': 'METHODS',\n",
              "  'text': 'pain was assessed using the visual analog pain scale ( @-@ mm ) .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 4,\n",
              "  'target': 'METHODS',\n",
              "  'text': 'secondary outcome measures included the western ontario and mcmaster universities osteoarthritis index scores , patient global assessment ( pga ) of the severity of knee oa , and @-min walk distance ( @mwd ) .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 5,\n",
              "  'target': 'METHODS',\n",
              "  'text': 'serum levels of interleukin @ ( il-@ ) , il-@ , tumor necrosis factor ( tnf ) - , and high-sensitivity c-reactive protein ( hscrp ) were measured .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 6,\n",
              "  'target': 'RESULTS',\n",
              "  'text': 'there was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , pga , and @mwd at @ weeks .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 7,\n",
              "  'target': 'RESULTS',\n",
              "  'text': 'the mean difference between treatment arms ( @ % ci ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 8,\n",
              "  'target': 'RESULTS',\n",
              "  'text': 'further , there was a clinically relevant reduction in the serum levels of il-@ , il-@ , tnf - , and hscrp at @ weeks in the intervention group when compared to the placebo group .',\n",
              "  'total_lines': 11},\n",
              " {'line_number': 9,\n",
              "  'target': 'RESULTS',\n",
              "  'text': 'these differences remained significant at @ weeks .',\n",
              "  'total_lines': 11}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 483
        },
        "id": "v0sJK23Wam7F",
        "outputId": "204cb14b-e68f-46a1-c3a7-a0a286093d47"
      },
      "source": [
        "import pandas as pd\n",
        "train_df = pd.DataFrame(train_samples)\n",
        "val_df = pd.DataFrame(val_samples)\n",
        "test_df = pd.DataFrame(test_samples)\n",
        "train_df.head(14)"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>text</th>\n",
              "      <th>line_number</th>\n",
              "      <th>total_lines</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>to investigate the efficacy of @ weeks of dail...</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>a total of @ patients with primary knee oa wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>outcome measures included pain reduction and i...</td>\n",
              "      <td>2</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>pain was assessed using the visual analog pain...</td>\n",
              "      <td>3</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>secondary outcome measures included the wester...</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>serum levels of interleukin @ ( il-@ ) , il-@ ...</td>\n",
              "      <td>5</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>there was a clinically relevant reduction in t...</td>\n",
              "      <td>6</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the mean difference between treatment arms ( @...</td>\n",
              "      <td>7</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>further , there was a clinically relevant redu...</td>\n",
              "      <td>8</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>these differences remained significant at @ we...</td>\n",
              "      <td>9</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the outcome measures in rheumatology clinical ...</td>\n",
              "      <td>10</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>low-dose oral prednisolone had both a short-te...</td>\n",
              "      <td>11</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>emotional eating is associated with overeating...</td>\n",
              "      <td>0</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>yet , empirical evidence for individual ( trai...</td>\n",
              "      <td>1</td>\n",
              "      <td>10</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         target  ... total_lines\n",
              "0     OBJECTIVE  ...          11\n",
              "1       METHODS  ...          11\n",
              "2       METHODS  ...          11\n",
              "3       METHODS  ...          11\n",
              "4       METHODS  ...          11\n",
              "5       METHODS  ...          11\n",
              "6       RESULTS  ...          11\n",
              "7       RESULTS  ...          11\n",
              "8       RESULTS  ...          11\n",
              "9       RESULTS  ...          11\n",
              "10      RESULTS  ...          11\n",
              "11  CONCLUSIONS  ...          11\n",
              "12   BACKGROUND  ...          10\n",
              "13   BACKGROUND  ...          10\n",
              "\n",
              "[14 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "z4A6lpD1a61i",
        "outputId": "31b512d1-30e9-4848-8bff-254d4ac44bb8"
      },
      "source": [
        "# Distribution of labels in training data\n",
        "train_df.target.value_counts()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "METHODS        59353\n",
              "RESULTS        57953\n",
              "CONCLUSIONS    27168\n",
              "BACKGROUND     21727\n",
              "OBJECTIVE      13839\n",
              "Name: target, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "id": "5mqVEHKpbH3o",
        "outputId": "d436df37-6732-486f-fe60-53e8b94c58b7"
      },
      "source": [
        "train_df.total_lines.plot.hist();"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD6CAYAAABgZXp6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXpUlEQVR4nO3df7BfdX3n8efLRCpSkVDSLJNgg21Gl7r+gCvg1HatjCHg1tBdl4WtS5ZhiDNgV8f9QXQ6i8Uyk+5spdJatqlkTVwV8SfZEppGxHb7Bz+CIAjo5IqwJAJJDRDRFhZ97x/fz5Wv4ebyzbn53i/35vmY+c49530+55zPZ74TXpxzPt/vN1WFJEldvGjUHZAkzV6GiCSpM0NEktSZISJJ6swQkSR1ZohIkjobWogkeVWSO/tee5O8L8nRSbYm2d7+Lmjtk+TKJONJ7kpyYt+xVrX225Os6quflOTuts+VSTKs8UiSnisz8TmRJPOAncApwMXAnqpam2QNsKCqLklyJvC7wJmt3Uer6pQkRwPbgDGggNuBk6rqsSS3Av8BuAXYDFxZVTdM1Zdjjjmmli5dOpRxStJcdPvtt/99VS2cbNv8GerDacB3qurBJCuBt7T6BuBrwCXASmBj9VLt5iRHJTm2td1aVXsAkmwFViT5GnBkVd3c6huBs4ApQ2Tp0qVs27bt4I5OkuawJA/ub9tMPRM5B/hMW15UVQ+35UeARW15MfBQ3z47Wm2q+o5J6pKkGTL0EElyGPAO4HP7bmtXHUO/n5ZkdZJtSbbt3r172KeTpEPGTFyJnAF8vaoebeuPtttUtL+7Wn0ncFzffktabar6kknqz1FV66pqrKrGFi6c9LaeJKmDmQiRc3n2VhbAJmBihtUq4Lq++nltltapwBPtttcWYHmSBW0m13JgS9u2N8mpbVbWeX3HkiTNgKE+WE9yBPA24N195bXAtUkuAB4Ezm71zfRmZo0DPwLOB6iqPUk+DNzW2l028ZAduAj4BHA4vQfqUz5UlyQdXDMyxfeFZGxsrJydJUmDS3J7VY1Nts1PrEuSOjNEJEmdGSKSpM5m6hPrmqWWrrl+JOd9YO3bR3JeSQfGKxFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSps6GGSJKjknw+ybeS3JfkTUmOTrI1yfb2d0FrmyRXJhlPcleSE/uOs6q1355kVV/9pCR3t32uTJJhjkeS9LOGfSXyUeCvqurVwOuA+4A1wI1VtQy4sa0DnAEsa6/VwFUASY4GLgVOAU4GLp0Intbmwr79Vgx5PJKkPkMLkSQvB34DuBqgqp6uqseBlcCG1mwDcFZbXglsrJ6bgaOSHAucDmytqj1V9RiwFVjRth1ZVTdXVQEb+44lSZoBw7wSOR7YDfzPJHck+XiSI4BFVfVwa/MIsKgtLwYe6tt/R6tNVd8xSV2SNEOGGSLzgROBq6rqDcAPefbWFQDtCqKG2AcAkqxOsi3Jtt27dw/7dJJ0yBhmiOwAdlTVLW398/RC5dF2K4r2d1fbvhM4rm//Ja02VX3JJPXnqKp1VTVWVWMLFy6c1qAkSc8aWohU1SPAQ0le1UqnAfcCm4CJGVargOva8ibgvDZL61TgiXbbawuwPMmC9kB9ObClbdub5NQ2K+u8vmNJkmbA/CEf/3eBTyU5DLgfOJ9ecF2b5ALgQeDs1nYzcCYwDvyotaWq9iT5MHBba3dZVe1pyxcBnwAOB25oL0nSDBlqiFTVncDYJJtOm6RtARfv5zjrgfWT1LcBr5lmNyVJHfmJdUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOhtqiCR5IMndSe5Msq3Vjk6yNcn29ndBqyfJlUnGk9yV5MS+46xq7bcnWdVXP6kdf7ztm2GOR5L0s2biSuQ3q+r1VTXW1tcAN1bVMuDGtg5wBrCsvVYDV0EvdIBLgVOAk4FLJ4Kntbmwb78Vwx+OJGnCKG5nrQQ2tOUNwFl99Y3VczNwVJJjgdOBrVW1p6oeA7YCK9q2I6vq5qoqYGPfsSRJM2DYIVLAXye5PcnqVltUVQ+35UeARW15MfBQ3747Wm2q+o5J6s+RZHWSbUm27d69ezrjkST1mT/k47+5qnYm+UVga5Jv9W+sqkpSQ+4DVbUOWAcwNjY29PNJ0qFiqFciVbWz/d0FfIneM41H260o2t9drflO4Li+3Ze02lT1JZPUJUkzZGghkuSIJC+bWAaWA98ENgETM6xWAde15U3AeW2W1qnAE+221xZgeZIF7YH6cmBL27Y3yaltVtZ5fceSJM2AYd7OWgR8qc26nQ98uqr+KsltwLVJLgAeBM5u7TcDZwLjwI+A8wGqak+SDwO3tXaXVdWetnwR8AngcOCG9pIkzZChhUhV3Q+8bpL694HTJqkXcPF+jrUeWD9JfRvwmml3VpLUiZ9YlyR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktTZQCGS5J8NuyOSpNln0CuRP0tya5KLkrx8qD2SJM0aA4VIVf068DvAccDtST6d5G1D7Zkk6QVv4GciVbUd+D3gEuCfA1cm+VaSfzmszkmSXtgGfSby2iRXAPcBbwV+q6r+aVu+Yoj9kyS9gM0fsN2fAB8HPlhV/zBRrKrvJfm9ofRMkvSCN+jtrLcDn54IkCQvSvJSgKr65FQ7JpmX5I4kf9nWj09yS5LxJJ9Nclir/1xbH2/bl/Yd4wOt/u0kp/fVV7TaeJI1BzJwSdL0DRoiXwEO71t/aasN4r30boNN+EPgiqr6FeAx4IJWvwB4rNWvaO1IcgJwDvCrwAp6M8XmJZkHfAw4AzgBOLe1lSTNkEFvZ72kqp6cWKmqJyeuRKaSZAm9q5jLgfcnCb3nKP+2NdkAfAi4CljZlgE+D/xpa78SuKaqngK+m2QcOLm1G6+q+9u5rmlt7x1wTHoBW7rm+pGd+4G1bx/ZuaXZZtArkR8mOXFiJclJwD9M0X7CHwP/BfhJW/8F4PGqeqat7wAWt+XFwEMAbfsTrf1P6/vss7+6JGmGDHol8j7gc0m+BwT4J8C/mWqHJP8C2FVVtyd5y7R6OU1JVgOrAV7xileMsiuSNKcMFCJVdVuSVwOvaqVvV9X/e57dfg14R5IzgZcARwIfBY5KMr9dbSwBdrb2O+l9mHFHkvnAy4Hv99Un9O+zv/q+/V8HrAMYGxur5+m3JGlAB/IFjG8EXgucSO8h9nlTNa6qD1TVkqpaSu/B+Fer6neAm4B3tmargOva8qa2Ttv+1aqqVj+nzd46HlgG3ArcBixrs70Oa+fYdADjkSRN00BXIkk+CfwycCfw41YuYGOHc14CXJPkD4A7gKtb/Wrgk+3B+R56oUBV3ZPkWnoPzJ8BLq6qH7d+vQfYAswD1lfVPR36I0nqaNBnImPACe3K4IBV1deAr7Xl+3l2dlV/m38E/vV+9r+c3gyvfeubgc1d+iRJmr5Bb2d9k97DdEmSfmrQK5FjgHuT3Ao8NVGsqncMpVeSpFlh0BD50DA7IUmanQad4vs3SX4JWFZVX2mfVp833K5Jkl7oBv0q+AvpfRXJn7fSYuDLw+qUJGl2GPTB+sX0Pjy4F376A1W/OKxOSZJmh0FD5KmqenpipX2i3E9+S9IhbtAQ+ZskHwQOb7+t/jngfw+vW5Kk2WDQEFkD7AbuBt5N7wN+/qKhJB3iBp2d9RPgL9pLkiRg8O/O+i6TPAOpqlce9B5JkmaNA/nurAkvofcdV0cf/O5IkmaTgZ6JVNX3+147q+qP6f3srSTpEDbo7awT+1ZfRO/KZNCrGEnSHDVoEPxR3/IzwAPA2Qe9N5KkWWXQ2Vm/OeyOSJJmn0FvZ71/qu1V9ZGD0x1J0mxyILOz3sizv2H+W/R+53z7MDoljdLSNdeP5LwPrHWuimafQUNkCXBiVf0AIMmHgOur6l3D6pgk6YVv0K89WQQ83bf+dKtJkg5hg16JbARuTfKltn4WsGE4XZIkzRaDzs66PMkNwK+30vlVdcfwuiVJmg0GvZ0F8FJgb1V9FNiR5PipGid5SZJbk3wjyT1Jfr/Vj09yS5LxJJ9Nclir/1xbH2/bl/Yd6wOt/u0kp/fVV7TaeJI1BzAWSdJBMOjP414KXAJ8oJVeDPyv59ntKeCtVfU64PXAiiSnAn8IXFFVvwI8BlzQ2l8APNbqV7R2JDkBOAf4VWAF8GdJ5iWZB3wMOAM4ATi3tZUkzZBBr0R+G3gH8EOAqvoe8LKpdqieJ9vqi9urgLfS+7126D1XOastr+TZ5yyfB05Lkla/pqqeqqrvAuPAye01XlX3t19dvKa1lSTNkEFD5OmqKtrXwSc5YpCd2hXDncAuYCvwHeDxqnqmNdkBLG7Li4GHANr2J4Bf6K/vs8/+6pKkGTJoiFyb5M+Bo5JcCHyFAX6gqqp+XFWvp/c5k5OBV3fu6TQkWZ1kW5Jtu3fvHkUXJGlOet7ZWe2W0mfpBcBe4FXAf62qrYOepKoeT3IT8CZ6QTS/XW0sAXa2ZjuB4+g9tJ8PvBz4fl99Qv8++6vve/51wDqAsbGx5/y4liSpm+e9Emm3sTZX1daq+s9V9Z8GCZAkC5Mc1ZYPB94G3AfcBLyzNVsFXNeWN7V12vavtnNvAs5ps7eOB5bR+8qV24BlbbbXYfQevk98LYskaQYM+mHDryd5Y1XddgDHPhbY0GZRvQi4tqr+Msm9wDVJ/gC4A7i6tb8a+GSScWAPvVCgqu5Jci1wL72vob+4qn4MkOQ9wBZgHrC+qu45gP5JkqZp0BA5BXhXkgfozdAKvYuU1+5vh6q6C3jDJPX76T0f2bf+j/R+dneyY10OXD5JfTOwebAhSJIOtilDJMkrqur/AqdP1U6SdGh6viuRL9P79t4Hk3yhqv7VTHRKkjQ7PN+D9fQtv3KYHZEkzT7PFyK1n2VJkp73dtbrkuyld0VyeFuGZx+sHznU3kmSXtCmDJGqmjdTHZEkzT4H8lXwkiT9DENEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktTZoD9KpRFauub6UXdBkibllYgkqTNDRJLUmSEiSerMEJEkdWaISJI6G1qIJDkuyU1J7k1yT5L3tvrRSbYm2d7+Lmj1JLkyyXiSu5Kc2HesVa399iSr+uonJbm77XNlkjy3J5KkYRnmlcgzwH+sqhOAU4GLk5wArAFurKplwI1tHeAMYFl7rQaugl7oAJcCpwAnA5dOBE9rc2HffiuGOB5J0j6GFiJV9XBVfb0t/wC4D1gMrAQ2tGYbgLPa8kpgY/XcDByV5FjgdGBrVe2pqseArcCKtu3Iqrq5qgrY2HcsSdIMmJFnIkmWAm8AbgEWVdXDbdMjwKK2vBh4qG+3Ha02VX3HJPXJzr86ybYk23bv3j2tsUiSnjX0EEny88AXgPdV1d7+be0Koobdh6paV1VjVTW2cOHCYZ9Okg4ZQw2RJC+mFyCfqqovtvKj7VYU7e+uVt8JHNe3+5JWm6q+ZJK6JGmGDHN2VoCrgfuq6iN9mzYBEzOsVgHX9dXPa7O0TgWeaLe9tgDLkyxoD9SXA1vatr1JTm3nOq/vWJKkGTDML2D8NeDfAXcnubPVPgisBa5NcgHwIHB227YZOBMYB34EnA9QVXuSfBi4rbW7rKr2tOWLgE8AhwM3tJckaYYMLUSq6u+A/X1u47RJ2hdw8X6OtR5YP0l9G/CaaXRTkjQNfmJdktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnQ0tRJKsT7IryTf7akcn2Zpke/u7oNWT5Mok40nuSnJi3z6rWvvtSVb11U9Kcnfb58okGdZYJEmTmz/EY38C+FNgY19tDXBjVa1NsqatXwKcASxrr1OAq4BTkhwNXAqMAQXcnmRTVT3W2lwI3AJsBlYANwxxPNJQLV1z/UjO+8Dat4/kvJobhnYlUlV/C+zZp7wS2NCWNwBn9dU3Vs/NwFFJjgVOB7ZW1Z4WHFuBFW3bkVV1c1UVvaA6C0nSjJrpZyKLqurhtvwIsKgtLwYe6mu3o9Wmqu+YpC5JmkEje7DeriBqJs6VZHWSbUm27d69eyZOKUmHhJkOkUfbrSja312tvhM4rq/dklabqr5kkvqkqmpdVY1V1djChQunPQhJUs9Mh8gmYGKG1Srgur76eW2W1qnAE+221xZgeZIFbSbXcmBL27Y3yaltVtZ5fceSJM2Qoc3OSvIZ4C3AMUl20JtltRa4NskFwIPA2a35ZuBMYBz4EXA+QFXtSfJh4LbW7rKqmnhYfxG9GWCH05uV5cwsSZphQwuRqjp3P5tOm6RtARfv5zjrgfWT1LcBr5lOHyVJ0+Mn1iVJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSps/mj7oCk0Vq65vqRnfuBtW8f2bl1cHglIknqbNZfiSRZAXwUmAd8vKrWDutco/w/NmkuGtW/Ka+ADp5ZfSWSZB7wMeAM4ATg3CQnjLZXknTomNUhApwMjFfV/VX1NHANsHLEfZKkQ8Zsv521GHiob30HcMqI+iJplnAywcEz20NkIElWA6vb6pNJvj3K/kziGODvR92JIZvrY3R8s9+MjDF/OOwz7Nd0xvdL+9sw20NkJ3Bc3/qSVvsZVbUOWDdTnTpQSbZV1dio+zFMc32Mjm/2m+tjHNb4ZvszkduAZUmOT3IYcA6wacR9kqRDxqy+EqmqZ5K8B9hCb4rv+qq6Z8TdkqRDxqwOEYCq2gxsHnU/pukFe6vtIJrrY3R8s99cH+NQxpeqGsZxJUmHgNn+TESSNEKGyIgleSDJ3UnuTLJt1P05GJKsT7IryTf7akcn2Zpke/u7YJR9nI79jO9DSXa29/HOJGeOso/TkeS4JDcluTfJPUne2+pz4j2cYnxz6T18SZJbk3yjjfH3W/34JLckGU/y2TYhaXrn8nbWaCV5ABirqjkzBz/JbwBPAhur6jWt9t+APVW1NskaYEFVXTLKfna1n/F9CHiyqv77KPt2MCQ5Fji2qr6e5GXA7cBZwL9nDryHU4zvbObOexjgiKp6MsmLgb8D3gu8H/hiVV2T5H8A36iqq6ZzLq9EdNBV1d8Ce/YprwQ2tOUN9P7Rzkr7Gd+cUVUPV9XX2/IPgPvofTvEnHgPpxjfnFE9T7bVF7dXAW8FPt/qB+U9NERGr4C/TnJ7+2T9XLWoqh5uy48Ai0bZmSF5T5K72u2uWXmrZ19JlgJvAG5hDr6H+4wP5tB7mGRekjuBXcBW4DvA41X1TGuyg4MQnobI6L25qk6k903EF7dbJXNa9e6hzrX7qFcBvwy8HngY+KPRdmf6kvw88AXgfVW1t3/bXHgPJxnfnHoPq+rHVfV6et/kcTLw6mGcxxAZsara2f7uAr5E782eix5t96In7knvGnF/DqqqerT9o/0J8BfM8vex3Uf/AvCpqvpiK8+Z93Cy8c2193BCVT0O3AS8CTgqycTnAyf9mqgDZYiMUJIj2oM9khwBLAe+OfVes9YmYFVbXgVcN8K+HHQT/3FtfptZ/D62h7JXA/dV1Uf6Ns2J93B/45tj7+HCJEe15cOBt9F79nMT8M7W7KC8h87OGqEkr6R39QG9bw/4dFVdPsIuHRRJPgO8hd63hj4KXAp8GbgWeAXwIHB2Vc3Kh9P7Gd9b6N0GKeAB4N19zw9mlSRvBv4PcDfwk1b+IL3nBrP+PZxifOcyd97D19J7cD6P3sXCtVV1WftvzjXA0cAdwLuq6qlpncsQkSR15e0sSVJnhogkqTNDRJLUmSEiSerMEJEkdWaISJI6M0QkSZ0ZIpKkzv4/2LyLCkd/AwYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k-cjhPuebNeF"
      },
      "source": [
        "### Get lists of sentences"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7olPj8Exb1ly",
        "outputId": "add884d7-0486-40ba-8fa9-ca590975207b"
      },
      "source": [
        "# Convert abstract text lines into lists \n",
        "train_sentences = train_df[\"text\"].tolist()\n",
        "val_sentences = val_df[\"text\"].tolist()\n",
        "test_sentences = test_df[\"text\"].tolist()\n",
        "len(train_sentences), len(val_sentences), len(test_sentences)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(180040, 30212, 30135)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DSU4YdF0b4U_",
        "outputId": "f5354887-0459-4320-b61b-ca06d68f3e46"
      },
      "source": [
        "# View first 10 lines of training sentences\n",
        "train_sentences[:10]"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['to investigate the efficacy of @ weeks of daily low-dose oral prednisolone in improving pain , mobility , and systemic low-grade inflammation in the short term and whether the effect would be sustained at @ weeks in older adults with moderate to severe knee osteoarthritis ( oa ) .',\n",
              " 'a total of @ patients with primary knee oa were randomized @:@ ; @ received @ mg/day of prednisolone and @ received placebo for @ weeks .',\n",
              " 'outcome measures included pain reduction and improvement in function scores and systemic inflammation markers .',\n",
              " 'pain was assessed using the visual analog pain scale ( @-@ mm ) .',\n",
              " 'secondary outcome measures included the western ontario and mcmaster universities osteoarthritis index scores , patient global assessment ( pga ) of the severity of knee oa , and @-min walk distance ( @mwd ) .',\n",
              " 'serum levels of interleukin @ ( il-@ ) , il-@ , tumor necrosis factor ( tnf ) - , and high-sensitivity c-reactive protein ( hscrp ) were measured .',\n",
              " 'there was a clinically relevant reduction in the intervention group compared to the placebo group for knee pain , physical function , pga , and @mwd at @ weeks .',\n",
              " 'the mean difference between treatment arms ( @ % ci ) was @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; @ ( @-@ @ ) , p < @ ; and @ ( @-@ @ ) , p < @ , respectively .',\n",
              " 'further , there was a clinically relevant reduction in the serum levels of il-@ , il-@ , tnf - , and hscrp at @ weeks in the intervention group when compared to the placebo group .',\n",
              " 'these differences remained significant at @ weeks .']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0WmxmMebb7AV"
      },
      "source": [
        "## Make numeric labels \n",
        "\n",
        "We're going to create one hot and label encoded labels.\n",
        "\n",
        "We could get away with just making label encoded labels, however, TensorFlow's CategoricalCrossentropy loss function likes to have one hot encoded labels (this will enable us to use label smoothing later on).\n",
        "\n",
        "To numerically encode labels we'll use Scikit-Learn's OneHotEncoder and LabelEncoder classes.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7hqVk3rAcgbJ",
        "outputId": "2f6790f7-377c-4d85-b553-4b8573e051fe"
      },
      "source": [
        "# One hot encode labels\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "one_hot_encoder = OneHotEncoder(sparse=False)\n",
        "train_labels_one_hot = one_hot_encoder.fit_transform(train_df[\"target\"].to_numpy().reshape(-1, 1))\n",
        "val_labels_one_hot = one_hot_encoder.transform(val_df[\"target\"].to_numpy().reshape(-1, 1))\n",
        "test_labels_one_hot = one_hot_encoder.transform(test_df[\"target\"].to_numpy().reshape(-1, 1))\n",
        "\n",
        "# Check what training labels look like\n",
        "train_labels_one_hot"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0., 0., 0., 1., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       [0., 0., 1., 0., 0.],\n",
              "       ...,\n",
              "       [0., 0., 0., 0., 1.],\n",
              "       [0., 1., 0., 0., 0.],\n",
              "       [0., 1., 0., 0., 0.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FqZ4a-UCckXH"
      },
      "source": [
        "### Label encode labels"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3_3fhlWVcnVX",
        "outputId": "cb50aff6-ff98-4bdf-b705-a7301c6cff33"
      },
      "source": [
        "# Extract labels (\"target\" columns) and encode them into integers \n",
        "from sklearn.preprocessing import LabelEncoder\n",
        "label_encoder = LabelEncoder()\n",
        "train_labels_encoded = label_encoder.fit_transform(train_df[\"target\"].to_numpy())\n",
        "val_labels_encoded = label_encoder.transform(val_df[\"target\"].to_numpy())\n",
        "test_labels_encoded = label_encoder.transform(test_df[\"target\"].to_numpy())\n",
        "\n",
        "# Check what training labels look like\n",
        "train_labels_encoded"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([3, 2, 2, ..., 4, 1, 1])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gPHnRKKScpwV",
        "outputId": "834674f2-17d8-4cef-8f4c-b016f9a01e81"
      },
      "source": [
        "# Get class names and number of classes from LabelEncoder instance \n",
        "num_classes = len(label_encoder.classes_)\n",
        "class_names = label_encoder.classes_\n",
        "num_classes, class_names"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(5, array(['BACKGROUND', 'CONCLUSIONS', 'METHODS', 'OBJECTIVE', 'RESULTS'],\n",
              "       dtype=object))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6XK9k-DHcwpE"
      },
      "source": [
        "# Creating a series of model experiments"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F4v7aRdvFInv"
      },
      "source": [
        "## Model 0 : Getting a baseline\n",
        "\n",
        "Our first model we'll be a TF-IDF Multinomial Naive Bayes as recommended by Scikit-Learn's machine learning map.\n",
        "\n",
        "To build it, we'll create a Scikit-Learn Pipeline which uses the TfidfVectorizer class to convert our abstract sentences to numbers using the TF-IDF (term frequency-inverse document frequecy) algorithm and then learns to classify our sentences using the MultinomialNB aglorithm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F2a4ozzxFPKc"
      },
      "source": [
        "from sklearn.feature_extraction.text import TfidfVectorizer\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline\n",
        "\n",
        "# Create a pipeline\n",
        "model_0 = Pipeline([\n",
        "    (\"tf-idf\", TfidfVectorizer()),\n",
        "    (\"clf\", MultinomialNB())\n",
        "])\n",
        "\n",
        "# Fit the pipeline to the training data\n",
        "model_0.fit(X=train_sentences,\n",
        "            y = train_labels_encoded);"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zgkr8gYaF5Pv",
        "outputId": "0e95e792-94db-4731-95a5-cb4fe5e3735f"
      },
      "source": [
        "# Evaluate baseline on validation dataset\n",
        "model_0.score(X = val_sentences,\n",
        "              y = val_labels_encoded)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7218323844829869"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3rHdfS3bGENE",
        "outputId": "cb7af861-649d-45c8-bb5c-60c6320ee9ee"
      },
      "source": [
        "# Make predictions\n",
        "baseline_preds = model_0.predict(val_sentences)\n",
        "baseline_preds"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4, 1, 3, ..., 4, 4, 1])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FxVpyW4xGNZA"
      },
      "source": [
        "### Download helper functions script"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7dO5dSbPGU-K",
        "outputId": "d7eb1354-6bc7-45b4-d449-a2ca0acc0643"
      },
      "source": [
        "# Download helper functions script\n",
        "!wget https://raw.githubusercontent.com/prachuryanath/TF-Learning/main/extras/helper_functions.py"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-27 16:50:08--  https://raw.githubusercontent.com/prachuryanath/TF-Learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10235 (10.0K) [text/plain]\n",
            "Saving to: ‘helper_functions.py’\n",
            "\n",
            "\rhelper_functions.py   0%[                    ]       0  --.-KB/s               \rhelper_functions.py 100%[===================>]  10.00K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-06-27 16:50:08 (64.7 MB/s) - ‘helper_functions.py’ saved [10235/10235]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UeJolLFJGaFG",
        "outputId": "05fa39d9-72d5-4fcc-df09-4028d65aa78e"
      },
      "source": [
        "# Import calculate_results helper function\n",
        "from helper_functions import calculate_results\n",
        "\n",
        "# Calculate baseline results\n",
        "baseline_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                     y_pred=baseline_preds)\n",
        "baseline_results"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 72.1832384482987,\n",
              " 'f1': 0.6989250353450294,\n",
              " 'precision': 0.7186466952323352,\n",
              " 'recall': 0.7218323844829869}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LWE3W4EYGfEy"
      },
      "source": [
        "## Preparing our data for deep sequence models\n",
        "\n",
        "Excellent! We've got a working baseline to try and improve upon.\n",
        "\n",
        "But before we start building deeper models, we've got to create vectorization and embedding layers.\n",
        "\n",
        "The vectorization layer will convert our text to numbers and the embedding layer will capture the relationships between those numbers.\n",
        "\n",
        "To start creating our vectorization and embedding layers, we'll need to import the appropriate libraries (namely TensorFlow and NumPy)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0SUIHdX1GrOm"
      },
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers"
      ],
      "execution_count": 24,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Tx7tIeidIbX6",
        "outputId": "008dd622-dfff-43d3-b48b-4e686106e6f6"
      },
      "source": [
        "# How long is each sentence on average ?\n",
        "sent_lens = [len(sentence.split()) for sentence in train_sentences]\n",
        "avg_sent_lens = np.mean(sent_lens)\n",
        "avg_sent_lens"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "26.338269273494777"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "IEbJrjoIIrfy",
        "outputId": "35c522cb-650a-4209-a026-dd068e48617c"
      },
      "source": [
        "# What's the distribution look like ?\n",
        "import matplotlib.pyplot as plt\n",
        "plt.hist(sent_lens, bins=15)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([7.3163e+04, 8.1733e+04, 1.8772e+04, 4.5680e+03, 1.2400e+03,\n",
              "        3.5400e+02, 1.1600e+02, 4.5000e+01, 1.7000e+01, 1.2000e+01,\n",
              "        1.2000e+01, 3.0000e+00, 3.0000e+00, 1.0000e+00, 1.0000e+00]),\n",
              " array([  1.        ,  20.66666667,  40.33333333,  60.        ,\n",
              "         79.66666667,  99.33333333, 119.        , 138.66666667,\n",
              "        158.33333333, 178.        , 197.66666667, 217.33333333,\n",
              "        237.        , 256.66666667, 276.33333333, 296.        ]),\n",
              " <a list of 15 Patch objects>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD4CAYAAAAO9oqkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWt0lEQVR4nO3df6zd9X3f8eerNr9KAjZwZzHbmp3FauSghcAVOEoUbXgxNplqKhEEqoaFLDwN2JJp02ZWaW4hTDBtZWEiVG7wsKMshtJEWI2p6xmqan8YuAQCGEp9w49iy+BbbKApCtT0vT/Ox8np5f44Ntf3+trPh3R0Pt/39/P9ns8nX4fX/X7P95yTqkKSdHL7lakegCRp6hkGkiTDQJJkGEiSMAwkScDMqR7A0TrvvPNqwYIFUz0MSZo2nnrqqb+qqr6R1k3bMFiwYAEDAwNTPQxJmjaSvDbaOi8TSZIMA0mSYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSSJafwJ5OPJgrU/mtD9vXrHVyd0f5I0np7ODJL8uyS7kjyf5PtJTk+yMMnjSQaTPJDk1Nb3tLY82NYv6NrPLa3+UpLLu+rLW20wydqJnqQkaWzjhkGSucC/Bfqr6gJgBnANcCdwV1V9GjgIrG6brAYOtvpdrR9JFrftPgssB76dZEaSGcA9wApgMXBt6ytJmiS9vmcwEzgjyUzgV4F9wGXAQ239RuDK1l7ZlmnrlyZJq2+uqver6hVgELikPQar6uWq+gDY3PpKkibJuGFQVXuB/w78JZ0QeAd4Cni7qg61bnuAua09F3i9bXuo9T+3uz5sm9HqH5FkTZKBJANDQ0O9zE+S1INeLhPNpvOX+kLgHwJn0rnMM+mqan1V9VdVf1/fiF/JLUk6Cr1cJvrnwCtVNVRVfwv8APgiMKtdNgKYB+xt7b3AfIC2/mzgre76sG1Gq0uSJkkvYfCXwJIkv9qu/S8FXgAeA65qfVYBD7f2lrZMW/9oVVWrX9PuNloILAKeAJ4EFrW7k06l8ybzlo8/NUlSr8b9nEFVPZ7kIeDHwCHgaWA98CNgc5Jvttp9bZP7gO8mGQQO0PmPO1W1K8mDdILkEHBTVX0IkORmYBudO5U2VNWuiZuiJGk8PX3orKrWAeuGlV+mcyfQ8L4/B742yn5uB24fob4V2NrLWCRJE8+vo5AkGQaSJMNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJJEj99NdKKZ6B+wl6TpzjMDSZJhIEkyDCRJGAaSJHoIgyS/luSZrse7Sb6R5Jwk25Psbs+zW/8kuTvJYJJnk1zUta9Vrf/uJKu66hcnea5tc3f7eU1J0iQZNwyq6qWqurCqLgQuBt4DfgisBXZU1SJgR1sGWEHn940XAWuAewGSnEPn19IupfMLaesOB0jrc0PXdssnZHaSpJ4c6WWipcBPq+o1YCWwsdU3Ale29kpgU3XsBGYlOR+4HNheVQeq6iCwHVje1p1VVTurqoBNXfuSJE2CIw2Da4Dvt/acqtrX2m8Ac1p7LvB61zZ7Wm2s+p4R6h+RZE2SgSQDQ0NDRzh0SdJoeg6DJKcCvw78wfB17S/6msBxjaiq1ldVf1X19/X1HeuXk6STxpGcGawAflxVb7blN9slHtrz/lbfC8zv2m5eq41VnzdCXZI0SY4kDK7ll5eIALYAh+8IWgU83FW/rt1VtAR4p11O2gYsSzK7vXG8DNjW1r2bZEm7i+i6rn1JkiZBT99NlORM4CvAv+oq3wE8mGQ18BpwdatvBa4ABunceXQ9QFUdSHIb8GTrd2tVHWjtG4H7gTOAR9pDkjRJegqDqvob4Nxhtbfo3F00vG8BN42ynw3AhhHqA8AFvYxFkjTx/ASyJMkwkCQZBpIkDANJEoaBJAnDQJKEYSBJwjCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiS6DEMksxK8lCSP0/yYpIvJDknyfYku9vz7NY3Se5OMpjk2SQXde1nVeu/O8mqrvrFSZ5r29zdfgtZkjRJej0z+Bbwx1X1GeBzwIvAWmBHVS0CdrRlgBXAovZYA9wLkOQcYB1wKXAJsO5wgLQ+N3Rtt/zjTUuSdCTGDYMkZwNfBu4DqKoPquptYCWwsXXbCFzZ2iuBTdWxE5iV5HzgcmB7VR2oqoPAdmB5W3dWVe1sv5+8qWtfkqRJ0MuZwUJgCPjfSZ5O8p0kZwJzqmpf6/MGMKe15wKvd22/p9XGqu8Zof4RSdYkGUgyMDQ01MPQJUm96CUMZgIXAfdW1eeBv+GXl4QAaH/R18QP7++rqvVV1V9V/X19fcf65STppNFLGOwB9lTV4235ITrh8Ga7xEN73t/W7wXmd20/r9XGqs8boS5JmiTjhkFVvQG8nuTXWmkp8AKwBTh8R9Aq4OHW3gJc1+4qWgK80y4nbQOWJZnd3jheBmxr695NsqTdRXRd174kSZNgZo/9/g3wvSSnAi8D19MJkgeTrAZeA65ufbcCVwCDwHutL1V1IMltwJOt361VdaC1bwTuB84AHmkPSdIk6SkMquoZoH+EVUtH6FvATaPsZwOwYYT6AHBBL2ORJE08P4EsSTIMJEmGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEn0GAZJXk3yXJJnkgy02jlJtifZ3Z5nt3qS3J1kMMmzSS7q2s+q1n93klVd9Yvb/gfbtpnoiUqSRnckZwb/rKourKrDv3i2FthRVYuAHW0ZYAWwqD3WAPdCJzyAdcClwCXAusMB0vrc0LXd8qOekSTpiH2cy0QrgY2tvRG4squ+qTp2ArOSnA9cDmyvqgNVdRDYDixv686qqp3tJzM3de1LkjQJeg2DAv4kyVNJ1rTanKra19pvAHNaey7wete2e1ptrPqeEeofkWRNkoEkA0NDQz0OXZI0npk99vtSVe1N8g+A7Un+vHtlVVWSmvjh/X1VtR5YD9Df33/MX0+SThY9nRlU1d72vB/4IZ1r/m+2Szy05/2t+15gftfm81ptrPq8EeqSpEkybhgkOTPJJw+3gWXA88AW4PAdQauAh1t7C3Bdu6toCfBOu5y0DViWZHZ743gZsK2tezfJknYX0XVd+5IkTYJeLhPNAX7Y7vacCfyfqvrjJE8CDyZZDbwGXN36bwWuAAaB94DrAarqQJLbgCdbv1ur6kBr3wjcD5wBPNIekqRJMm4YVNXLwOdGqL8FLB2hXsBNo+xrA7BhhPoAcEEP45UkHQN+AlmSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjiCMEgyI8nTSf6oLS9M8niSwSQPJDm11U9ry4Nt/YKufdzS6i8lubyrvrzVBpOsnbjpSZJ6cSRnBl8HXuxavhO4q6o+DRwEVrf6auBgq9/V+pFkMXAN8FlgOfDtFjAzgHuAFcBi4NrWV5I0SXoKgyTzgK8C32nLAS4DHmpdNgJXtvbKtkxbv7T1Xwlsrqr3q+oVOr+RfEl7DFbVy1X1AbC59ZUkTZJezwz+J/Afgb9ry+cCb1fVoba8B5jb2nOB1wHa+nda/1/Uh20zWv0jkqxJMpBkYGhoqMehS5LGM24YJPkXwP6qemoSxjOmqlpfVf1V1d/X1zfVw5GkE8bMHvp8Efj1JFcApwNnAd8CZiWZ2f76nwfsbf33AvOBPUlmAmcDb3XVD+veZrS6JGkSjHtmUFW3VNW8qlpA5w3gR6vqN4HHgKtat1XAw629pS3T1j9aVdXq17S7jRYCi4AngCeBRe3upFPba2yZkNlJknrSy5nBaP4TsDnJN4Gngfta/T7gu0kGgQN0/uNOVe1K8iDwAnAIuKmqPgRIcjOwDZgBbKiqXR9jXJKkI3REYVBVfwr8aWu/TOdOoOF9fg58bZTtbwduH6G+Fdh6JGORJE0cP4EsSTIMJEmGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkegiDJKcneSLJT5LsSvI7rb4wyeNJBpM80H6/mPYbxw+0+uNJFnTt65ZWfynJ5V315a02mGTtxE9TkjSWXs4M3gcuq6rPARcCy5MsAe4E7qqqTwMHgdWt/2rgYKvf1fqRZDGd30P+LLAc+HaSGUlmAPcAK4DFwLWtryRpkowbBtXxs7Z4SnsUcBnwUKtvBK5s7ZVtmbZ+aZK0+uaqer+qXgEG6fyG8iXAYFW9XFUfAJtbX0nSJOnpPYP2F/wzwH5gO/BT4O2qOtS67AHmtvZc4HWAtv4d4Nzu+rBtRquPNI41SQaSDAwNDfUydElSD3oKg6r6sKouBObR+Uv+M8d0VKOPY31V9VdVf19f31QMQZJOSEd0N1FVvQ08BnwBmJVkZls1D9jb2nuB+QBt/dnAW931YduMVpckTZJe7ibqSzKrtc8AvgK8SCcUrmrdVgEPt/aWtkxb/2hVVatf0+42WggsAp4AngQWtbuTTqXzJvOWiZicJKk3M8fvwvnAxnbXz68AD1bVHyV5Adic5JvA08B9rf99wHeTDAIH6PzHnaraleRB4AXgEHBTVX0IkORmYBswA9hQVbsmbIaSpHGNGwZV9Szw+RHqL9N5/2B4/efA10bZ1+3A7SPUtwJbexivJOkY8BPIkiTDQJJkGEiSMAwkSRgGkiQMA0kShoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkYBpIkDANJEoaBJAnDQJJEbz97OT/JY0leSLIryddb/Zwk25Psbs+zWz1J7k4ymOTZJBd17WtV6787yaqu+sVJnmvb3J0kx2KykqSR9XJmcAj491W1GFgC3JRkMbAW2FFVi4AdbRlgBZ3fN14ErAHuhU54AOuAS+n8Qtq6wwHS+tzQtd3yjz81SVKvxg2DqtpXVT9u7b8GXgTmAiuBja3bRuDK1l4JbKqOncCsJOcDlwPbq+pAVR0EtgPL27qzqmpnVRWwqWtfkqRJcETvGSRZQOf3kB8H5lTVvrbqDWBOa88FXu/abE+rjVXfM0J9pNdfk2QgycDQ0NCRDF2SNIaewyDJJ4A/BL5RVe92r2t/0dcEj+0jqmp9VfVXVX9fX9+xfjlJOmn0FAZJTqETBN+rqh+08pvtEg/teX+r7wXmd20+r9XGqs8boS5JmiQzx+vQ7uy5D3ixqn63a9UWYBVwR3t+uKt+c5LNdN4sfqeq9iXZBvzXrjeNlwG3VNWBJO8mWULn8tN1wP+agLlNWwvW/mjC9/nqHV+d8H1KOnGMGwbAF4F/CTyX5JlW+890QuDBJKuB14Cr27qtwBXAIPAecD1A+4/+bcCTrd+tVXWgtW8E7gfOAB5pD0nSJBk3DKrq/wGj3fe/dIT+Bdw0yr42ABtGqA8AF4w3FknSseEnkCVJhoEkyTCQJGEYSJIwDCRJGAaSJAwDSRKGgSQJw0CShGEgScIwkCRhGEiSMAwkSRgGkiQMA0kShoEkiR7CIMmGJPuTPN9VOyfJ9iS72/PsVk+Su5MMJnk2yUVd26xq/XcnWdVVvzjJc22bu9vPbEqSJlEvZwb3A8uH1dYCO6pqEbCjLQOsABa1xxrgXuiEB7COzm8iXwKs6/ot5HuBG7q2G/5akqRjbNwwqKo/Aw4MK68ENrb2RuDKrvqm6tgJzEpyPnA5sL2qDlTVQWA7sLytO6uqdrafy9zUtS9J0iQ52vcM5lTVvtZ+A5jT2nOB17v67Wm1sep7RqiPKMmaJANJBoaGho5y6JKk4T72G8jtL/qagLH08lrrq6q/qvr7+vom4yUl6aRwtGHwZrvEQ3ve3+p7gfld/ea12lj1eSPUJUmT6GjDYAtw+I6gVcDDXfXr2l1FS4B32uWkbcCyJLPbG8fLgG1t3btJlrS7iK7r2pckaZLMHK9Dku8D/xQ4L8keOncF3QE8mGQ18Bpwdeu+FbgCGATeA64HqKoDSW4Dnmz9bq2qw29K30jnjqUzgEfaQ5I0icYNg6q6dpRVS0foW8BNo+xnA7BhhPoAcMF445AkHTt+AlmSZBhIkgwDSRKGgSQJw0CShGEgScIwkCRhGEiS6OFDZzoxLFj7ownd36t3fHVC9ydpanlmIEkyDCRJhoEkCcNAkoRhIEnCMJAkYRhIkjAMJEkcRx86S7Ic+BYwA/hOVd0xxUPSGPwQm3RiOS7ODJLMAO4BVgCLgWuTLJ7aUUnSyeN4OTO4BBisqpcBkmwGVgIvTOmoNGkm+kwDPNuQjsTxEgZzgde7lvcAlw7vlGQNsKYt/izJS0fxWucBf3UU2x2PTqS5wATPJ3dO1J6O2ol0fE6kucCJNZ8jmcs/Gm3F8RIGPamq9cD6j7OPJANV1T9BQ5pSJ9JcwPkcz06kucCJNZ+Jmstx8Z4BsBeY37U8r9UkSZPgeAmDJ4FFSRYmORW4BtgyxWOSpJPGcXGZqKoOJbkZ2Ebn1tINVbXrGL3cx7rMdJw5keYCzud4diLNBU6s+UzIXFJVE7EfSdI0drxcJpIkTSHDQJJ08oRBkuVJXkoymGTtVI/naCR5NclzSZ5JMtBq5yTZnmR3e5491eMcTZINSfYneb6rNuL403F3O17PJrlo6kb+UaPM5beT7G3H55kkV3Stu6XN5aUkl0/NqEeWZH6Sx5K8kGRXkq+3+nQ9NqPNZ7oen9OTPJHkJ20+v9PqC5M83sb9QLv5hiSnteXBtn5BTy9UVSf8g86b0j8FPgWcCvwEWDzV4zqKebwKnDes9t+Ata29Frhzqsc5xvi/DFwEPD/e+IErgEeAAEuAx6d6/D3M5beB/zBC38Xt39xpwML2b3HGVM+ha3znAxe19ieBv2hjnq7HZrT5TNfjE+ATrX0K8Hj73/1B4JpW/z3gX7f2jcDvtfY1wAO9vM7Jcmbwi6+7qKoPgMNfd3EiWAlsbO2NwJVTOJYxVdWfAQeGlUcb/0pgU3XsBGYlOX9yRjq+UeYympXA5qp6v6peAQbp/Js8LlTVvqr6cWv/NfAinW8FmK7HZrT5jOZ4Pz5VVT9ri6e0RwGXAQ+1+vDjc/i4PQQsTZLxXudkCYORvu5irH8cx6sC/iTJU+2rOQDmVNW+1n4DmDM1Qztqo41/uh6zm9ulkw1dl+ymzVzaJYXP0/nrc9ofm2HzgWl6fJLMSPIMsB/YTufs5e2qOtS6dI/5F/Np698Bzh3vNU6WMDhRfKmqLqLz7a43Jfly98rqnBdO23uFp/v4gXuBfwxcCOwD/sfUDufIJPkE8IfAN6rq3e510/HYjDCfaXt8qurDqrqQzrczXAJ8ZqJf42QJgxPi6y6qam973g/8kM4/ijcPn6K35/1TN8KjMtr4p90xq6o32/9p/w74fX55qeG4n0uSU+j8h/N7VfWDVp62x2ak+Uzn43NYVb0NPAZ8gc7lucMfHO4e8y/m09afDbw13r5PljCY9l93keTMJJ883AaWAc/Tmceq1m0V8PDUjPCojTb+LcB17c6VJcA7XZcsjkvDrpv/Bp3jA525XNPu8lgILAKemOzxjaZdT74PeLGqfrdr1bQ8NqPNZxofn74ks1r7DOArdN4HeQy4qnUbfnwOH7ergEfbmd3Ypvqd8sl60LkD4i/oXGv7rakez1GM/1N07nj4CbDr8BzoXAvcAewG/i9wzlSPdYw5fJ/O6fnf0rnGuXq08dO5g+KedryeA/qnevw9zOW7bazPtv9Dnt/V/7faXF4CVkz1+IfN5Ut0LgE9CzzTHldM42Mz2nym6/H5J8DTbdzPA/+l1T9FJ7QGgT8ATmv109vyYFv/qV5ex6+jkCSdNJeJJEljMAwkSYaBJMkwkCRhGEiSMAwkSRgGkiTg/wMqybg/6oo3YAAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "je8OphXJI_4T",
        "outputId": "ffab61e2-542b-4b08-b45e-55419163b9af"
      },
      "source": [
        "# How long of a sentence covers 95% of the lengths?\n",
        "output_seq_len = int(np.percentile(sent_lens, 95))\n",
        "output_seq_len"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "55"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xWAg6f0oJPeA"
      },
      "source": [
        "Wonderful! It looks like 95% of the sentences in our training set have a length of 55 tokens or less.\n",
        "\n",
        "When we create our tokenization layer, we'll use this value to turn all of our sentences into the same length. Meaning sentences with a length below 55 get padded with zeros and sentences with a length above 55 get truncated (words after 55 get cut off)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FE5cD-M-JYaM",
        "outputId": "2179d227-a69c-4b4d-b36f-7084fe473aaa"
      },
      "source": [
        "# Maximum sentence length in the training set\n",
        "max(sent_lens)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "296"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EnUNv660JaSG"
      },
      "source": [
        "### Create text vectorizer\n",
        "\n",
        "Now we've got a little more information about our texts, let's create a way to turn it into numbers.\n",
        "\n",
        "To do so, we'll use the TextVectorization layer from TensorFlow.\n",
        "\n",
        "We'll keep all the parameters default except for max_tokens (the number of unique words in our dataset) and output_sequence_length (our desired output length for each vectorized sentence).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VuMMxPTIJori"
      },
      "source": [
        "# How many words are in our vocabulary? (taken from 3.2 in https://arxiv.org/pdf/1710.06071.pdf)\n",
        "max_tokens = 68000"
      ],
      "execution_count": 29,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kX-T69OpJqts"
      },
      "source": [
        "# Create text vectorizer\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "text_vectorizer = TextVectorization(max_tokens=max_tokens, # number of words in vocabulary\n",
        "                                    output_sequence_length=55) # desired output length of vectorized sequences"
      ],
      "execution_count": 30,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jS_MU-0KJz9i"
      },
      "source": [
        "Great! Looks like our text_vectorizer is ready, let's adapt it to the training data (let it read the training data and figure out what number should represent what word) and then test it out."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XmIQcCE1JvO7"
      },
      "source": [
        "# Adapt text vectorizer to training sentences\n",
        "text_vectorizer.adapt(train_sentences)"
      ],
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vbymR2tRJyLZ",
        "outputId": "86960fc8-dba4-407f-df98-8a47f8262977"
      },
      "source": [
        "# Test out text vectorizer\n",
        "import random\n",
        "target_sentence = random.choice(train_sentences)\n",
        "print(f\"Text:\\n{target_sentence}\")\n",
        "print(f\"\\nLength of text: {len(target_sentence.split())}\")\n",
        "print(f\"\\nVectorized text:\\n{text_vectorizer([target_sentence])}\")"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Text:\n",
            "we compared the cost-utility from the healthcare and societal perspectives of cbt versus rpt ( combination of pregabalin + duloxetine ) and usual care ( tau ) groups in the treatment of fm .\n",
            "\n",
            "Length of text: 34\n",
            "\n",
            "Vectorized text:\n",
            "[[   43    34     2  6816    27     2  1055     3  3766  4694     4  1239\n",
            "    110 21872   269     4  2179  5293     3   370    77  2301    24     5\n",
            "      2    19     4  2974     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FAqJhJOJJ32h",
        "outputId": "1c6a8940-2766-4f1d-85bb-f05b522d54a9"
      },
      "source": [
        "# How many words in our training vocabulary?\n",
        "rct_20k_text_vocab = text_vectorizer.get_vocabulary()\n",
        "print(f\"Number of words in vocabulary: {len(rct_20k_text_vocab)}\"), \n",
        "print(f\"Most common words in the vocabulary: {rct_20k_text_vocab[:5]}\")\n",
        "print(f\"Least common words in the vocabulary: {rct_20k_text_vocab[-5:]}\")"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of words in vocabulary: 64841\n",
            "Most common words in the vocabulary: ['', '[UNK]', 'the', 'and', 'of']\n",
            "Least common words in the vocabulary: ['aainduced', 'aaigroup', 'aachener', 'aachen', 'aaacp']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ESWjEHYMKD5t",
        "outputId": "513a1398-87aa-4f24-a150-00cab8aba2fd"
      },
      "source": [
        "# Get the config of our text vectorizer\n",
        "text_vectorizer.get_config()"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'dtype': 'string',\n",
              " 'max_tokens': 68000,\n",
              " 'name': 'text_vectorization',\n",
              " 'ngrams': None,\n",
              " 'output_mode': 'int',\n",
              " 'output_sequence_length': 55,\n",
              " 'pad_to_max_tokens': False,\n",
              " 'split': 'whitespace',\n",
              " 'standardize': 'lower_and_strip_punctuation',\n",
              " 'trainable': True,\n",
              " 'vocabulary_size': 64841}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ikIQmf_fKKNI"
      },
      "source": [
        "### Create custom text embedding\n",
        "\n",
        "Our token_vectorization layer maps the words in our text directly to numbers. However, this doesn't necessarily capture the relationships between those numbers.\n",
        "\n",
        "To create a richer numerical representation of our text, we can use an **embedding**.\n",
        "\n",
        "As our model learns (*by going through many different examples of abstract sentences and their labels*), it'll update its embedding to better represent the relationships between tokens in our corpus.\n",
        "\n",
        "We can create a trainable embedding layer using TensorFlow's `Embedding layer`.\n",
        "\n",
        "Once again, the main parameters we're concerned with here are the inputs and outputs of our `Embedding layer`.\n",
        "\n",
        "The input_dim parameter defines the size of our vocabulary. And the output_dim parameter defines the dimension of the embedding output.\n",
        "\n",
        "Once created, our embedding layer will take the integer outputs of our `text_vectorization` layer as inputs and convert them to feature vectors of size `output_dim`.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hq32taDUMUHl",
        "outputId": "d3fef36c-c9c7-47da-dc5b-949252e14eb1"
      },
      "source": [
        "# Create token embedding layer\n",
        "token_embed = layers.Embedding(input_dim=len(rct_20k_text_vocab), # length of vocabulary\n",
        "                               output_dim=128, # Note: different embedding sizes result in drastically different numbers of parameters to train\n",
        "                               # Use masking to handle variable sequence lengths (save space)\n",
        "                               mask_zero=True,\n",
        "                               name=\"token_embedding\") \n",
        "\n",
        "# Show example embedding\n",
        "print(f\"Sentence before vectorization:\\n{target_sentence}\\n\")\n",
        "vectorized_sentence = text_vectorizer([target_sentence])\n",
        "print(f\"Sentence after vectorization (before embedding):\\n{vectorized_sentence}\\n\")\n",
        "embedded_sentence = token_embed(vectorized_sentence)\n",
        "print(f\"Sentence after embedding:\\n{embedded_sentence}\\n\")\n",
        "print(f\"Embedded sentence shape: {embedded_sentence.shape}\")"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sentence before vectorization:\n",
            "we compared the cost-utility from the healthcare and societal perspectives of cbt versus rpt ( combination of pregabalin + duloxetine ) and usual care ( tau ) groups in the treatment of fm .\n",
            "\n",
            "Sentence after vectorization (before embedding):\n",
            "[[   43    34     2  6816    27     2  1055     3  3766  4694     4  1239\n",
            "    110 21872   269     4  2179  5293     3   370    77  2301    24     5\n",
            "      2    19     4  2974     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0     0     0     0     0     0\n",
            "      0     0     0     0     0     0     0]]\n",
            "\n",
            "Sentence after embedding:\n",
            "[[[ 0.01223968 -0.03764788 -0.00448176 ...  0.01726845 -0.00430101\n",
            "    0.02710989]\n",
            "  [-0.02797055 -0.03720463  0.02124098 ...  0.02473954 -0.01824323\n",
            "   -0.02119844]\n",
            "  [-0.04372934  0.01093026  0.04511075 ... -0.01182992  0.04657802\n",
            "    0.03785614]\n",
            "  ...\n",
            "  [-0.04738771  0.01247416 -0.02610459 ... -0.03845491 -0.0449123\n",
            "    0.00945884]\n",
            "  [-0.04738771  0.01247416 -0.02610459 ... -0.03845491 -0.0449123\n",
            "    0.00945884]\n",
            "  [-0.04738771  0.01247416 -0.02610459 ... -0.03845491 -0.0449123\n",
            "    0.00945884]]]\n",
            "\n",
            "Embedded sentence shape: (1, 55, 128)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rBbbVDpKMwbH"
      },
      "source": [
        "### Create datasets \n",
        "Namely, the tf.data API provides methods which enable faster data loading.\n",
        "\n",
        "The main steps we'll want to use with our data is to turn it into a `PrefetchDataset` of batches.\n",
        "\n",
        "Doing so we'll ensure TensorFlow loads our data onto the GPU as fast as possible, in turn leading to faster training time.\n",
        "\n",
        "To create a batched PrefetchDataset we can use the methods `batch()` and `prefetch()`, the parameter `tf.data.AUTOTUNE` will also allow TensorFlow to determine the optimal amount of compute to use to prepare datasets.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uZXBLJ62NBZ2",
        "outputId": "6a5af520-2fdb-46c8-d0c1-75c10298354b"
      },
      "source": [
        "# Turn our data into TensorFlow Datasets\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((train_sentences, train_labels_one_hot))\n",
        "valid_dataset = tf.data.Dataset.from_tensor_slices((val_sentences, val_labels_one_hot))\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((test_sentences, test_labels_one_hot))\n",
        "\n",
        "train_dataset"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<TensorSliceDataset shapes: ((), (5,)), types: (tf.string, tf.float64)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_rrX-anYNPe7",
        "outputId": "c2c89d3a-810a-4f06-9a15-87af734fbe0e"
      },
      "source": [
        "# Take the TensorSliceDataset's and turn them into prefetched batches\n",
        "train_dataset = train_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "valid_dataset = valid_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "test_dataset = test_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "\n",
        "train_dataset"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<PrefetchDataset shapes: ((None,), (None, 5)), types: (tf.string, tf.float64)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_dn9x-rwNRgQ"
      },
      "source": [
        "## Model 1 : Conv1D with token embeddings\n",
        "\n",
        "All of our deep models will follow a similar structure:\n",
        "\n",
        "`Input (text) -> Tokenize -> Embedding -> Layers -> Output (label probability)`\n",
        "\n",
        "The main component we'll be changing throughout is the Layers component. Because any modern deep NLP model requires text to be converted into an embedding before meaningful patterns can be discovered within.\n",
        "\n",
        "The first model we're going to build is a 1-dimensional Convolutional Neural Network.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "StCVFrLwNg4j"
      },
      "source": [
        "# Create 1D convolutional model to process sequences\n",
        "inputs = layers.Input(shape=(1,), dtype=tf.string)\n",
        "text_vectors = text_vectorizer(inputs) # vectorize text inputs\n",
        "token_embeddings = token_embed(text_vectors) # create embedding\n",
        "x = layers.Conv1D(64, kernel_size=5, padding=\"same\", activation=\"relu\")(token_embeddings)\n",
        "x = layers.GlobalAveragePooling1D()(x) # condense the output of our feature vector\n",
        "outputs = layers.Dense(num_classes, activation=\"softmax\")(x)\n",
        "model_1 = tf.keras.Model(inputs, outputs)\n",
        "\n",
        "# Compile\n",
        "model_1.compile(loss=\"categorical_crossentropy\", # if your labels are integer form (not one hot) use sparse_categorical_crossentropy\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 38,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mPA6RZ65N5dk",
        "outputId": "a4c3a260-7394-4a59-fb60-d379d733e3d7"
      },
      "source": [
        "# Get summary of Conv1D model\n",
        "model_1.summary()"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_1 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "text_vectorization (TextVect (None, 55)                0         \n",
            "_________________________________________________________________\n",
            "token_embedding (Embedding)  (None, 55, 128)           8299648   \n",
            "_________________________________________________________________\n",
            "conv1d (Conv1D)              (None, 55, 64)            41024     \n",
            "_________________________________________________________________\n",
            "global_average_pooling1d (Gl (None, 64)                0         \n",
            "_________________________________________________________________\n",
            "dense (Dense)                (None, 5)                 325       \n",
            "=================================================================\n",
            "Total params: 8,340,997\n",
            "Trainable params: 8,340,997\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bz3ZVL8NN-E8"
      },
      "source": [
        "Since our training data contains nearly 200,000 sentences, fitting a deep model may take a while even with a GPU. So to keep our experiments swift, we're going to run them on a subset of the training dataset.\n",
        "\n",
        "More specifically, we'll only use the first 10% of batches (about 18,000 samples) of the training set to train on and the first 10% of batches from the validation set to validate on.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K7GdhSyXOQEG",
        "outputId": "c1ee618e-0519-48f3-9ad7-61a0864934a4"
      },
      "source": [
        "# Fit the model\n",
        "model_1_history = model_1.fit(train_dataset,\n",
        "                              steps_per_epoch=int(0.1 * len(train_dataset)), # only fit on 10% of batches for faster training time\n",
        "                              epochs=3,\n",
        "                              validation_data=valid_dataset,\n",
        "                              validation_steps=int(0.1 * len(valid_dataset))) # only validate on 10% of batches"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "562/562 [==============================] - 77s 81ms/step - loss: 0.9142 - accuracy: 0.6423 - val_loss: 0.6831 - val_accuracy: 0.7377\n",
            "Epoch 2/3\n",
            "562/562 [==============================] - 45s 81ms/step - loss: 0.6551 - accuracy: 0.7580 - val_loss: 0.6291 - val_accuracy: 0.7670\n",
            "Epoch 3/3\n",
            "562/562 [==============================] - 46s 81ms/step - loss: 0.6149 - accuracy: 0.7744 - val_loss: 0.5965 - val_accuracy: 0.7856\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TwM9iMbNOVkM",
        "outputId": "af17d4e0-af2c-443e-b08d-a27b86648cb6"
      },
      "source": [
        "# Evaluate on whole validation dataset (we only validated on 10% of batches during training)\n",
        "model_1.evaluate(valid_dataset)"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 3s 3ms/step - loss: 0.5974 - accuracy: 0.7855\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.5974297523498535, 0.7854825854301453]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1m_AXUlQP_Fa",
        "outputId": "a5b4755c-4abb-4d78-cbd9-db44a374380d"
      },
      "source": [
        "# Make predictions (our model outputs prediction probabilities for each class)\n",
        "model_1_pred_probs = model_1.predict(valid_dataset)\n",
        "model_1_pred_probs"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4.3972251e-01, 1.4293425e-01, 6.5332189e-02, 3.2375631e-01,\n",
              "        2.8254781e-02],\n",
              "       [4.4512346e-01, 2.9036015e-01, 1.4561634e-02, 2.3880926e-01,\n",
              "        1.1145523e-02],\n",
              "       [1.4519608e-01, 4.4064992e-03, 3.0453885e-03, 8.4731352e-01,\n",
              "        3.8443770e-05],\n",
              "       ...,\n",
              "       [6.3077068e-06, 5.6378311e-04, 7.4399973e-04, 3.6405795e-06,\n",
              "        9.9868220e-01],\n",
              "       [4.9201358e-02, 5.4950482e-01, 7.2523177e-02, 4.9564149e-02,\n",
              "        2.7920642e-01],\n",
              "       [1.6325441e-01, 6.8794167e-01, 4.3809447e-02, 5.4477666e-02,\n",
              "        5.0516829e-02]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "on46vMnoQCjy",
        "outputId": "a8d65371-d9e1-4b0d-e045-79aee34b1aaf"
      },
      "source": [
        "# Convert pred probs to classes\n",
        "model_1_preds = tf.argmax(model_1_pred_probs, axis=1)\n",
        "model_1_preds"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(30212,), dtype=int64, numpy=array([0, 0, 3, ..., 4, 1, 1])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "joubzlk9TfHY",
        "outputId": "6d95727d-084f-42d5-e598-16c62db2f84f"
      },
      "source": [
        "# Calculate model_1 results\n",
        "model_1_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                    y_pred=model_1_preds)\n",
        "model_1_results"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 78.54825896994572,\n",
              " 'f1': 0.7826409682036137,\n",
              " 'precision': 0.7825727953838002,\n",
              " 'recall': 0.7854825896994572}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioEtyCWVThue"
      },
      "source": [
        "## Model 2 : Feature extraction with pretrained token embeddings\n",
        "\n",
        "The model structure will look like:\n",
        "\n",
        "`Inputs (string) -> Pretrained embeddings from TensorFlow Hub (Universal Sentence Encoder) -> Layers -> Output (prediction probabilities)`\n",
        "\n",
        "You'll notice the lack of tokenization layer we've used in a previous model. This is because the `Universal Sentence Encoder (USE)` takes care of tokenization for us.\n",
        "\n",
        "This type of model is called transfer learning, or more specifically, **feature extraction transfer learning**. In other words, taking the patterns a model has learned elsewhere and applying it to our own problem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8S5Yh76iT76y"
      },
      "source": [
        "# Download pretrained TensorFlow Hub USE\n",
        "import tensorflow_hub as hub\n",
        "tf_hub_embedding_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        trainable=False,\n",
        "                                        name=\"universal_sentence_encoder\")"
      ],
      "execution_count": 45,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rnTm9-ArUPWI",
        "outputId": "330360aa-d933-4594-cffc-18f8d4cf5809"
      },
      "source": [
        "# Test out the embedding on a random sentence\n",
        "random_training_sentence = random.choice(train_sentences)\n",
        "print(f\"Random training sentence:\\n{random_training_sentence}\\n\")\n",
        "use_embedded_sentence = tf_hub_embedding_layer([random_training_sentence])\n",
        "print(f\"Sentence after embedding:\\n{use_embedded_sentence[0][:30]} (truncated output)...\\n\")\n",
        "print(f\"Length of sentence embedding:\\n{len(use_embedded_sentence[0])}\")"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Random training sentence:\n",
            "secondary outcome measures were a change in pain and postconcussive symptoms within @ to @ days of fluid administration .\n",
            "\n",
            "Sentence after embedding:\n",
            "[-0.04461711 -0.03223816 -0.04212618  0.02210478 -0.01730661  0.04980363\n",
            " -0.03563878  0.00914231 -0.02384312  0.06856254  0.08184546 -0.02110292\n",
            "  0.07041292  0.02427626  0.04708336  0.01584669 -0.09069967  0.00371628\n",
            "  0.07962336  0.03195264  0.04089073  0.04776753 -0.00710841 -0.04335723\n",
            "  0.04926984 -0.00795815  0.03854619  0.07939983 -0.06945099  0.04340277] (truncated output)...\n",
            "\n",
            "Length of sentence embedding:\n",
            "512\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0BrYwbV7Ui9W"
      },
      "source": [
        "### Building and fitting an NLP feature extraction model from TensorFlow Hub"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o3PJYUHkUry3"
      },
      "source": [
        "# Define feature extractor model using TF Hub layer\n",
        "inputs = layers.Input(shape=[], dtype=tf.string)\n",
        "pretrained_embedding = tf_hub_embedding_layer(inputs) # tokenize text and create embedding\n",
        "x = layers.Dense(128, activation=\"relu\")(pretrained_embedding) # add a fully connected layer on top of the embedding\n",
        "# Note: you could add more layers here if you wanted to\n",
        "outputs = layers.Dense(5, activation=\"softmax\")(x) # create the output layer\n",
        "model_2 = tf.keras.Model(inputs=inputs,\n",
        "                        outputs=outputs)\n",
        "\n",
        "# Compile the model\n",
        "model_2.compile(loss=\"categorical_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 47,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JWzn7queUvJn",
        "outputId": "3fca8b06-d823-42eb-998e-e4cb7a306d45"
      },
      "source": [
        "# Get a summary of the model\n",
        "model_2.summary()"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_2 (InputLayer)         [(None,)]                 0         \n",
            "_________________________________________________________________\n",
            "universal_sentence_encoder ( (None, 512)               256797824 \n",
            "_________________________________________________________________\n",
            "dense_1 (Dense)              (None, 128)               65664     \n",
            "_________________________________________________________________\n",
            "dense_2 (Dense)              (None, 5)                 645       \n",
            "=================================================================\n",
            "Total params: 256,864,133\n",
            "Trainable params: 66,309\n",
            "Non-trainable params: 256,797,824\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IFlxT8IjUxGv",
        "outputId": "cd1989bd-f14f-40e0-d6e2-558962d9f174"
      },
      "source": [
        "# Fit feature extractor model for 3 epochs\n",
        "model_2.fit(train_dataset,\n",
        "            steps_per_epoch=int(0.1 * len(train_dataset)),\n",
        "            epochs=3,\n",
        "            validation_data=valid_dataset,\n",
        "            validation_steps=int(0.1 * len(valid_dataset)))"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "562/562 [==============================] - 9s 12ms/step - loss: 0.9153 - accuracy: 0.6515 - val_loss: 0.7961 - val_accuracy: 0.6882\n",
            "Epoch 2/3\n",
            "562/562 [==============================] - 6s 11ms/step - loss: 0.7657 - accuracy: 0.7046 - val_loss: 0.7508 - val_accuracy: 0.7081\n",
            "Epoch 3/3\n",
            "562/562 [==============================] - 6s 11ms/step - loss: 0.7479 - accuracy: 0.7135 - val_loss: 0.7330 - val_accuracy: 0.7191\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.callbacks.History at 0x7f1b66102950>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oYO8oGtZU1o9",
        "outputId": "f6e16259-adb1-4fa4-a5dc-a8bd467bf0b6"
      },
      "source": [
        "# Evaluate on whole validation dataset\n",
        "model_2.evaluate(valid_dataset)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 9s 10ms/step - loss: 0.7363 - accuracy: 0.7167\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.7362678050994873, 0.7166688442230225]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZUxpOCHyVEhI",
        "outputId": "8451e4bf-ccbd-4a3d-9610-31474be6661b"
      },
      "source": [
        "# Make predictions with feature extraction model\n",
        "model_2_pred_probs = model_2.predict(valid_dataset)\n",
        "model_2_pred_probs"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4.2138055e-01, 3.7459037e-01, 2.0373834e-03, 1.9416024e-01,\n",
              "        7.8314757e-03],\n",
              "       [3.4396559e-01, 5.0439215e-01, 4.3844962e-03, 1.4445728e-01,\n",
              "        2.8004525e-03],\n",
              "       [2.1493916e-01, 1.5683734e-01, 1.9048801e-02, 5.6750810e-01,\n",
              "        4.1666582e-02],\n",
              "       ...,\n",
              "       [2.1218751e-03, 6.4071780e-03, 4.3606125e-02, 9.1092283e-04,\n",
              "        9.4695389e-01],\n",
              "       [3.5173313e-03, 4.1535258e-02, 1.9679344e-01, 1.3398867e-03,\n",
              "        7.5681412e-01],\n",
              "       [1.6621438e-01, 3.0003798e-01, 4.7229695e-01, 7.8768078e-03,\n",
              "        5.3573865e-02]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 51
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kY2TbJqLVGLd",
        "outputId": "5a61e48b-f115-4644-9e4d-91def4f34388"
      },
      "source": [
        "# Convert the predictions with feature extraction model to classes\n",
        "model_2_preds = tf.argmax(model_2_pred_probs, axis=1)\n",
        "model_2_preds"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(30212,), dtype=int64, numpy=array([0, 1, 3, ..., 4, 4, 2])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Env6Ch23VINu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b1f66511-4f62-47f0-8cb6-c2e60689c8ca"
      },
      "source": [
        "# Calculate results from TF Hub pretrained embeddings results on validation set\n",
        "model_2_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                    y_pred=model_2_preds)\n",
        "model_2_results"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 71.66688732953793,\n",
              " 'f1': 0.71368365848539,\n",
              " 'precision': 0.7173482917303345,\n",
              " 'recall': 0.7166688732953793}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iJiGRfXgVKT0"
      },
      "source": [
        "## Model 3: Conv1D with character embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bC2BUk8p5n_4"
      },
      "source": [
        "### Creating a character-level tokenizer\n",
        "\n",
        "We've built models with a custom token embedding and a pretrained token embedding, how about we build one using a character embedding?\n",
        "\n",
        "The difference between a character and token embedding is that the character embedding is created using sequences split into characters *(e.g. hello -> [h, e, l, l, o])* where as a token embedding is created on sequences split into tokens.\n",
        "\n",
        "We can create a character-level embedding by first vectorizing our sequences (after they've been split into characters) using the `TextVectorization` class and then passing those vectorized sequences through an `Embedding` layer.\n",
        "\n",
        "Before we can vectorize our sequences on a character-level we'll need to split them into characters. Let's write a function to do so."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 69
        },
        "id": "lkAYI4dl5yg9",
        "outputId": "93e3fd47-d18d-47cc-9b2c-3e84438dc218"
      },
      "source": [
        "# Make function to split sentences into characters\n",
        "def split_chars(text):\n",
        "  return \" \".join(list(text))\n",
        "\n",
        "# Test splitting non-character-level sequence into characters\n",
        "split_chars(random_training_sentence)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'s e c o n d a r y   o u t c o m e   m e a s u r e s   w e r e   a   c h a n g e   i n   p a i n   a n d   p o s t c o n c u s s i v e   s y m p t o m s   w i t h i n   @   t o   @   d a y s   o f   f l u i d   a d m i n i s t r a t i o n   .'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-ltr7Aa_6D18",
        "outputId": "bde94515-2822-4cca-a63f-20a2c28df5d0"
      },
      "source": [
        "# Split sequence-level data splits into character-level data splits\n",
        "train_chars = [split_chars(sentence) for sentence in train_sentences]\n",
        "val_chars = [split_chars(sentence) for sentence in val_sentences]\n",
        "test_chars = [split_chars(sentence) for sentence in test_sentences]\n",
        "print(train_chars[0])"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "t o   i n v e s t i g a t e   t h e   e f f i c a c y   o f   @   w e e k s   o f   d a i l y   l o w - d o s e   o r a l   p r e d n i s o l o n e   i n   i m p r o v i n g   p a i n   ,   m o b i l i t y   ,   a n d   s y s t e m i c   l o w - g r a d e   i n f l a m m a t i o n   i n   t h e   s h o r t   t e r m   a n d   w h e t h e r   t h e   e f f e c t   w o u l d   b e   s u s t a i n e d   a t   @   w e e k s   i n   o l d e r   a d u l t s   w i t h   m o d e r a t e   t o   s e v e r e   k n e e   o s t e o a r t h r i t i s   (   o a   )   .\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lu5fg7Gq6PTL",
        "outputId": "37906ae6-7946-4cf7-a72a-0a4067ec8f5e"
      },
      "source": [
        "# What's the average character length?\n",
        "char_lens = [len(sentence) for sentence in train_sentences]\n",
        "mean_char_len = np.mean(char_lens)\n",
        "mean_char_len"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "149.3662574983337"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "8wXXI0Ol6Sk0",
        "outputId": "7504df6b-0bcb-47fa-9615-4a93b68a2f92"
      },
      "source": [
        "# Check the distribution of our sequences at character-level\n",
        "import matplotlib.pyplot as plt\n",
        "plt.hist(char_lens, bins=7);"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAWqUlEQVR4nO3df6zddZ3n8edr2wF/zEqLdBimbbZ1bNxUsrNigzVuJsY6paCxbIKmxCzVYW12xV1n1kSLJkNWJYGdyTCSKA4jHYthQZZxlkZhu13EmE0W5CLKT5EroLQBe6UIu2P8Uee9f5zPhWO9/ZTec3vuFZ6P5OR+v+/P53vO+3xz73n1++PepqqQJOlw/sl8NyBJWtgMCklSl0EhSeoyKCRJXQaFJKlr8Xw3MNdOOumkWrVq1Xy3IUm/Ue68884fVdWymcZecEGxatUqJiYm5rsNSfqNkuT7hxvz1JMkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeo6YlAk2ZFkf5J7Zxj7UJJKclJbT5LLk0wmuTvJaUNztyZ5qD22DtVfn+Sets3lSdLqJybZ0+bvSbJ0bt6yJOloPJ8jis8Dmw4tJlkJbAR+MFQ+E1jTHtuAK9rcE4GLgDcApwMXDX3wXwG8b2i76dfaDtxSVWuAW9q6JGnMjvib2VX19SSrZhi6DPgwcONQbTNwdQ3+N6TbkixJcgrwZmBPVR0ASLIH2JTka8Arquq2Vr8aOBu4uT3Xm9vz7gS+BnzkqN7dUVq1/SvH8unn3KOXvG2+W5D0IjCraxRJNgP7qurbhwwtBx4bWt/bar363hnqACdX1eNt+Qng5E4/25JMJJmYmpo62rcjSeo46qBI8jLgo8CfzX07M2tHKIf9P1ur6sqqWldV65Ytm/FvWkmSZmk2RxS/D6wGvp3kUWAF8M0kvwvsA1YOzV3Rar36ihnqAD9sp61oX/fPoldJ0oiOOiiq6p6q+p2qWlVVqxicLjqtqp4AdgHntbuf1gNPt9NHu4GNSZa2i9gbgd1t7Jkk69vdTufx3DWPXcD03VFb+dVrIZKkMXk+t8deC/wf4DVJ9iY5vzP9JuBhYBL4G+D9AO0i9ieAO9rj49MXttucz7VtvsfgQjbAJcAfJXkIeGtblySN2fO56+ncI4yvGlou4ILDzNsB7JihPgGcOkP9SWDDkfqTJB1b/ma2JKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkrqOGBRJdiTZn+TeodqfJ/lOkruT/H2SJUNjFyaZTPJgkjOG6ptabTLJ9qH66iS3t/oXkxzX6se39ck2vmqu3rQk6fl7PkcUnwc2HVLbA5xaVf8C+C5wIUCStcAW4LVtm88kWZRkEfBp4ExgLXBumwtwKXBZVb0aeAo4v9XPB55q9cvaPEnSmB0xKKrq68CBQ2r/s6oOttXbgBVteTNwXVX9rKoeASaB09tjsqoerqqfA9cBm5MEeAtwQ9t+J3D20HPtbMs3ABvafEnSGM3FNYo/Bm5uy8uBx4bG9rba4eqvBH48FDrT9V95rjb+dJv/a5JsSzKRZGJqamrkNyRJes5IQZHkY8BB4Jq5aWd2qurKqlpXVeuWLVs2n61I0gvO4tlumOQ9wNuBDVVVrbwPWDk0bUWrcZj6k8CSJIvbUcPw/Onn2ptkMXBCmy9JGqNZHVEk2QR8GHhHVf1kaGgXsKXdsbQaWAN8A7gDWNPucDqOwQXvXS1gbgXOadtvBW4ceq6tbfkc4KtDgSRJGpMjHlEkuRZ4M3BSkr3ARQzucjoe2NOuL99WVf+uqu5Lcj1wP4NTUhdU1S/b83wA2A0sAnZU1X3tJT4CXJfkk8BdwFWtfhXwhSSTDC6mb5mD9ytJOkpHDIqqOneG8lUz1KbnXwxcPEP9JuCmGeoPM7gr6tD6T4F3Hqk/SdKx5W9mS5K6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXUcMiiQ7kuxPcu9Q7cQke5I81L4ubfUkuTzJZJK7k5w2tM3WNv+hJFuH6q9Pck/b5vIk6b2GJGm8ns8RxeeBTYfUtgO3VNUa4Ja2DnAmsKY9tgFXwOBDH7gIeANwOnDR0Af/FcD7hrbbdITXkCSN0RGDoqq+Dhw4pLwZ2NmWdwJnD9WvroHbgCVJTgHOAPZU1YGqegrYA2xqY6+oqtuqqoCrD3mumV5DkjRGs71GcXJVPd6WnwBObsvLgceG5u1ttV597wz13mv8miTbkkwkmZiamprF25EkHc7IF7PbkUDNQS+zfo2qurKq1lXVumXLlh3LViTpRWe2QfHDdtqI9nV/q+8DVg7NW9FqvfqKGeq915AkjdFsg2IXMH3n0lbgxqH6ee3up/XA0+300W5gY5Kl7SL2RmB3G3smyfp2t9N5hzzXTK8hSRqjxUeakORa4M3ASUn2Mrh76RLg+iTnA98H3tWm3wScBUwCPwHeC1BVB5J8Arijzft4VU1fIH8/gzurXgrc3B50XkOSNEZHDIqqOvcwQxtmmFvABYd5nh3AjhnqE8CpM9SfnOk1JEnj5W9mS5K6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXSMFRZI/TXJfknuTXJvkJUlWJ7k9yWSSLyY5rs09vq1PtvFVQ89zYas/mOSMofqmVptMsn2UXiVJszProEiyHPiPwLqqOhVYBGwBLgUuq6pXA08B57dNzgeeavXL2jySrG3bvRbYBHwmyaIki4BPA2cCa4Fz21xJ0hiNeuppMfDSJIuBlwGPA28BbmjjO4Gz2/Lmtk4b35AkrX5dVf2sqh4BJoHT22Oyqh6uqp8D17W5kqQxmnVQVNU+4C+AHzAIiKeBO4EfV9XBNm0vsLwtLwcea9sebPNfOVw/ZJvD1X9Nkm1JJpJMTE1NzfYtSZJmMMqpp6UM/oW/Gvg94OUMTh2NXVVdWVXrqmrdsmXL5qMFSXrBGuXU01uBR6pqqqp+AXwJeBOwpJ2KAlgB7GvL+4CVAG38BODJ4foh2xyuLkkao1GC4gfA+iQva9caNgD3A7cC57Q5W4Eb2/Kutk4b/2pVVatvaXdFrQbWAN8A7gDWtLuojmNwwXvXCP1KkmZh8ZGnzKyqbk9yA/BN4CBwF3Al8BXguiSfbLWr2iZXAV9IMgkcYPDBT1Xdl+R6BiFzELigqn4JkOQDwG4Gd1TtqKr7ZtuvJGl2Zh0UAFV1EXDRIeWHGdyxdOjcnwLvPMzzXAxcPEP9JuCmUXqUJI3G38yWJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUtdIQZFkSZIbknwnyQNJ3pjkxCR7kjzUvi5tc5Pk8iSTSe5OctrQ82xt8x9KsnWo/vok97RtLk+SUfqVJB29UY8oPgX8j6r658AfAA8A24FbqmoNcEtbBzgTWNMe24ArAJKcCFwEvAE4HbhoOlzanPcNbbdpxH4lSUdp1kGR5ATgD4GrAKrq51X1Y2AzsLNN2wmc3ZY3A1fXwG3AkiSnAGcAe6rqQFU9BewBNrWxV1TVbVVVwNVDzyVJGpNRjihWA1PA3ya5K8nnkrwcOLmqHm9zngBObsvLgceGtt/bar363hnqvybJtiQTSSampqZGeEuSpEONEhSLgdOAK6rqdcA/8NxpJgDakUCN8BrPS1VdWVXrqmrdsmXLjvXLSdKLyihBsRfYW1W3t/UbGATHD9tpI9rX/W18H7ByaPsVrdarr5ihLkkao1kHRVU9ATyW5DWttAG4H9gFTN+5tBW4sS3vAs5rdz+tB55up6h2AxuTLG0XsTcCu9vYM0nWt7udzht6LknSmCwecfv/AFyT5DjgYeC9DMLn+iTnA98H3tXm3gScBUwCP2lzqaoDST4B3NHmfbyqDrTl9wOfB14K3NwekqQxGikoqupbwLoZhjbMMLeACw7zPDuAHTPUJ4BTR+lRkjQafzNbktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqGjkokixKcleSL7f11UluTzKZ5ItJjmv149v6ZBtfNfQcF7b6g0nOGKpvarXJJNtH7VWSdPTm4ojig8ADQ+uXApdV1auBp4DzW/184KlWv6zNI8laYAvwWmAT8JkWPouATwNnAmuBc9tcSdIYjRQUSVYAbwM+19YDvAW4oU3ZCZzdlje3ddr4hjZ/M3BdVf2sqh4BJoHT22Oyqh6uqp8D17W5kqQxGvWI4q+ADwP/2NZfCfy4qg629b3A8ra8HHgMoI0/3eY/Wz9km8PVf02SbUkmkkxMTU2N+JYkScNmHRRJ3g7sr6o757CfWamqK6tqXVWtW7Zs2Xy3I0kvKItH2PZNwDuSnAW8BHgF8ClgSZLF7ahhBbCvzd8HrAT2JlkMnAA8OVSfNrzN4eqSpDGZ9RFFVV1YVSuqahWDi9Ffrap3A7cC57RpW4Eb2/Kutk4b/2pVVatvaXdFrQbWAN8A7gDWtLuojmuvsWu2/UqSZmeUI4rD+QhwXZJPAncBV7X6VcAXkkwCBxh88FNV9yW5HrgfOAhcUFW/BEjyAWA3sAjYUVX3HYN+f2Ot2v6V+W7heXv0krfNdwuSZmlOgqKqvgZ8rS0/zOCOpUPn/BR452G2vxi4eIb6TcBNc9GjJGl2/M1sSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpa9ZBkWRlkluT3J/kviQfbPUTk+xJ8lD7urTVk+TyJJNJ7k5y2tBzbW3zH0qydaj++iT3tG0uT5JR3qwk6eiNckRxEPhQVa0F1gMXJFkLbAduqao1wC1tHeBMYE17bAOugEGwABcBbwBOBy6aDpc2531D220aoV9J0izMOiiq6vGq+mZb/r/AA8ByYDOws03bCZzdljcDV9fAbcCSJKcAZwB7qupAVT0F7AE2tbFXVNVtVVXA1UPPJUkakzm5RpFkFfA64Hbg5Kp6vA09AZzclpcDjw1ttrfVevW9M9Rnev1tSSaSTExNTY30XiRJv2rkoEjy28DfAX9SVc8Mj7UjgRr1NY6kqq6sqnVVtW7ZsmXH+uUk6UVlpKBI8lsMQuKaqvpSK/+wnTaifd3f6vuAlUObr2i1Xn3FDHVJ0hiNctdTgKuAB6rqL4eGdgHTdy5tBW4cqp/X7n5aDzzdTlHtBjYmWdouYm8EdrexZ5Ksb6913tBzSZLGZPEI274J+DfAPUm+1WofBS4Brk9yPvB94F1t7CbgLGAS+AnwXoCqOpDkE8Adbd7Hq+pAW34/8HngpcDN7SFJGqNZB0VV/W/gcL/XsGGG+QVccJjn2gHsmKE+AZw62x4lSaPzN7MlSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1LV4vhs4kiSbgE8Bi4DPVdUl89ySZmHV9q/MdwtH5dFL3jbfLUgLxoI+okiyCPg0cCawFjg3ydr57UqSXlwWdFAApwOTVfVwVf0cuA7YPM89SdKLykI/9bQceGxofS/whkMnJdkGbGur/y/Jg7N8vZOAH81y2/lgv8dILgV+g/pt7PfYeqH3+88ON7DQg+J5qaorgStHfZ4kE1W1bg5aGgv7Pbbs99iy32NrLvtd6Kee9gErh9ZXtJokaUwWelDcAaxJsjrJccAWYNc89yRJLyoL+tRTVR1M8gFgN4PbY3dU1X3H8CVHPn01ZvZ7bNnvsWW/x9ac9ZuqmqvnkiS9AC30U0+SpHlmUEiSugwKBn8mJMmDSSaTbJ/vfgCSrExya5L7k9yX5IOtfmKSPUkeal+XtnqSXN7ew91JTpunvhcluSvJl9v66iS3t76+2G5KIMnxbX2yja+ah16XJLkhyXeSPJDkjQt5/yb50/a9cG+Sa5O8ZCHt3yQ7kuxPcu9Q7aj3Z5Ktbf5DSbaOud8/b98Pdyf5+yRLhsYubP0+mOSMofpYPj9m6ndo7ENJKslJbX1u929VvagfDC6Sfw94FXAc8G1g7QLo6xTgtLb8T4HvMvgzJv8F2N7q24FL2/JZwM1AgPXA7fPU938C/ivw5bZ+PbClLX8W+Pdt+f3AZ9vyFuCL89DrTuDftuXjgCULdf8y+OXTR4CXDu3X9yyk/Qv8IXAacO9Q7aj2J3Ai8HD7urQtLx1jvxuBxW350qF+17bPhuOB1e0zY9E4Pz9m6rfVVzK44ef7wEnHYv+O9QdzIT6ANwK7h9YvBC6c775m6PNG4I+AB4FTWu0U4MG2/NfAuUPzn503xh5XALcAbwG+3L5JfzT0g/fsvm7f2G9sy4vbvIyx1xPaB28OqS/I/ctzf6XgxLa/vgycsdD2L7DqkA/eo9qfwLnAXw/Vf2Xese73kLF/DVzTln/lc2F6/47782OmfoEbgD8AHuW5oJjT/eupp5n/TMjyeeplRu20weuA24GTq+rxNvQEcHJbXgjv46+ADwP/2NZfCfy4qg7O0NOz/bbxp9v8cVkNTAF/206VfS7Jy1mg+7eq9gF/AfwAeJzB/rqThbt/px3t/lwI38fT/pjBv8phgfabZDOwr6q+fcjQnPZrUCxwSX4b+DvgT6rqmeGxGvyTYEHc35zk7cD+qrpzvnt5nhYzOIy/oqpeB/wDg1Mjz1pg+3cpgz+IuRr4PeDlwKZ5beooLaT9eSRJPgYcBK6Z714OJ8nLgI8Cf3asX8ugWMB/JiTJbzEIiWuq6kut/MMkp7TxU4D9rT7f7+NNwDuSPMrgr/y+hcH/I7IkyfQvdg739Gy/bfwE4Mkx9rsX2FtVt7f1GxgEx0Ldv28FHqmqqar6BfAlBvt8oe7faUe7P+d7P5PkPcDbgXe3cKPT13z2+/sM/uHw7fZztwL4ZpLf7fQ1q34NigX6Z0KSBLgKeKCq/nJoaBcwfafCVgbXLqbr57W7HdYDTw8d8h9zVXVhVa2oqlUM9uFXq+rdwK3AOYfpd/p9nNPmj+1fm1X1BPBYkte00gbgfhbo/mVwyml9kpe1743pfhfk/h1ytPtzN7AxydJ2FLWx1cYig/8o7cPAO6rqJ0NDu4At7W6y1cAa4BvM4+dHVd1TVb9TVavaz91eBjfAPMFc799jddHlN+nB4A6B7zK4e+Fj891P6+lfMThMvxv4VnucxeA88y3AQ8D/Ak5s88PgP3n6HnAPsG4ee38zz9319CoGP1CTwH8Djm/1l7T1yTb+qnno818CE20f/3cGd4Es2P0L/GfgO8C9wBcY3IGzYPYvcC2D6ye/aB9a589mfzK4NjDZHu8dc7+TDM7hT//MfXZo/sdavw8CZw7Vx/L5MVO/h4w/ynMXs+d0//onPCRJXZ56kiR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXf8fWfBom7qekSwAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4ifuLnxi6WWN",
        "outputId": "9f0d7f80-1f22-48b6-a908-8c104333ff0c"
      },
      "source": [
        "# Find what character length covers 95% of seuences\n",
        "output_seq_char_len = int(np.percentile(char_lens, 95))\n",
        "output_seq_char_len\n"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "290"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6_-KpsgC6cup"
      },
      "source": [
        "We'll set max_tokens (the total number of different characters in our sequences) to 28, in other words, 26 letters of the alphabet + space + OOV (out of vocabulary or unknown) tokens."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "5AH0YKoX6k_D",
        "outputId": "6b3bfb91-3c56-4ec2-9cca-e6ac884b0543"
      },
      "source": [
        "# Get all keyboard characters for char-level embedding\n",
        "import string\n",
        "alphabet = string.ascii_lowercase + string.digits + string.punctuation\n",
        "alphabet"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'abcdefghijklmnopqrstuvwxyz0123456789!\"#$%&\\'()*+,-./:;<=>?@[\\\\]^_`{|}~'"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gkfcUvwY6m8S"
      },
      "source": [
        "# Create char-level token vectorizer instance\n",
        "NUM_CHAR_TOKENS = len(alphabet) + 2 # num characters in alphabet + space + OOV token\n",
        "char_vectorizer = TextVectorization(max_tokens=NUM_CHAR_TOKENS,  \n",
        "                                    output_sequence_length=output_seq_char_len,\n",
        "                                    standardize=\"lower_and_strip_punctuation\",\n",
        "                                    name=\"char_vectorizer\")\n",
        "\n",
        "# Adapt character vectorizer to training characters\n",
        "char_vectorizer.adapt(train_chars)"
      ],
      "execution_count": 60,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1Rb3poVE6pif",
        "outputId": "f33baf86-b769-4847-c298-0079b7912a9d"
      },
      "source": [
        "# Check character vocabulary characteristics\n",
        "char_vocab = char_vectorizer.get_vocabulary()\n",
        "print(f\"Number of different characters in character vocab: {len(char_vocab)}\")\n",
        "print(f\"5 most common characters: {char_vocab[:5]}\")\n",
        "print(f\"5 least common characters: {char_vocab[-5:]}\")"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number of different characters in character vocab: 28\n",
            "5 most common characters: ['', '[UNK]', 'e', 't', 'i']\n",
            "5 least common characters: ['k', 'x', 'z', 'q', 'j']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SIK0hHQE6tUX",
        "outputId": "a6274524-72e3-4042-a9d3-12a9b335fd6b"
      },
      "source": [
        "# Test out character vectorizer\n",
        "random_train_chars = random.choice(train_chars)\n",
        "print(f\"Charified text:\\n{random_train_chars}\")\n",
        "print(f\"\\nLength of chars: {len(random_train_chars.split())}\")\n",
        "vectorized_chars = char_vectorizer([random_train_chars])\n",
        "print(f\"\\nVectorized chars:\\n{vectorized_chars}\")\n",
        "print(f\"\\nLength of vectorized chars: {len(vectorized_chars[0])}\")"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Charified text:\n",
            "w e   a s s e s s e d   p u l m o n a r y   f u n c t i o n   ,   d i s t a n c e   c o v e r e d   d u r i n g   t h e   @ - m i n u t e   w a l k   t e s t   (   @ m w t   )   ,   r e s p i r a t o r y   a n d   p e r i p h e r a l   m u s c l e   s t r e n g t h   ,   h e a l t h - r e l a t e d   q u a l i t y   o f   l i f e   (   h r q o l   )   ,   b o d y   c o m p o s i t i o n   ,   a n d   l e v e l   o f   a c t i v i t i e s   o f   d a i l y   l i v i n g   (   a d l s   )   b e f o r e   a n d   a f t e r   a n   @ - w e e k   w a l k i n g   p r o g r a m   .\n",
            "\n",
            "Length of chars: 241\n",
            "\n",
            "Vectorized chars:\n",
            "[[20  2  5  9  9  2  9  9  2 10 14 16 12 15  7  6  5  8 19 17 16  6 11  3\n",
            "   4  7  6 10  4  9  3  5  6 11  2 11  7 21  2  8  2 10 10 16  8  4  6 18\n",
            "   3 13  2 15  4  6 16  3  2 20  5 12 23  3  2  9  3 15 20  3  8  2  9 14\n",
            "   4  8  5  3  7  8 19  5  6 10 14  2  8  4 14 13  2  8  5 12 15 16  9 11\n",
            "  12  2  9  3  8  2  6 18  3 13 13  2  5 12  3 13  8  2 12  5  3  2 10 26\n",
            "  16  5 12  4  3 19  7 17 12  4 17  2 13  8 26  7 12 22  7 10 19 11  7 15\n",
            "  14  7  9  4  3  4  7  6  5  6 10 12  2 21  2 12  7 17  5 11  3  4 21  4\n",
            "   3  4  2  9  7 17 10  5  4 12 19 12  4 21  4  6 18  5 10 12  9 22  2 17\n",
            "   7  8  2  5  6 10  5 17  3  2  8  5  6 20  2  2 23 20  5 12 23  4  6 18\n",
            "  14  8  7 18  8  5 15  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
            "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
            "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
            "   0  0]]\n",
            "\n",
            "Length of vectorized chars: 290\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xCvrv496vtE"
      },
      "source": [
        "You'll notice sequences with a length shorter than 290 (`output_seq_char_length`) get padded with zeros on the end, this ensures all sequences passed to our model are the same length.\n",
        "\n",
        "Also, due to the standardize parameter of `TextVectorization` being \"`lower_and_strip_punctuation`\" and the split parameter being \"`whitespace`\" by default, symbols (such as @) and spaces are removed."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wL3O6RWI7yGH"
      },
      "source": [
        "### Creating a character-level embedding\n",
        "\n",
        "We've got a way to vectorize our character-level sequences, now's time to create a character-level embedding.\n",
        "\n",
        "Just like our custom token embedding, we can do so using the `tensorflow.keras.layers.Embedding` class.\n",
        "\n",
        "Our character-level embedding layer requires an input dimension and output dimension.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S_llxFGT8CWD",
        "outputId": "fc3ee612-e10e-417c-8faa-eed8ac85c2ca"
      },
      "source": [
        "# Create char embedding layer\n",
        "char_embed = layers.Embedding(input_dim=NUM_CHAR_TOKENS, # number of different characters\n",
        "                              output_dim=25, # embedding dimension of each character (same as Figure 1 in https://arxiv.org/pdf/1612.05251.pdf)\n",
        "                              mask_zero=True,\n",
        "                              name=\"char_embed\")\n",
        "\n",
        "# Test out character embedding layer\n",
        "print(f\"Charified text (before vectorization and embedding):\\n{random_train_chars}\\n\")\n",
        "char_embed_example = char_embed(char_vectorizer([random_train_chars]))\n",
        "print(f\"Embedded chars (after vectorization and embedding):\\n{char_embed_example}\\n\")\n",
        "print(f\"Character embedding shape: {char_embed_example.shape}\")"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Charified text (before vectorization and embedding):\n",
            "w e   a s s e s s e d   p u l m o n a r y   f u n c t i o n   ,   d i s t a n c e   c o v e r e d   d u r i n g   t h e   @ - m i n u t e   w a l k   t e s t   (   @ m w t   )   ,   r e s p i r a t o r y   a n d   p e r i p h e r a l   m u s c l e   s t r e n g t h   ,   h e a l t h - r e l a t e d   q u a l i t y   o f   l i f e   (   h r q o l   )   ,   b o d y   c o m p o s i t i o n   ,   a n d   l e v e l   o f   a c t i v i t i e s   o f   d a i l y   l i v i n g   (   a d l s   )   b e f o r e   a n d   a f t e r   a n   @ - w e e k   w a l k i n g   p r o g r a m   .\n",
            "\n",
            "Embedded chars (after vectorization and embedding):\n",
            "[[[-0.04713828  0.01089247  0.00599311 ...  0.00050195  0.03528389\n",
            "   -0.00238868]\n",
            "  [-0.0356781  -0.0016773  -0.04435882 ... -0.02763401 -0.01474966\n",
            "   -0.00032056]\n",
            "  [-0.03886486  0.0038087   0.04308527 ...  0.00375962  0.01215118\n",
            "   -0.0483592 ]\n",
            "  ...\n",
            "  [ 0.04746694  0.0296837   0.01055058 ... -0.01572698  0.01604297\n",
            "    0.00695161]\n",
            "  [ 0.04746694  0.0296837   0.01055058 ... -0.01572698  0.01604297\n",
            "    0.00695161]\n",
            "  [ 0.04746694  0.0296837   0.01055058 ... -0.01572698  0.01604297\n",
            "    0.00695161]]]\n",
            "\n",
            "Character embedding shape: (1, 290, 25)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvJnKmS_644R"
      },
      "source": [
        "### Building a Conv1D model to fit on character embeddings\n",
        "\n",
        "Now we've got a way to turn our character-level sequences into numbers (`char_vectorizer`) as well as numerically represent them as an embedding (`char_embed`) let's test how effective they are at encoding the information in our sequences by creating a character-level sequence model.\n",
        "\n",
        "The model will have the same structure as our custom token embedding model (`model_1`) except it'll take character-level sequences as input instead of token-level sequences.\n",
        "\n",
        "`Input (character-level text) -> Tokenize -> Embedding -> Layers (Conv1D, GlobalMaxPool1D) -> Output (label probability)`\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hN0SlIG57ORy"
      },
      "source": [
        "# Make Conv1D on chars only\n",
        "inputs = layers.Input(shape=(1,), dtype=\"string\")\n",
        "char_vectors = char_vectorizer(inputs)\n",
        "char_embeddings = char_embed(char_vectors)\n",
        "x = layers.Conv1D(64, kernel_size=5, padding=\"same\", activation=\"relu\")(char_embeddings)\n",
        "x = layers.GlobalMaxPool1D()(x)\n",
        "outputs = layers.Dense(num_classes, activation=\"softmax\")(x)\n",
        "model_3 = tf.keras.Model(inputs=inputs,\n",
        "                         outputs=outputs,\n",
        "                         name=\"model_3_conv1D_char_embedding\")\n",
        "\n",
        "# Compile model\n",
        "model_3.compile(loss=\"categorical_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 64,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6ofjKXir7cQF",
        "outputId": "bf4721b1-c56d-4b39-da31-5f0ddb8c0a3d"
      },
      "source": [
        "# Check the summary of conv1d_char_model\n",
        "model_3.summary()"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_3_conv1D_char_embedding\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input_3 (InputLayer)         [(None, 1)]               0         \n",
            "_________________________________________________________________\n",
            "char_vectorizer (TextVectori (None, 290)               0         \n",
            "_________________________________________________________________\n",
            "char_embed (Embedding)       (None, 290, 25)           1750      \n",
            "_________________________________________________________________\n",
            "conv1d_1 (Conv1D)            (None, 290, 64)           8064      \n",
            "_________________________________________________________________\n",
            "global_max_pooling1d (Global (None, 64)                0         \n",
            "_________________________________________________________________\n",
            "dense_3 (Dense)              (None, 5)                 325       \n",
            "=================================================================\n",
            "Total params: 10,139\n",
            "Trainable params: 10,139\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z9kZizVf7eHQ",
        "outputId": "e6915345-145d-46dd-e487-7d8c4c03c9cd"
      },
      "source": [
        "# Create char datasets\n",
        "train_char_dataset = tf.data.Dataset.from_tensor_slices((train_chars, train_labels_one_hot)).batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "val_char_dataset = tf.data.Dataset.from_tensor_slices((val_chars, val_labels_one_hot)).batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "\n",
        "train_char_dataset"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<PrefetchDataset shapes: ((None,), (None, 5)), types: (tf.string, tf.float64)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iDJ8Jvct7gY1",
        "outputId": "d09fbb2a-4cd1-4645-dd96-c41663720ca7"
      },
      "source": [
        "# Fit the model on chars only\n",
        "model_3_history = model_3.fit(train_char_dataset,\n",
        "                              steps_per_epoch=int(0.1 * len(train_char_dataset)),\n",
        "                              epochs=3,\n",
        "                              validation_data=val_char_dataset,\n",
        "                              validation_steps=int(0.1 * len(val_char_dataset)))"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "562/562 [==============================] - 4s 7ms/step - loss: 1.2649 - accuracy: 0.4928 - val_loss: 1.0475 - val_accuracy: 0.5851\n",
            "Epoch 2/3\n",
            "562/562 [==============================] - 3s 6ms/step - loss: 1.0114 - accuracy: 0.5945 - val_loss: 0.9480 - val_accuracy: 0.6307\n",
            "Epoch 3/3\n",
            "562/562 [==============================] - 3s 6ms/step - loss: 0.9413 - accuracy: 0.6282 - val_loss: 0.8767 - val_accuracy: 0.6536\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PXxPl8fwdZ5P",
        "outputId": "26a96ffb-3a68-4efd-9597-b86730ef4403"
      },
      "source": [
        "# Evaluate model_3 on whole validation char dataset\n",
        "model_3.evaluate(val_char_dataset)"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 4s 4ms/step - loss: 0.9522 - accuracy: 0.6267\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.9521686434745789, 0.6267377138137817]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B6-NoSBmdea8",
        "outputId": "eb7f9278-629b-4a1b-b85b-bbc3288047a6"
      },
      "source": [
        "# Make predictions with character model only\n",
        "model_3_pred_probs = model_3.predict(val_char_dataset)\n",
        "model_3_pred_probs"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.09241142, 0.23828042, 0.10692347, 0.5061677 , 0.05621697],\n",
              "       [0.12516859, 0.3878611 , 0.04478889, 0.3438581 , 0.09832323],\n",
              "       [0.054471  , 0.2428969 , 0.5701923 , 0.09047921, 0.04196063],\n",
              "       ...,\n",
              "       [0.00787386, 0.01267407, 0.21849519, 0.01833189, 0.742625  ],\n",
              "       [0.01226825, 0.10055239, 0.21885613, 0.03358892, 0.63473433],\n",
              "       [0.3872196 , 0.34191394, 0.1420926 , 0.08443779, 0.04433604]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 100
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t2LvtfOvdeJ7",
        "outputId": "c6f5963d-29df-429a-c767-8ccb5a296d50"
      },
      "source": [
        "# Convert predictions to classes\n",
        "model_3_preds = tf.argmax(model_3_pred_probs, axis=1)\n",
        "model_3_preds"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(30212,), dtype=int64, numpy=array([3, 1, 2, ..., 4, 4, 0])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 101
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QP0_u8w-dh8Q",
        "outputId": "9759fc2b-a533-4e4f-c1e3-493501c4ba6b"
      },
      "source": [
        "# Calculate Conv1D char only model results\n",
        "model_3_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                        y_pred=model_3_preds)\n",
        "model_3_results"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 62.67377201112141,\n",
              " 'f1': 0.6014980783743493,\n",
              " 'precision': 0.6221765159955959,\n",
              " 'recall': 0.6267377201112141}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 102
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hs03Xb7K7kKt"
      },
      "source": [
        "## Model 4: Combining pretrained token embeddings + character embeddings (hybrid embedding layer)\n",
        "\n",
        "To start replicating (or getting close to replicating) the model in Figure 1, we're going to go through the following steps:\n",
        "\n",
        "1. Create a token-level model (similar to `model_1`)\n",
        "2. Create a character-level model (similar to `model_3` with a slight modification to reflect the paper)\n",
        "3. Combine (using `layers.Concatenate`) the outputs of 1 and 2\n",
        "4. Build a series of output layers on top of 3\n",
        "5. Construct a model which takes token and character-level sequences as input and produces sequence label probabilities as output"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AgTvcwOt8_mZ"
      },
      "source": [
        "# 1. Setup token inputs/model\n",
        "token_inputs = layers.Input(shape=[], dtype=tf.string, name=\"token_input\")\n",
        "token_embeddings = tf_hub_embedding_layer(token_inputs)\n",
        "token_output = layers.Dense(128, activation=\"relu\")(token_embeddings)\n",
        "token_model = tf.keras.Model(inputs=token_inputs,\n",
        "                             outputs=token_output)\n",
        "\n",
        "# 2. Setup char inputs/model\n",
        "char_inputs = layers.Input(shape=(1,), dtype=tf.string, name=\"char_input\")\n",
        "char_vectors = char_vectorizer(char_inputs)\n",
        "char_embeddings = char_embed(char_vectors)\n",
        "char_bi_lstm = layers.Bidirectional(layers.LSTM(25))(char_embeddings) # bi-LSTM shown in Figure 1 of https://arxiv.org/pdf/1612.05251.pdf\n",
        "char_model = tf.keras.Model(inputs=char_inputs,\n",
        "                            outputs=char_bi_lstm)\n",
        "\n",
        "# 3. Concatenate token and char inputs (create hybrid token embedding)\n",
        "token_char_concat = layers.Concatenate(name=\"token_char_hybrid\")([token_model.output, \n",
        "                                                                  char_model.output])\n",
        "\n",
        "# 4. Create output layers - addition of dropout discussed in 4.2 of https://arxiv.org/pdf/1612.05251.pdf\n",
        "combined_dropout = layers.Dropout(0.5)(token_char_concat)\n",
        "combined_dense = layers.Dense(200, activation=\"relu\")(combined_dropout) # slightly different to Figure 1 due to different shapes of token/char embedding layers\n",
        "final_dropout = layers.Dropout(0.5)(combined_dense)\n",
        "output_layer = layers.Dense(num_classes, activation=\"softmax\")(final_dropout)\n",
        "\n",
        "# 5. Construct model with char and token inputs\n",
        "model_4 = tf.keras.Model(inputs=[token_model.input, char_model.input],\n",
        "                         outputs=output_layer,\n",
        "                         name=\"model_4_token_and_char_embeddings\")"
      ],
      "execution_count": 68,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "sk9S9v1B9QUz",
        "outputId": "9e94b5fc-788a-41b5-94ad-811ac583dc19"
      },
      "source": [
        "# Get summary of token and character model\n",
        "model_4.summary()"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_4_token_and_char_embeddings\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "char_input (InputLayer)         [(None, 1)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "token_input (InputLayer)        [(None,)]            0                                            \n",
            "__________________________________________________________________________________________________\n",
            "char_vectorizer (TextVectorizat (None, 290)          0           char_input[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "universal_sentence_encoder (Ker (None, 512)          256797824   token_input[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "char_embed (Embedding)          (None, 290, 25)      1750        char_vectorizer[1][0]            \n",
            "__________________________________________________________________________________________________\n",
            "dense_4 (Dense)                 (None, 128)          65664       universal_sentence_encoder[1][0] \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional (Bidirectional)   (None, 50)           10200       char_embed[1][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "token_char_hybrid (Concatenate) (None, 178)          0           dense_4[0][0]                    \n",
            "                                                                 bidirectional[0][0]              \n",
            "__________________________________________________________________________________________________\n",
            "dropout (Dropout)               (None, 178)          0           token_char_hybrid[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "dense_5 (Dense)                 (None, 200)          35800       dropout[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dropout_1 (Dropout)             (None, 200)          0           dense_5[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "dense_6 (Dense)                 (None, 5)            1005        dropout_1[0][0]                  \n",
            "==================================================================================================\n",
            "Total params: 256,912,243\n",
            "Trainable params: 114,419\n",
            "Non-trainable params: 256,797,824\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 856
        },
        "id": "ayc8_xZk9rJU",
        "outputId": "3b8c70e5-c261-424d-d079-2d66c18f7395"
      },
      "source": [
        "# Plot hybrid token and character model\n",
        "from tensorflow.keras.utils import plot_model\n",
        "plot_model(model_4)"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAm8AAANHCAYAAABgvrpnAAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdeVxU5f4H8M8BBoZBGCARVEAUNVzA3MpMK7RbqTdLASW10jKXFrXUzCWz0sz0ijeXlqt5f+m9CmhXTVv13lKvS2oaaoprLgli7LLIAN/fH14nR7ZhmOFw8PN+veYPzznznO9zloePZ86cUUREQERERESa4KR2AURERERkPYY3IiIiIg1heCMiIiLSEIY3IiIiIg1xUbsAR9m9ezcWLlyodhlEddK9996LV199Ve0yiIjIBvX2ytuFCxewbt06tcsgqnP27NmD3bt3q10GERHZqN5eebshMTFR7RKI6pSYmBi1SyAiohqot1feiIiIiOojhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4s9KIESOg1+uhKAoKCwtVreXLL7+E0WjEF198oWodNbFnzx60adMGTk5OUBQF/v7+mD17ttplWVi/fj1atGgBRVGgKAoCAgIwbNgwtcsiIqLbnIvaBWjFypUr0bRpU8yZM0ftUiAiapdQY926dcOxY8fw6KOP4ptvvkFycjK8vb3VLstCVFQUoqKi0LJlS/z+++9ITU1VuyQiIiJeedOifv36ITs7G4899pjapaCgoADdu3dXuwy7qE99ISKi+ovhzQaKoqhdQp2xYsUKpKWlqV2GXdSnvhARUf3F8HaLVatWoUuXLtDr9fDw8EBISAjeeecd83wnJyds2bIFffr0gdFoROPGjfHpp59atLFjxw60bdsWRqMRer0e4eHh+OabbwAA77//PgwGAzw9PZGWloaJEyeiadOmSE5Otqq+nTt3Ijg4GIqiYMmSJQCAZcuWwcPDAwaDARs3bkSfPn3g5eWFwMBArFmzxvzeDz74AHq9Ho0aNcKYMWPQuHFj6PV6dO/eHXv37jUvN27cOLi6uiIgIMA87cUXX4SHhwcURcHvv/8OAJgwYQImTpyI06dPQ1EUtGzZEgDw9ddfw8vLy6aPmOtaX6qrsn0/cuRI8/1zoaGhOHjwIIDr91MaDAYYjUZs2rQJAFBSUoKZM2ciODgY7u7uiIiIQHx8PICaH0NERKRxUk/Fx8dLdbsXFxcnAGTu3LmSnp4uGRkZ8vHHH8vQoUNFRGT69OkCQLZt2yZZWVmSkZEhffv2FTc3N8nLyzO3k5iYKLNmzZKMjAxJT0+Xbt26yR133GGef6Od8ePHy+LFi2XgwIFy7Ngxq+u8cOGCAJDFixeXaXPbtm2SnZ0taWlp0rNnT/Hw8JCioiLzcqNHjxYPDw/55ZdfpLCwUI4ePSpdu3YVT09POX/+vHm5oUOHir+/v8V658+fLwDkypUr5mlRUVESGhpqsdzmzZvF09NT3n777Sr78sgjjwgAyczMrJN9EREJDQ0Vo9FYZV9Eqt73UVFR4uzsLL/99pvF+4YMGSKbNm0y/3vSpEni5uYm69atk8zMTJk2bZo4OTnJvn37LLaRLcdQdHS0REdHW7UsERHVPbzy9j8mkwlvvfUWIiMj8frrr8PX1xc+Pj547rnn0LVrV4tlu3fvDqPRCB8fH8TGxuLatWs4e/aseX50dDTefPNN+Pj4wNfXF/3790d6ejquXLli0c57772Hl156CevXr0dYWJhd+tG9e3d4eXnBz88PsbGxyMvLw/nz5y2WcXFxQZs2beDm5oa2bdti2bJlyM3NxcqVK+1SQ79+/ZCTk4M33nijRu3Uhb5UV1X7fuzYsSgpKbGoLycnB/v27UPfvn0BAIWFhVi2bBkGDBiAqKgoeHt7Y8aMGdDpdGX65YhjiIiI6jaGt/9JSkpCVlYWHnnkEYvpzs7OGD9+fIXv0+l0AK6Hv6qWKSkpsUOl1nN1dQVQeW0A0KVLFxgMBhw/frw2yrKJVvty677v1asXWrdujU8//dT8reG1a9ciNjYWzs7OAIDk5GTk5+ejffv25nbc3d0REBBQZ/pFRETqYXj7n5ycHACwy+MqtmzZggcffBB+fn5wc3PDa6+9VuM2Hc3Nza3MlUGtUrMvVe17RVEwZswYnDlzBtu2bQMAfPbZZ3juuefMy+Tl5QEAZsyYYb5HTlEUnDt3Dvn5+bXXGSIiqpMY3v6nSZMmAGC+gd1W58+fx4ABAxAQEIC9e/ciOzsb8+bNs0eJDmMymZCVlYXAwEC1S6mx2u7L9u3bERcXB8D6fT98+HDo9XosX74cycnJ8PLyQrNmzczz/fz8AABxcXEQEYvX7t27a6VfRERUdzG8/U9ISAh8fX3x7bff1qidw4cPw2Qy4YUXXkCLFi3Mv8pQl33//fcQEXTr1s08zcXFpcqPKOui2u7LgQMH4OHhAcD6fe/j44PBgwdjw4YNWLBgAZ5//nmL+UFBQdDr9Th06JBDaiYiIm1jePsfNzc3TJs2Ddu3b8e4cePw22+/obS0FLm5ufjll1+sbic4OBgAsHXrVhQWFuLkyZMWj66oC0pLS5GZmYni4mIkJSVhwoQJCA4OxvDhw83LtGzZEhkZGdiwYQNMJhOuXLmCc+fOlWnL19cXly5dwq+//orc3FyYTCZ89dVXNj8qpK71pSImkwmXL1/G999/bw5v1dn3Y8eOxbVr17B58+YyD1vW6/UYMWIE1qxZg2XLliEnJwclJSW4ePEiUlJSqruJiIiovlHxm64OZcujQkRElixZIuHh4aLX60Wv10vHjh1l6dKlMm/ePHF3dxcA0qpVKzl9+rSsXr1afHx8BIAEBgbKkSNHRERkypQp4uvrK97e3hITEyNLliwRABIaGiovvfSSuZ2goCBZtWpVtepbvHixBAQECAAxGAzSv39/Wbp0qRgMBovaPvnkE/Hy8hIA0qxZMzlx4oSIXH+8hk6nk6ZNm4qLi4t4eXnJE088IadPn7ZYT3p6ukRGRoper5fmzZvLyy+/LJMnTxYA0rJlS/OjOH766Sdp1qyZuLu7S48ePSQ1NVW+/PJL8fT0lNmzZ1fYjz179ki7du3EyclJAEhAQIDMmTOnTvXlww8/lNDQUAFQ6evzzz83r6uyfX/z40tERDp27ChTp04td/tcu3ZNpkyZIsHBweLi4iJ+fn4SFRUlR48etTgWbTmG+KgQIiJtU0TqwQ9lliMhIQGDBw+uF78Dak9jxoxBYmIi0tPT1S6lxrTel379+mHJkiVo3rx5ra43JiYGAJCYmFir6yUiIvvgx6a3odp+ZIkjaakvN38Mm5SUBL1eX+vBjYiItI/hrQ44fvy4xSMhKnrFxsaqXSrVwJQpU3Dy5EmcOHECI0aMsPjZNSIiImsxvNUBYWFhZR4JUd5r7dq1NVrPtGnTsHLlSmRnZ6N58+ZYt26dnXpQ+7TYF4PBgLCwMDz00EOYNWsW2rZtq3ZJRESkQbznjeg2w3veiIi0jVfeiIiIiDSE4Y2IiIhIQxjeiIiIiDSE4Y2IiIhIQxjeiIiIiDSE4Y2IiIhIQxjeiIiIiDSE4Y2IiIhIQxjeiIiIiDSE4Y2IiIhIQxjeiIiIiDSE4Y2IiIhIQxjeiIiIiDTERe0CHC0mJkbtEojqlD179qBbt25ql0FERDaqt1fegoKCEB0drXYZ9D+bNm3CpUuX1C6DAHTr1g333nuv2mUQEZGNFBERtYug+k9RFMTHx2PQoEFql0JERKRp9fbKGxEREVF9xPBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQaooiIqF0E1S9PPfUUDh06ZDHt119/hZ+fHzw8PMzTdDodvvjiCzRt2rS2SyQiItIsF7ULoPrnzjvvxOrVq8tMv3r1qsW/w8LCGNyIiIiqiR+bkt09+eSTUBSl0mV0Oh2GDx9eOwURERHVI/zYlByic+fOOHToEEpLS8udrygKzpw5g5CQkNotjIiISON45Y0c4umnn4aTU/mHl6IouPvuuxnciIiIbMDwRg4xePDgCq+6OTk54emnn67lioiIiOoHhjdyiICAAPTs2RPOzs7lzo+KiqrlioiIiOoHhjdymKeeeqrMNCcnJ0RGRsLf31+FioiIiLSP4Y0cJiYmptz73soLdURERGQdhjdyGC8vLzz66KNwcfnjcYLOzs54/PHHVayKiIhI2xjeyKGGDRuGkpISAICLiwv69+8Po9GoclVERETaxfBGDtW/f3+4u7sDAEpKSjB06FCVKyIiItI2hjdyKL1ej4EDBwIADAYD+vTpo3JFRERE2lbt3za9ePEidu3a5YhaqJ4KCgoCAHTt2hWbNm1SuRrSkqCgINx7771ql1FjCQkJapdAVG8MGjRI7RJUV+2fx0pISMDgwYMdVQ8RkVl0dDQSExPVLqPGqvqtXyKyHn/V04Yrbzdw41F1zJo1CzNmzLD45ilRZWJiYtQuwa7i4+N5xYCoBnjx6A+8541qBYMbERGRfTC8Ua1gcCMiIrIPhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQhjciIiIiDWF4IyIiItIQ1cPbiBEjoNfroSgKCgsL1S5Hs7788ksYjUZ88cUXapcCAIiNjYWiKFa9Nm/e7LA6Ro8eDQ8PDyiKAp1Ohw4dOuDYsWMWy3z66acIDg6Goijw9/fH3//+d4fVY6va2r917Tii6pk7dy6MRiMURcGhQ4fULsdCfTi29uzZgzZt2sDJyck8XsyePVvtsiysX78eLVq0MI+vAQEBGDZsmNplkZ2pHt5WrlyJSZMmqV2G5omI2iWU8e233yIrKwsmkwkpKSkAgP79+6OoqAh5eXlIS0vD888/79AaPv74Y+zevRsA0LlzZ/z8889o06aNxTLPPvssduzYgSZNmuDixYsYPny4Q2uyRW3t37p4HJH1pk6dio8//ljtMspVH46tbt264dixY3j44YcBAMnJyZgxY4bKVVmKiorCmTNnEBoaCqPRiNTUVKxevVrtssjOVA9vt6OCggJ0797drm3269cP2dnZeOyxx+zarq0URcF9990Ho9EIFxcXi+k6nQ4GgwF+fn7o3LmzXddb3raNiIhAjx49sHfvXvz000/lvu+jjz7Cs88+C51O55AaasoR+7e8OuvacXS7csQxpLa6dGzVp+1bn/pC1qtT4U1RFLVLqBUrVqxAWlqa2mU41Jo1a2AwGKpcbvTo0fjzn/9st/VWtG1feuklAMDSpUvLzCsqKsJnn32G0aNHO7SGukYrdd6OuG8cqz5t3/rUF7JerYW3VatWoUuXLtDr9fDw8EBISAjeeeedPwpxcsKWLVvQp08fGI1GNG7cGJ9++qlFGzt27EDbtm1hNBqh1+sRHh6Ob775BgDw/vvvw2AwwNPTE2lpaZg4cSKaNm2K5ORkq+pr06YNFEWBk5MTOnfujPz8fADAa6+9Zl7fjXuhSkpKMHPmTAQHB8Pd3R0RERGIj4+3qr8TJkzAxIkTcfr0aSiKgpYtWwK4/pHCwoUL0aZNG7i5ucHHxwdPPPEEjh8/bm6zoj6uWLHCfM/WkiVLAACnTp2q8B6z7777rsp+VLY9v/76a3h5eWHOnDlWbVtrVFbL3//+dzRo0ACKosDHxwcbNmzA/v370axZMzg7O2PIkCEAUOG2Ba5/lNCkSROsXbsWWVlZFutet24d7rnnHgQGBtar/VvZ+VJenTt37iyzHmtrX7ZsGTw8PGAwGLBx40b06dMHXl5eCAwMxJo1a2pwZNx+anIMlefy5csICQmBi4sLHn30UfP0yo5ze+/P8o4ta9fxwQcfQK/Xo1GjRhgzZgwaN24MvV6P7t27Y+/eveblxo0bB1dXVwQEBJinvfjii+Z7Xn///fdKt29NxrW61pfqqmysGDlypHlsCQ0NxcGDBwFcv1/dYDDAaDRi06ZNAGz/m0I2kGqKj4+X6r4tLi5OAMjcuXMlPT1dMjIy5OOPP5ahQ4eKiMj06dMFgGzbtk2ysrIkIyND+vbtK25ubpKXl2duJzExUWbNmiUZGRmSnp4u3bp1kzvuuMM8/0Y748ePl8WLF8vAgQPl2LFjVtVYXFwsISEhEhwcLMXFxRbzXnnlFYmLizP/e9KkSeLm5ibr1q2TzMxMmTZtmjg5Ocm+ffus6m9UVJSEhoZarGPmzJni6uoqq1atkqysLElKSpJOnTpJw4YNJTU1tco+XrhwQQDI4sWLRUTk5MmT8vrrr5u3X0pKivj4+Ej37t2lpKTEqn5UtK7NmzeLp6envP3221Zt2xvrByCPP/54ufOrquWXX34Rg8EgzzzzjPk9U6dOleXLl1u0U962vWHWrFkCQBYuXGgxvUePHrJ161ara9HK/q3qfCmvzlvXY0vt27Ztk+zsbElLS5OePXuKh4eHFBUVlbtPKhMdHS3R0dHVfl9dBEDi4+OtXr4mx9CaNWsEgBw8eFBERIqKiiQqKko2btxo0Z6157+99md5x5a16xg9erR4eHjIL7/8IoWFhXL06FHp2rWreHp6yvnz583LDR06VPz9/S3WO3/+fAEgV65cqXT7Vmdce+SRRwSAZGZm1sm+iIiEhoaK0Wissi8i1o0Vzs7O8ttvv1m8b8iQIbJp0ybzv239m2ItW/JHfeXw8FZUVCTe3t4SGRlpMb24uFgWLVokIn/s0IKCAvP8zz77TADIkSNHKmz73XffFQCSlpZWYTvVceOPckJCgnlaXl6eBAcHS3Z2toiIFBQUiMFgkNjYWPMy+fn54ubmJi+88IJV/b31ZMvPz5cGDRpYtCki8uOPPwoAi8Gkoj6WNzDebMCAAaLX6+X48eNW9aOyddmisvBmTS0iIh9//LEAkNWrV8s///lPefXVV8u0VVl4S0lJEZ1OJ61bt5bS0lIREUlKSpKwsDCra9HK/i3PreeLNeGtprUvXbpUAMipU6cqrKsiDG+2HUM3hzeTySRPPvmkfPXVVxbvs/X8r8n+rCy8VbWO0aNHlwki+/btEwDy1ltvmafVNPBYq7LwVlf6Up3wdqtbx4qtW7cKAJk9e7Z5mezsbGnVqpX5Ykdt/E1hePuDwz82TUpKQlZWFh555BGL6c7Ozhg/fnyF77tx47jJZKpymZKSEjtUev3ysNFoxKJFi8zTVq9ejSeeeAJeXl4Arn+7KD8/H+3btzcv4+7ujoCAABw/ftym/h49ehRXr15Fly5dLKZ37doVrq6uFpfTbZGQkIB//etfeOutt3DnnXda1Y/aZG0to0aNQnR0NMaMGYOEhAS8//771VpPQEAAoqKicOLECWzduhUA8OGHH2Ls2LFW16KV/VseW86Xmtbu6uoKoPLzmKpmy34oKSnBkCFD0KhRI4uPSwHbz//a2J/WrqNLly4wGAy1Pl5Vh1b7cutY0atXL7Ru3Rqffvqp+VvDa9euRWxsLJydnQHUrb8ptwOHh7ecnBwAgLe3d43b2rJlCx588EH4+fnBzc0Nr732Wo3bvFmDBg0watQo7Nq1Cz/++COA63/cx40bZ14mLy8PADBjxgyL+4zOnTuH/Px8m/p74x6sBg0alJnn7e2N3Nxcm/uUnp6Ol19+GV27dsXEiROt7kdtqk4tc+bMwdWrV22+QffGFxeWLVuG3Nxc/Otf/8IzzzxjdS1a2b+Afc4XR9ZO1rNlP7z00ks4efIkPvroI/zyyy8W8+rS+V8Tbm5uuHLlitpl2IWafalqrFAUBWPGjMGZM2ewbds2AMBnn32G5557zrxMfTmmtMLh4a1JkyYAYL7B0lbnz5/HgAEDEBAQgL179yI7Oxvz5s2zR4kWxo0bB51Oh7i4OGzfvh1BQUEIDQ01z/fz8wMAxMXFQa5/7Gx+7d6926b+3ggC5Q3AWVlZ5hvpbTF+/HhkZWVh5cqV5v8hWdOP2mRtLSaTCePHj8fChQuxe/dumx6Oed9996Fjx4744osvMHfuXDz++OMwGo1W16KV/Wuv88WRtZP1bNkPgwYNwnfffQdvb288/fTTKC4uNs+rS+e/rUwmU705Bmu7L9u3b0dcXBwA68eK4cOHQ6/XY/ny5UhOToaXlxeaNWtmnl8fjiktcXh4CwkJga+vL7799tsatXP48GGYTCa88MILaNGihflXGewtMDAQgwYNwrp16/DGG29gwoQJFvODgoKg1+srfHq5Lf1t3749GjRogP3791tM37t3L4qKimx+FtqWLVvwj3/8A2+88QbatWtnnj558uQq+1GbrK3l5ZdfxvPPP49XXnkFr776Kt555x2bBoUXX3wRJSUleO+99/DCCy9Uqxat7F97nS+Oqp2qx5b9EBkZiYYNG+KTTz7BgQMHLP6zU5fOf1t9//33EBF069bNPM3FxUWTH9HXdl8OHDgADw8PANb/bfXx8cHgwYOxYcMGLFiwoMwD1uvDMaUlDg9vbm5umDZtGrZv345x48bht99+Q2lpKXJzc8tcyq9McHAwAGDr1q0oLCzEyZMna3yvUEUmTpyI4uJiZGZmolevXhbz9Ho9RowYgTVr1mDZsmXIyclBSUkJLl68iJSUFKv66+vri0uXLuHXX39Fbm4unJ2dMXHiRHz++edYvXo1cnJycPjwYYwdOxaNGze26fljOTk5GDNmDO666y68/vrrAIDCwkLs378fhw4dqrIflfnqq6/s+qgQa2pZunQpmjZtioEDBwIA3n33XbRt2xZDhw41f5QJlN225Q1+Q4YMga+vL+677z5ERERUqxat7F9rzhdrtpVer7d77VQ1ex5D/fv3x/DhwzFnzhwcOHAAgHXnXF1TWlqKzMxMFBcXIykpCRMmTEBwcLDFL6K0bNkSGRkZ2LBhA0wmE65cuYJz586Vaau8Y9/e45qafamIyWTC5cuX8f3335vDW3X+to4dOxbXrl3D5s2byzxsWYvHlKZV9xsOtn7bY8mSJRIeHi56vV70er107NhRli5dKvPmzRN3d3cBIK1atZLTp0/L6tWrxcfHRwBIYGCg+RunU6ZMEV9fX/H29paYmBhZsmSJAJDQ0FB56aWXzO0EBQXJqlWrql3jzSIjI8s8huKGa9euyZQpUyQ4OFhcXFzEz89PoqKi5OjRo1X2V0Tkp59+kmbNmom7u7v06NFDUlNTpbS0VObPny+tWrUSnU4nPj4+MmDAAElOTja3efO2urmPixcvloCAAAEgBoNB+vfvLwsWLBAA5b769u1bZT8qWpeIyJdffimenp4W3zyqSE5Ojtx///3i6+srAMTJyUlatmwpc+bMsXqbPvbYY6Ioivj6+squXbtE5PrjW5ycnASAGI1G2b9/f4XbtjyTJ0+Wf/7zn/V6/1Z2vpw/f75MnTNmzCizHhGxqvalS5eKwWCwOI8/+eQT8fLyEgDSrFkzOXHiRJXHy81u52+b2noMrV+/3jx2hoSESFpamuTk5EhQUJAAkAYNGshnn30mIpUf5/ben+Udw9VZx+jRo0Wn00nTpk3FxcVFvLy85IknnpDTp09brCc9PV0iIyNFr9dL8+bN5eWXX5bJkycLAGnZsqX5URzlbV9rxrU9e/ZIu3btzGNPQECAzJkzp0715cMPP5TQ0NAKx4cbr88//9y8rqrGipt17NhRpk6dWu72sfVvirX4bdM/KCLV+8G5hIQEDB48uF78Th0R1V0xMTEAgMTERJUrqTlFURAfH49BgwapXYomjRkzBomJiUhPT1e7lBrTel/69euHJUuWoHnz5rW+buaPP9Spn8ciIiIqj70eCVUXaKkvN38Mm5SUBL1er0pwI0v1OrwdP368wp8QuvkVGxurdqlERJrC8fX2MGXKFJw8eRInTpzAiBEjLH7WktTjonYBjhQWFsbLq0REDlBb4+u0adOwcuVKFBUVoXnz5pg/fz6io6Mdvl5H0GJfDAYDwsLC0LRpUyxduhRt27ZVuyQCwHveiKhO4j1vRHQz5o8/1OuPTYmIiIjqG4Y3IiIiIg1heCMiIiLSEIY3IiIiIg1heCMiIiLSEIY3IiIiIg1heCMiIiLSEIY3IiIiIg1heCMiIiLSEIY3IiIiIg1heCMiIiLSEIY3IiIiIg1heCMiIiLSEBdb35iQkGDPOoiILFy8eBGBgYFql2E3u3fvVrsEIk3jOfQHRUSkOm9ISEjA4MGDHVUPEZFZdHQ0EhMT1S6jxhRFUbsEonqjmrGlXqp2eCOyhaIoiI+Px6BBg9QuhYiojBtjEz9VIi3gPW9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGsLwRkRERKQhDG9EREREGuKidgFU/3zyySfIzMwsM33jxo04e/asxbThw4fD39+/tkojIsIPP/yAPXv2WEw7fvw4AGDevHkW07t164YHHnig1mojsiBvLR8AACAASURBVIYiIqJ2EVS/jB49Gp988gnc3NzM00QEiqKY/11cXAyj0YjU1FTodDo1yiSi29R3332Hhx9+GDqdDk5O5X8AVVpaCpPJhG+//RZ/+tOfarlCosoxvJHdff/994iMjKx0GZ1Oh1GjRmHJkiW1VBUR0XUlJSXw9/dHenp6pcv5+PggLS0NLi78kIrqFt7zRnZ3//33o1GjRpUuYzKZ8OSTT9ZSRUREf3B2dsbQoUPh6upa4TKurq546qmnGNyoTmJ4I7tzcnLCsGHDKh0YGzdujO7du9diVUREf3jyySdRVFRU4fyioiL+B5PqLIY3cojKBkadToenn37a4h44IqLa1K1bNwQHB1c4PzAwEPfcc08tVkRkPYY3coguXbqgefPm5c7jR6ZEVBcMGzas3C9Mubq64plnnuF/MKnOYngjh3n66afLHRhbtGiBDh06qFAREdEfhg0bBpPJVGZ6UVERYmNjVaiIyDoMb+Qw5Q2MOp0OI0aMUKkiIqI/tGnTBm3atCkzPSwsDO3bt1ehIiLrMLyRw7Rs2RLh4eEWHz2YTCYMHjxYxaqIiP5w6ycEOp0OzzzzjIoVEVWN4Y0c6umnn4azszMAQFEUdOzYEa1atVK5KiKi64YMGYLi4mLzv4uLi/mRKdV5DG/kUEOGDEFJSQmA689W4v9oiaguCQ4ORpcuXeDk5ARFUdC1a1eEhISoXRZRpRjeyKGaNGmC7t27Q1EUlJaWIiYmRu2SiIgsPP3003BycoKzszOeeuoptcshqhLDGzncU089BRHB/fffjyZNmqhdDhGRhcGDB0NEICL8DyZpg2hUfHy8AOCLL74c8IqOjnbYuRsdHa16//jiiy++tPIqbzzW/I+2xcfHq10CWeEvf/kLRo8ejQYNGqhdClUhLi7O4evo1q0bXnnlFYevh8haP/zwAxRFwf333692KURmFY3Hmg9vgwYNUrsEskL37t0RGBiodhlkhcTERIevIzAwkOcu1SmPPvooAMDLy0vlSoj+UNF4rPnwRtrA4EZEdRlDG2kJv7BAREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb/8zYsQI6PV6KIqCwsJCtctxiK5du8LZ2Rl33XWX3dseOXIkPD09oSgKDh06VK33rl+/Hi1atICiKBW+QkJC7FJnXdgGFS335Zdfwmg04osvvrB7bWRJS9t6wYIFaNSoERRFwUcffaR2OaSSmoyxtrp1bA4KCsKKFSvM83/44Qc0bdoUiqIgICAAn3zySa3UZU2tAQEBGDZsmGr1OBrD2/+sXLkSkyZNUrsMh9q3bx8iIyMd0vby5cvxt7/9zab3RkVF4cyZMwgNDYXRaISIQERQXFyM/Px8XL58GQaDwS511oVtUNFyIuKIsqgcWtrWkyZNwq5du9Qug1RWkzHWVreOzRcuXMBzzz1nnn///fejb9++GDVqFFJSUjBq1Khara+yWlNTU7F69WrV6nE0F7ULoNqnKIraJVjF2dkZ7u7ucHd3R+vWre3adl3cBv369UN2drbaZdwWuK3tq6CgAL1792bIvI2UlpZi5MiR0Ov1WLp0aZ0cU+szXnkrR30/CHU6nUPadeR227Bhg13bU3sb1MYxJiJITExU9aMMuj2sWLECaWlpapdR79WVv02lpaV49tlnYTAYsGzZsjpT1+3ktgtvq1atQpcuXaDX6+Hh4YGQkBC888475vlOTk7YsmUL+vTpA6PRiMaNG+PTTz+1aGPHjh1o27YtjEYj9Ho9wsPD8c033wAA3n//fRgMBnh6eiItLQ0TJ05E06ZNkZycbHWNJSUlmDlzJoKDg+Hu7o6IiAjEx8cDABYtWgQPDw84OTmhc+fO8Pf3h06ng4eHBzp16oSePXsiKCgIer0e3t7eeO2118q0f+rUKYSFhcHDwwPu7u7o2bMndu7caXUNwPVgMH/+fNx5551wc3OD0WjE5MmTy6zr66+/hpeXF+bMmWN1/6uitW1gzXI7d+5EcHAwFEXBkiVLAADLli2Dh4cHDAYDNm7ciD59+sDLywuBgYFYs2ZNmVrfffdd3HnnnXB3d0fDhg3RvHlzvPvuuxg0aJDN27ouGDduHFxdXREQEGCe9uKLL8LDwwOKouD3338HYP32Km9bt2nTBoqimI+p/Px8AMBrr71mPs///ve/A6j8uKjs/P/hhx9w9913w2AwwMvLC+Hh4cjJyQFQ+ZhSU5Wtt7K+WLs9J0yYgIkTJ+L06dNQFAUtW7a0W9s3VDZuV3WeWqu26hURLFy4EG3atIGbmxt8fHzwxBNP4Pjx4xZtWDu+2Ho82jo2l5aWYvjw4TAajebzp7rbs7K6qjoXKjueq6uydY0cOdJ8/1xoaCgOHjwI4Pr98QaDAUajEZs2bapRX2tENCo+Pl6qW35cXJwAkLlz50p6erpkZGTIxx9/LEOHDhURkenTpwsA2bZtm2RlZUlGRob07dtX3NzcJC8vz9xOYmKizJo1SzIyMiQ9PV26desmd9xxh3n+jXbGjx8vixcvloEDB8qxY8esrnPSpEni5uYm69atk8zMTJk2bZo4OTnJvn37RETkzTffFACyd+9eycvLk99//10effRRASBbtmyRK1euSF5enowbN04AyKFDh8xt9+7dW1q0aCFnz54Vk8kkR44ckXvuuUf0er2cOHHC6hqmT58uiqLIX/7yF8nMzJT8/HxZunSpAJCDBw+a29m8ebN4enrK22+/XWW/Q0NDxWg0WkwbP368HD58uMyyWtoG1i534cIFASCLFy+2eO+NYzI7O1vS0tKkZ8+e4uHhIUVFRebl5syZI87OzrJx40bJz8+XAwcOiL+/vzz44INVbvdbRUdHS3R0dLXf58j2hw4dKv7+/hbT5s+fLwDkypUr5mnWbq9bt3VxcbGEhIRIcHCwFBcXW6znlVdekbi4OPO/rTkubj3/9+/fL15eXjJv3jwpKCiQ1NRUGThwoLn2qsaUkydPCgD58MMPq7Xdrl69Wul6re1LVdszKipKQkNDLdZtr7arGrerWo+1aqvemTNniqurq6xatUqysrIkKSlJOnXqJA0bNpTU1FRzO9aOG7Ycj8eOHbNpbC4uLpahQ4eKTqeT5ORku2zPW+uq7Fyo6ni+uVZrVHXeRUVFibOzs/z2228W7xsyZIhs2rSpxn21RkXj5W0T3oqKisTb21siIyMtphcXF8uiRYtE5I8NXFBQYJ7/2WefCQA5cuRIhW2/++67AkDS0tIqbMdaBQUFYjAYJDY21jwtPz9f3Nzc5IUXXhCRP4JLbm6ueZn/+7//EwAWQefHH38UALJ27VrztN69e0uHDh0s1pmUlCQAZNKkSVbVkJ+fLwaDQf70pz9ZtLNmzZoyA0t1hIaGCoAyr8rCW13fBtXZVpWFt5uPpRsD+KlTp8zTunbtKnfffbfFOkaNGiVOTk5y7dq1MtuvMvUhvFW1vcrb1jf+6CYkJJin5eXlSXBwsGRnZ4uIdedneTUcOXJEAMjmzZut6u+tY4qt4a2y9dral/K2563hzV5tVzVuW7Mea9RWvfn5+dKgQQOL9Yj8MU7dCFLWjhu21l1doaGh4unpKU8++aR06tRJAEi7du3k6tWr5S5vz7puPhesOY+qE94qW5eIyNatWwWAzJ4927xMdna2tGrVyvyfPEfvg4rGy9vmY9OkpCRkZWXhkUcesZju7OyM8ePHV/i+G/dGmUymKpcpKSmpcZ3JycnIz89H+/btzdPc3d0REBBQ5rL6zVxdXQEAxcXFZeqqrHYACA8Ph9FoRFJSklU1nDp1Cvn5+ejdu3f1O1iFm79tKiKV7ptb1cVt4IhtdaOfN/epsLCwzDcoS0pKoNPp4OzsbLd1a1F526s8I0eOhNFoxKJFi8zTVq9ejSeeeAJeXl4AbD8/W7RogUaNGmHYsGGYNWsWfv3110prsdeYUtl6azrWVLY97dV2VeO2retRq96jR4/i6tWr6NKli8X8rl27wtXVFXv37gVg/bhhr/5bIz8/Hw888AAOHDiAAQMG4OjRoxg5cqTD67r5XKjueVRdt553vXr1QuvWrfHpp5+ax9e1a9ciNjbWPK7W5j642W0T3m58Ju7t7V3jtrZs2YIHH3wQfn5+cHNzK/eeKlvl5eUBAGbMmGHxnLNz586Z78NxBJ1OZx6Aqqrh4sWLAAA/Pz+H1XPDokWLLE4KR3LENqitbdW3b18cOHAAGzduREFBAfbv348NGzbgz3/+820f3qzVoEEDjBo1Crt27cKPP/4IAPjwww8xbtw48zK2np/u7u7497//jR49emDOnDlo0aIFYmNjUVBQAMBxY0pl63XkWGOvtqsat+21ntqqNysrC8D1Y+1W3t7eyM3NBWD9uFGbfy8aNGiA0aNHA7j+aK0WLVpg7dq1iIuLs2tdlZ0LVZ1H1VXVeacoCsaMGYMzZ85g27ZtAIDPPvvM4nEpav3Nvm3CW5MmTQDAfHOzrc6fP48BAwYgICAAe/fuRXZ2NubNm2ePEgH8cbLGxcVZXIESEezevdtu67lZcXExMjIyEBwcbFUNer0eAHDt2jWH1KMGR22D2tpWs2bNQq9evTB8+HB4eXlh4MCBGDRoUK0/F0rrxo0bB51Oh7i4OGzfvh1BQUEIDQ01z6/J+dmuXTt88cUXuHTpEqZMmYL4+HgsWLDA4WNKRet15Fhjr7arGrfttZ7aqvdGqLsR0m6WlZWFwMBAANaPG2r8vQCuf0KSmJhoDjzbt2+3S13WnAsVHc/W2L59uzlsWnveDR8+HHq9HsuXL0dycjK8vLzQrFmzGve1pm6b8BYSEgJfX198++23NWrn8OHDMJlMeOGFF9CiRQvzrzLYy41vSdbWE7QB4D//+Q9KS0vRqVMnq2po3749nJyc8MMPP9RajSkpKRgxYoTD2nfUNqitbXX06FGcPn0aV65cgclkwvnz57Fs2TL4+Pg4dL21xcXFpcqPPe0hMDAQgwYNwrp16/DGG29gwoQJFvNtPT8vXbqEX375BcD1wX7u3Lno1KkTfvnlF4eOKZWt15Fjjb3armrcttd6aqve9u3bo0GDBti/f7/F9L1796KoqAidO3c2L2fNuKHG34sbOnXqhLi4OBQXF2PQoEG4dOlSjeuq6lyo7Hi2xoEDB+Dh4WHVum7w8fHB4MGDsWHDBixYsADPP/+8xXy19sFtE97c3Nwwbdo0bN++HePGjcNvv/2G0tJS5ObmWr3jAZivzGzduhWFhYU4efKk+T4Fe9Dr9RgxYgTWrFmDZcuWIScnByUlJbh48SJSUlLsso6ioiJkZ2ejuLgYP/30E8aNG4dmzZph+PDhVtXg5+eHqKgorFu3DitWrEBOTg6SkpLKfZ7YV199VaNHhYgICgoKsH79evN9R/ZQW9ugOtuqJl566SUEBwfj6tWrdm23rmjZsiUyMjKwYcMGmEwmXLlyBefOnXPIuiZOnIji4mJkZmaiV69eFvNsPT8vXbqEMWPG4Pjx4ygqKsLBgwdx7tw5dOvWzaFjSmXrtedY4+vri0uXLuHXX39Fbm4unJ2d7dJ2VeO2vfpgr3asqXfixIn4/PPPsXr1auTk5ODw4cMYO3YsGjdubP5Y0tpxoyZ113RsBoCxY8fiySefxOXLlxETE2P+D5atdVV1LlR2PFfGZDLh8uXL+P77783hrTrn3dixY3Ht2jVs3rwZjz32mMW82vibXa5qf/WhjrDlUSEiIkuWLJHw8HDR6/Wi1+ulY8eOsnTpUpk3b564u7sLAGnVqpWcPn1aVq9eLT4+PgJAAgMDzd84nTJlivj6+oq3t7fExMTIkiVLBICEhobKSy+9ZG4nKChIVq1aVe0ar127JlOmTJHg4GBxcXERPz8/iYqKkqNHj8qiRYvEYDAIAAkJCZEdO3bIe++9J0ajUQCIv7+//OMf/5C1a9eKv7+/ABAfHx9Zs2aNiIisXLlSIiMjpVGjRuLi4iJ33HGHPPnkk3Lu3DmraxARyc3NlZEjR8odd9whDRo0kB49esjMmTPN2+rnn38WEZEvv/xSPD09Lb6tc6vPP/+8wm+a3vyaMWOGiIjmtoE1yy1evFgCAgIEgBgMBunfv78sXbrU3M8bx+Qnn3wiXl5eAkCaNWtmfrTJv//9b7njjjsstpdOp5M2bdrI+vXrq3X81cVvm6anp0tkZKTo9Xpp3ry5vPzyyzJ58mQBIC1btpTz589bvb3K29a3ioyMlOXLl5dbS2XHxc3jyM3n/6+//irdu3cXHx8fcXZ2liZNmsj06dPN31irbEyZMGGC+Tj28PCQgQMHWr3dqlpvZX2pzvH3008/SbNmzcTd3V169OghqampdmtbpOJxu6o+VEdt1VtaWirz58+XVq1aiU6nEx8fHxkwYECZR29YO77YcjyK2DY2BwYGyrRp08rUeeeddwoAadSokaxYsaJGdVV2LuzYsaPC49navyOff/65Ves6f/68RT87duwoU6dOrfaxU1lfrVHReKmIaOhH/m6SkJCAwYMHa+o3CokcZdmyZTh58qTFzcNFRUV4/fXXsWzZMmRmZsLd3d2qtmJiYgAAiYmJDqnV0e0TEdlbv379sGTJEjRv3rxW11vReMnfNiXSuNTUVIwbN67MPReurq4IDg6GyWSCyWSyOrwREd3uTCaT+dEhSUlJ0Ov1tR7cKnPb3POmpuPHj1t8hbiiV2xsrNqlkga5u7tDp9NhxYoVuHz5MkwmEy5duoTly5dj5syZiI2Ntev9gqQujifW4XaimpgyZQpOnjyJEydOYMSIERY/o1kX8MpbLQgLC+PHu+QwRqMR3377Ld5++220bt0aeXl5aNCgAdq1a4f33nsPo0aNUrtEsiOOJ9bhdqKaMBgMCAsLQ9OmTbF06VK0bdtW7ZIsMLwR1QM9e/bEd999p3YZRET1wuzZszF79my1y6gQPzYlIiIi0hCGNyIiIiINYXgjIiIi0hCGNyIiIiINYXgjIiIi0hCGNyIiIiINYXgjIiIi0hCGNyIiIiINYXgjIiIi0hCGNyIiIiINYXgjIiIi0hCGNyIiIiINYXgjIiIi0hAXtQuoKUVR1C6BqN6Jjo52aPvr1q3juUtEZIXyxmNFRESFWmrs4sWL2LVrl9plkI1WrFiBc+fO4e2331a7FCpHUFAQ7r33Xoe0vXv3bly4cMEhbRNVR0FBAYYPH46pU6firrvuUrsconKVNx5rNryRtn300UeYMmUKsrKyeAWGiFSRnJyMsLAwHDx4kOGNNIX3vJEqwsPDkZOTg3PnzqldChHdplJTUwEAjRs3VrkSoupheCNVREREQFEUHD58WO1SiOg2lZKSAmdnZzRs2FDtUoiqheGNVOHp6YlmzZohKSlJ7VKI6DaVmpqKRo0awdnZWe1SiKqF4Y1UExERwStvRKSa1NRUfmRKmsTwRqoJDw9neCMi1aSkpCAgIEDtMoiqjeGNVBMeHo4TJ06gsLBQ7VKI6DaUkpLCK2+kSQxvpJqIiAgUFxfj2LFjapdCRLeh1NRUXnkjTWJ4I9W0bt0a7u7u/NICEamCV95IqxjeSDXOzs4ICwvjfW9EVOtMJhMyMjJ45Y00ieGNVMVvnBKRGi5fvozS0lJeeSNNYngjVYWHh/NjUyKqdSkpKQDAK2+kSQxvpKqIiAikpqYiLS1N7VKI6DZy46exGN5IixjeSFXh4eEAgCNHjqhcCRHdTlJSUmA0GmEwGNQuhajaGN5IVQEBAWjUqBE/OiWiWsXHhJCWMbyR6vhLC0RU2/jTWKRlDG+kOn5pgYhqG38ai7SM4Y1UFx4ejqNHj6KkpETtUojoNsErb6RlDG+kuoiICBQUFOD06dNql0JEtwleeSMtY3gj1bVr1w7Ozs786JSIaoWI4PLlywxvpFkMb6Q6d3d3tGzZkl9aIKJakZmZicLCQn5sSprF8EZ1Ar9xSkS15cYDehneSKsY3qhO4DdOiai28KexSOsY3qhOiIiIwJkzZ5Cbm6t2KURUz6WmpkKn08HX11ftUohswvBGdUJERAREBEePHlW7FCKq525809TJiX8CSZt45FKd0Lx5c3h5efG+NyJyOP40FmkdwxvVCYqioG3btgxvRORwfEAvaR3DG9UZERER/NICETkcH9BLWsfwRnUGHxdCRLWBV95I6xjeqM4IDw9HRkYGLl68qHYpRFSP8cobaR3DG9UZERERAMCrb0TkMNeuXUNWVhbDG2kawxvVGT4+PggMDOR9b0TkMKmpqRARfmxKmsbwRnVKREQEr7wRkcPw1xWoPnBRuwCim4WHh+Orr77C2bNncfjwYRw+fBg///wzmjRpgkWLFqldHhFpyPnz5/G3v/0NjRo1QpMmTRAQEICkpCQoigJ/f3+1yyOymSIionYRdPvKysrCkSNHzCFt586dOHXqFK5duwYA0Ov1uHbtGiZNmoT3339f5WqJSEsKCwthNBpRXFyM0tJSi3nu7u4ICAiAv78/QkJC0LVrV7z66qsqVUpUPbzyRqrJyMhASEgIcnNz4eLiAkVRYDKZLJYpLCyEs7Mz2rRpo1KVRKRVer0eXbt2xa5du8rMKygowNmzZ3H27Fns2bMH3bt3V6FCItvwnjdSja+vL6ZNmwYnJycUFxeXCW43lJSUoG3btrVcHRHVB7169YJOp6t0GS8vLzz77LO1VBFRzTG8kapeffVVBAYGVvkD0WFhYbVUERHVJw888ACKiooqnK/T6fDKK6/Aw8OjFqsiqhne80aqW7t2LYYMGYKKDkU/Pz+kpaXVclVEVB/k5+eb73srj5ubGy5cuAA/P79arozIdrzyRqobPHgw7r77bri4lH8LZrt27Wq5IiKqLwwGAzp27FjuPJ1Oh+eff57BjTSH4Y1UpygK/vrXv6KkpKTMPFdXV/MvLxAR2eKhhx6Cq6trmeklJSWYMGGCChUR1QzDG9UJ99xzD2JiYsrcWCwi/KYpEdVIefe96XQ6xMTEIDQ0VKWqiGzHe96ozrhw4QJatWplfsbbDT/88APuv/9+laoiIq27evUqvL29y1zdP3DgADp16qRSVUS245U3qjOCgoLwyiuvlLn3jVfeiKgmGjRoYHH7hYuLCyIjIxncSLMY3qhOmTZtGoxGIxRFAXD9+Uu8mZiIaurm+96Ki4sxbdo0lSsish3DG9Upnp6emDNnjjm88eG8RGQPN+57c3JyQvv27fHQQw+pXRKRzRjeqM4ZOXIkWrduDQDo0KGDytUQUX3Qo0cPODk5obS0FNOnT1e7HKIaKfOFhd27d2PhwoVq1UMEALh8+TJ27NiBDh06oFWrVmqXQ1SlV199Fffee68q6+a4bZ3vvvsORUVF6Nu3r/nqPgH33nsvXn31VbXLoGooc+XtwoULWLdunRq1EJn5+/sjICAAXl5eapdCVKV169bhwoULqq2f47Z1GjVqhDvvvJPB7SZ79uzB7t271S6Dqqn8R9oDSExMrM06iMo4evQovL290bRpU7VLIapUXQkDHLcrt2PHDnTu3BkGg0HtUuqMmJgYtUsgG1QY3ojUxp/FIiJ76tmzp9olENkFv7BAREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCEMb0REREQawvBGREREpCE1Dm9du3aFs7Mz7rrrriqX/fLLL2E0GvHFF19UuMzIkSPh6ekJRVFw6NChar3XkdRe/4IFC9CoUSMoioKPPvqo3GW2bt2KqVOnWrWsI23atAnz5s1DSUmJTe9fv349WrRoAUVRLF4uLi5o2LAhHnroIXz++edl3sfjy3bVOb5u3T8BAQEYNmxYlev4+eefERsbi+bNm8PNzQ0NGzZEhw4dMHv2bPMysbGxZfZ7Ra/NmzeXqeWNN96otIaFCxdCURQ4OTkhLCwM27dvr/HxWl9UdG5oWWFhIcLCwjBjxoxqv7eiccjV1RWNGjXCgw8+iPnz5yMzM9MBlRNVrsbhbd++fYiMjLRqWRGpcpnly5fjb3/7m03vdSS11z9p0iTs2rWrwvlvvvkmPvjgA0ybNq3KZR2tf//+0Ov16N27N7Kysqr9/qioKJw5cwahoaEwGo0QEYgIrly5gvj4ePz222+IiopCfHy8xft4fNmuOsfXrfsnNTUVq1evrrT9w4cPo3v37ggICMB//vMfZGdnY9euXXj00Ufx/fffWyz77bffIisrCyaTCSkpKQCuH1NFRUXIy8tDWloann/+eQCWxwpwff+aTKZyaygpKcEHH3wAAOjVqxeOHz+O+++/v8bHa31R0bmhZdOnT0dycrJN7y1vHCotLUVaWhoSEhLQvHlzTJkyBe3atcP+/fvtXDlR5ez2samiKFUu069fP2RnZ+Oxxx6rdvs1eW91FRQUoHv37qqtv7ree+89rF27FgkJCfD09LSpjfL6XBPjx49Hhw4d0LdvXxQXF9ulTR8fH/Tu3Rt//etfAQAJCQkW83l8OYY9jq8FCxbA29sbixYtQkhICPR6PVq3bo133nkH7u7u5uUURcF9990Ho9EIFxcXi+k6nQ4GgwF+fn7o3LlzmXV07twZqamp2LBhQ7k1rF+/Hk2bNi13niOOV1LXrl27cOTIEbu2qSgKvL298eCDD2LlypVISEjA5cuXzecvUW2xW3jT6XT2asqqIOhIK1asQFpamqo1WOvUqVN444038NZbb0Gv19vcjiP6PGvWLBw6dAiLFi2ya7shISEAYPNVEh5f1rPX8ZWeno7s7GxkZGRYTHd1dbX4qHjNmjUwGAxVtjd69Gj8+c9/tpj2wgsvAAA+/PDDct+zP45pagAAIABJREFUcOFCTJw4scI2HXW8aona54a9FBQUYPLkyQ7fl9HR0Rg+fDjS0tJUuUWFbl92C2+nTp1CWFgYPDw84O7ujp49e2Lnzp3m+Tt37kRwcDAURcGSJUvM00UE8+fPx5133gk3NzcYjUZMnjzZou3y3vv+++/DYDDA09MTaWlpmDhxIpo2bYrk5GSUlJRg5syZCA4Ohru7OyIiIsp8xLZq1Sp06dIFer0eHh4eCAkJwTvvvIMJEyZg4sSJOH36NBRFQcuWLSutfeHChWjTpg3c3Nzg4+ODJ554AsePHzcvs2zZMnh4eMBgMGDjxo3o06cPvLy8EBgYiDVr1ljUtGPHDrRt2xZGoxF6vR7h4eH45ptvKt3uH3zwAUQE/fv3r3If/fDDD7j77rthMBjg5eWF8PBw5OTklNvnRYsWwcPDA05OTujcuTP8/f2h0+ng4eGBTp06oWfPnggKCoJer4e3tzdee+21Muvz8fHBAw88gEWLFpk/Fvz666/h5eWFOXPmVFlvRZKSkgAADzzwgHkajy/1j6/KdO3aFXl5eejVqxf++9//1qitivTq1Qtt2rTBf/7znzIflf33v/9Ffn4+Hn744QrfX97xWp9Zc24AqPR4r87xV9H4U9U6bDF9+nS8+OKL8PPzK3e+PcahG4YPHw4A+Oqrr8zTtLjNSGPkFvHx8VLO5Er17t1bWrRoIWfPnhWTySRHjhyRe+65R/R6vZw4ccK83IULFwSALF682Dxt+vTpoiiK/OUvf5HMzEzJz8+XpUuXCgA5ePBgle8FIOPHj5fFixfLwIED5dixYzJp0iRxc3OTdevWSWZmpkybNk2cnJxk3759IiISFxcnAGTu3LmSnp4uGRkZ8vHHH8vQoUNFRCQqKkpCQ0Mt+lje+mfOnCmurq6yatUqycrKkqSkJOnUqZM0bNhQUlNTy9S5bds2yc7OlrS0NOnZs6d4eHhIUVGRebnExESZNWuWZGRkSHp6unTr1k3uuOMO8/yTJ08KAPnwww/N01q0aCFt27Yts09uXfbq1avi5eUl8+bNk4KCAklNTZWBAwfKlStXKuzzm2++KQBk7969kpeXJ7///rs8+uijAkC2bNkiV65ckby8PBk3bpwAkEOHDpWpY+rUqRb7cvPmzeLp6Slvv/12mWVvFRoaKkaj0fzv/Px8+eqrr6RZs2by8MMPy9WrVy2W5/FVe8dXefunMvn5+dKlSxcBIACkbdu2Mm/ePElPT6/0fSkpKQJAHn/88UqXCw0NlbNnz8pf//pXASATJkywmD9gwABZuXKl5ObmCgDp3bt3ue3cerxaC4DEx8dX6z32ZMu4be25UdXxbs3xV9X4U9U6qmPnzp3Sv39/ERG5cuWKAJDp06dbLFOTcehWOTk5AkCCgoLM07S0zaKjoyU6Orpa7yH12S28dejQwWJaUlKSAJBJkyaZp936Byo/P18MBoP86U9/snjvmjVrqvXHtaCgwDytoKBADAaDxMbGmqfl5/8/e3ceH1V973/8PVknIRtLECRhR6jsCBYhWPihtUhFMImACAVLG8AWrBRxu1yKUEtRuVSxFqXcC/RCAnqRWsBWq9gqIMgSQFktIFIIAiGBhKyf3x99MDVmIQlJTk7yej4e8wdnvt/v+czJd2benG2yLDg42KZMmWK5ubkWFRVlgwYNKrLO/Px8+6//+i8zK9+Xa1ZWloWFhRVZj5nZxx9/bJKKfCiUVOfVD8kjR44U255X/fKXvzRJlpaWZmYlBzKPx2P33HNPsb7fbLtv3z6TZG+99VaJ6yorvGVmZvqW/c///I9Jsr179xZ7zatXry427u9//3uTZMuXLy/1dZamXbt2vi/7rz+6du1q//M//2M5OTlF2jO/am5+mVUsvJmZ5ebm2qJFi6xTp06+v2XTpk3t/fffL7VPRcNbenq6NWjQwBo2bGhZWVlmZnb06FGLiYmxnJyca4a3ys5Xt4W38r43rjXfzco3/8r6/CnPOiryunr37m0nT540s9LDW0WUZ557PB6LiooyM/dtM8KbO1Xbfd66du2qyMhI3yGukhw5ckRZWVkaPHhwla334MGDysrKUpcuXXzLQkJC1KxZMx04cECpqalKT0/XXXfdVaSfv7+/pk2bVu717N+/X5cuXVLv3r2LLO/Tp4+CgoK0bdu2MvsHBQVJUqlXxkn/Po+wtFsYpKWlyczKdY5Q27Zt1bRpUz344IOaPXu2jh07ds0+Jbla99dP6r5aZ0mv5WptZ86cqdT6vn61aV5enk6ePKmf/exnmjp1qrp166avvvqq1L7Mr5qbX+URGBioqVOn6rPPPtPWrVs1fPhwpaWlKTExscputxAZGakHHnhAFy5c0OrVqyVJCxcu1JQpU3zbpCzXO1/dorzvjWvN99J8c/6V9flT2XWU5Mknn9SPf/zjUi9MqQ6XL1+WmSkiIkKS+7YZ3Klab9IbGBhY5pfHyZMnJanU8xIq4/Lly5Kkp59+usi9eY4fP66srCzf+QJRUVHXtZ6rJ8uHhYUVey4qKkqZmZkVHvNPf/qTBg4cqOjoaAUHB5d4HtnXXblyRZIUHBx8zbFDQkL017/+VXFxcZo3b57atm2rUaNGKTs7u8J1VsTVKwmv1no9AgIC1KJFC02YMEHPPfecDh48qGeffbbU9syvoqpzflXUt7/9bf3f//2fJk+erLNnz+q9996rsrGvXrjwyiuvKD09XWvWrNGkSZPK1bcq52ttVt73xrXme3mV9flTVev4+9//rr1792rixInl7lMVDh06JEnq1KmTJHdtM7hXtYW3/Px8nT9/Xi1btiy1zdWr13JycqpsvVc/jBYuXOjbY3P1sWXLFt14442SVOYem/K4+uVc0pdoenq6YmJiKjTeiRMnNGLECDVr1kzbtm3TxYsXNX/+/DL7XP2iKe/NRTt37qw//vGPOnXqlGbOnKnk5GQ999xzFaqzonJzcyWpyO0gqkLXrl0lSZ9++mmpbZhf/1YT8+vrPvjgAy1cuND37/j4+BJvwTF27FhJqtIvnB49eqhv3776+OOPlZSUpMTERDVs2LBcfatrvtY25X1vXGu+V0Rpnz9VtY6lS5fq3XfflZ+fny/MXB173rx58ng81XI/tk2bNkmShgwZIsld2wzuVW3h7b333lNhYaF69epVapsuXbrIz89PmzdvrrL1Xr0CsrQ7hLdu3VqNGjXSn//85+taT5cuXRQWFlbsw2Dbtm3Kzc0t8T5UZdm7d6/y8vI0ZcoUtW3bVl6v95qX7V+9I3557i906tQpX9CJjo7Ws88+q169epUZfqrC1dpuuOGGKh33k08+kSR17Nix1DbMr3+r7vn1TZ988okaNGjg+3dOTk6Jc+3qVaHdunWr8DrKcnXv29q1a/Wzn/2s3P2qa77WNuV9b1xrvpdXWZ8/VbWOZcuWFQsyZ8+elfSvq0/NrNhpCNfr9OnTWrhwoWJiYvTQQw9Jctc2g3tVWXjLzc3VxYsXlZ+fr507d2rq1Klq1aqV7zLqkkRHRys+Pl5r167V0qVLlZGRodTUVC1ZsqTSdXi9Xk2YMEGrVq3Syy+/rIyMDBUUFOjkyZP65z//qeDgYD355JP64IMPNHXqVH355ZcqLCxUZmam743SqFEjnTp1SseOHVNmZmaJh369Xq+mT5+uN954QytXrlRGRob27t2ryZMnq3nz5kpKSqpQ3Vf3UL7zzju6cuWKDh8+fM3zmkJDQ9W2bVvfIZCynDp1SpMmTdKBAweUm5urXbt26fjx4+rbt2+5X3NlXK3t6p6yjRs3VvgS/ezsbBUWFsrMdOrUKS1btkxPP/20mjRpUuYXM/Pr36p7fl2Vl5enM2fO6P333y8S3iRpxIgRSklJUXp6ui5evKg333xTjz/+uO69994qD2/333+/mjRpohEjRqht27bl7vfN+VpXlfe9ca35Xl5lff5U1ToqoqKfQ2amS5cu+T6Hrv7aS//+/eXv769169b5znmrq9sMtcw3r2CozNWmy5Yts0GDBlnTpk0tICDAGjdubKNHj7bjx4/72rz44ovWrFkzk2ShoaG+S7kzMzNt4sSJ1rhxYwsLC7O4uDibNWuWSbKYmBjbs2dPiX3nz59vISEhvku0V6xY4VtXTk6OzZw501q2bGkBAQEWHR1t8fHxtn//fl+bl156ybp27Wper9e8Xq/17NnTFi9ebGZmO3futFatWllISIjFxcXZ008/XWLthYWFtmDBAuvQoYMFBgZaw4YNbcSIEXbw4EHfehYvXmyhoaEmyTp06GBHjx61JUuWWEREhEmyVq1a+W6nMnPmTGvUqJFFRUVZYmKivfTSSybJ2rVrZ4888ojdcMMNJskaNGhg9913n5mZTZ061QIDA31X1pmZPf/888XaHjt2zPr162cNGzY0f39/u/HGG+2pp56y/Pz8El/zE0884au7devW9re//c1+9atfWWRkpEmyG264wf7whz/Y6tWrfetq2LChrVq1qsjcGDp0qLVo0cIKCwvNzGzDhg0WHh5uc+fOLXU+vfHGG6VeaRocHGwdOnSwKVOm2IkTJ5hfDsyvsv4+X3+88cYbvj5//vOfbeTIkdauXTsLDg62oKAg69ixo82ePduuXLlSbA5kZGTY7bffbo0aNTJJ5ufnZ+3bt7d58+aVOleaNGliP/nJT3zPPfbYY/bRRx/5/v317ezn52c333yz/e1vfysy3jfna3nJZVebmpXvvWFW9nwv7/y71udPed5TlVHa1abl+Rxav369devWzUJDQy0oKMj8/PxMku/K0ltvvdXmzJlT4u1u3LTNuNrUnaokvME5hw8ftoCAgCLhorb46quvzOv12nPPPed0Kaik2jy/qtr1zFc3hjfAjPDmVtV6tSmqX/v27TVnzhzNmTNHly5dcrqcImbPnq0ePXpo6tSpTpeCSqrN86uqMV8BuAXhrQ544oknlJiYqFGjRtWaH0d+4YUXtHv3bm3YsKFKf/cWNa82zq+qxnytnQ4cOFDkVhilPUaNGuV0qUCNIrzVEfPmzdPUqVPLvO9ZTXnzzTeVk5Oj999/v9y3aEDtVpvmV1VjvtZenTp1KnYFaUmPqzdkBuqLAKcLQNX57ne/W+YPb9eUe++9V/fee6/TZaCK1Zb5VdWYrwDchj1vAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAiAaU9kZiYWJN1AK6Qnp6usLAwBQSU+tYBHFObPrfPnj2r6Ohop8vANWzdulV9+/Z1ugxUULE9b7GxsUpISHCiFqBWMzNt27ZNf/3rX5WRkeF0OahFEhISFBsb69j6a9Pndm5urrZu3arNmzfrwoULTpeDa+jbt69uu+02p8tABXnMzJwuAnCLkydPavTo0dqxY4d+9atfadq0aU6XBNQaf/nLXzRhwgQVFBRo6dKluvvuu50uCaiTOOcNqICYmBi99957mjlzph599FElJCTo4sWLTpcFOCo7O1uPP/64vve976lfv37av38/wQ2oRux5Ayrp3Xff1YMPPqiIiAglJyerR48eTpcE1Ljt27dr7NixOn36tF588UWNHTvW6ZKAOo89b0AlDR48WHv27FGrVq3Ut29fLVq0yOmSgBqTn5+v+fPnKy4uTrGxsdq3bx/BDagh7HkDrlNBQYGeeeYZzZ07V8OGDdPSpUvVsGFDp8sCqs0//vEPjRs3Tjt27NDs2bM1Y8YM+fmxLwCoKYQ3oIq89957GjNmjIKCgrR69Wouv0edtHz5cj388MNq06aNVq5cqW7dujldElDv8F8loIoMGjRIe/bs0be+9S3dfvvtmj9/vvi/EeqKM2fOaNiwYXrooYf08MMPa8eOHQQ3wCHseQOqmJnpN7/5jWbMmKG7775bv//979WoUSOnywIq7Y033lBSUpLCwsK0fPlyDRgwwOmSgHqNPW9AFfN4PJo2bZreeecd7dixQz179tRHH33kdFlAhWVkZCgpKUnx8fEaMmSI9u7dS3ADagHCG1BNbr/9du3evVtdunTRd77zHc2ePVuFhYVOlwWUy5YtW9SrVy+tW7dO69at0/LlyxUWFuZ0WQBEeAOqVZMmTfTWW2/pueee0y9/+Uvde++9OnfunNNlAaXKy8vT7NmzNWDAAN10003avXu37r33XqfLAvA1nPMG1JCPP/5Yo0aNUl5enlatWqW4uDinSwKK2L9/v8aOHasjR47oueee049//GOnSwJQAva8ATXk1ltv1a5du9S3b18NGjSIw6ioNcxMixYt0i233KLg4GDt3LmT4AbUYux5A2rY1atRH3vsMQ0YMEArV65Us2bNnC4L9dSJEyf0gx/8QB9++KGefPJJ/cd//If8/f2dLgtAGdjzBtSwq1ejfvjhhzp27Ji6d++uv/zlL06XhXpozZo16tGjh9LS0rR161bNnj2b4Aa4AOENcEjv3r21c+dODRw4UEOGDNHs2bNVUFDgdFmoB9LT0zVmzBiNHDlSiYmJ2r59u3r16uV0WQDKicOmQC2wZMkSTZ06Vf369dPKlSt14403Ol0S6qg///nPeuihh+Tv76///u//1qBBg5wuCUAFsecNqAV+/OMf66OPPtIXX3yhHj16aNOmTU6XhDomOztb06ZN0/e+9z3169dPu3btIrgBLkV4A2qJXr16aefOnbrjjjt09913a9q0acrLy3O6LNQBH3/8sXr27Knly5dr+fLlSklJ4SfbABcjvAG1SHh4uP73f/9X//3f/63XXntNd9xxh7788kuny4JL5efna/78+YqLi1NsbKz27t2rBx980OmyAFwnwhtQC40bN07bt2/XuXPn1KNHD23YsMHpkuAyn3/+uQYOHKhf/OIXeuaZZ/T2228rJibG6bIAVAHCG1BL3Xzzzdq6davuuusuff/73+cwKsrFzLRkyRJ169ZNubm52rlzp2bOnCk/Pz7ugbqCq00BF1i+fLmmTJmizp07a/Xq1WrTpo3TJaEWOnPmjCZOnKhNmzZp+vTpmjNnjoKCgpwuC0AV479igAuMGzdOO3bsUHZ2tnr27Km1a9c6XRJqmddff12dO3fW/v379d577+lXv/oVwQ2oowhvgEt06tRJ27Zt0w9+8APdf//9mjZtmnJzc50uCw7LyMhQUlKSEhISdPfddys1NVVxcXFOlwWgGnHYFHChFStWaMqUKerUqZNWr16tdu3aOV0SHPDRRx9p3LhxyszM1Kuvvqphw4Y5XRKAGsCeN8CFxo4dqx07digvL0+9evVSSkqK0yWhBuXk5Ojxxx/XgAED1L17d+3fv5/gBtQjhDfApTp27KitW7dq/PjxGjlypJKSkpSTk+N0Wahm+/btU9++ffXb3/5Wv/3tb/X666+rSZMmTpcFoAYR3gAX83q9WrRokV5//XWlpKSoX79+OnLkiNNloRqYmRYtWqTevXsrJCREn3zyiX784x87XRYABxDegDrgvvvu065duxQYGKhevXpp1apVTpeEKnT8+HENGjRIM2bM0OOPP66//e1vat++vdNlAXAI4Q2oI1q3bq3NmzdrwoQJeuCBBzRu3DhlZWU5XRau05o1a9SzZ0999dVX2rZtm2bPni1/f3+nywLgIMIbUIcEBwdr0aJF+r//+z+99dZb6tOnj/bv3+90WaiEs2fP6r777tPIkSM1duxYffLJJ+rZs6fTZQGoBQhvQB00fPhw7d69W5GRkerbt6/+8Ic/OF0SKuDtt99Wjx499Mknn+jdd9/VokWLFBwc7HRZAGoJwhtQR7Vs2VIffPCBHn74YY0dO1bjxo3T5cuXnS4LZcjOzta0adM0ZMgQ9e/fX7t379agQYOcLgtALcNNeoF6YP369ZowYYKaNm2qlJQUde3a1emS8A0ff/yxxo4dq7S0NL300ksaM2aM0yUBqKXY8wbUA8OGDdPu3bvVuHFjffvb39aiRYtKbWtm+tOf/lSD1dVtp06dKvP5/Px8zZ8/X3FxcWrVqpX27dtHcANQJsIbUE/Exsbq/fff12OPPaZHH31U48aN06VLl4q1W7hwoYYPH65PPvnEgSrrlrNnz6pPnz76y1/+UuLzBw4c0G233aZf/OIXWrBggd5++221aNGihqsE4DYcNgXqoXfeeUcPPvigoqKilJycrO7du0uStm3bpv79+6uwsFA33XST9uzZw4nylWRmGjp0qDZu3KimTZvqs88+U6NGjXzPvfrqq3r00Ud18803a8WKFerYsaPDFQNwC/a8AfXQHXfcoR07dqhp06bq27evFi1apPT0dCUkJMjj8cjMdOTIEf3nf/6n06W61osvvqi3335bknThwgU99NBDkqTTp0/rnnvu0cMPP6yf/OQn+vDDDwluACqEPW9APZafn6///M//1K9+9Su1adNGx48fV35+vu95j8ejDz74QHFxcQ5W6T779u3TLbfcotzc3CLLp02bppUrVyoyMlLLly9X//79HaoQgJsR3gBoypQpeuWVV/TNjwN/f3/fSfQhISEOVecuV65cUc+ePXXkyJFiQdjf31/333+/fve73yksLMzBKgG4GYdNgXouNTVVr732WrHgJkkFBQU6ceKEnnzySQcqc6ef/vSnxYKb9K/z3Dwej44dO6bQ0FCHqgNQF7DnDajHLl26pO7du+vEiRPFwsbXeTwevffee/rOd75Tg9W5zxtvvKH4+Pgy2/j5+en555/XI488UkNVAahrCG9APZaYmKi1a9des52/v79atGihTz/9VA0aNKiBytznxIkT6tKliy5dulTiXsyvCwwM1K5du9S5c+caqg5AXcJhU6CeOn/+vBo3bqzo6GhJKvOWIAUFBTp16pQee+yxmirPVQoKCjRy5EhduXLlmsEtKChIeXl5mjx58jXbAkBJ2PMGQPv379eaNWv0v//7vzp8+LACAwOVn59fLFx4PB5t3LhRd911l0OV1k6zZ8/WM888o8LCwmLP+fv7y8xUWFioli1bavjw4brnnns0YMAA7qEHoFIIbwCK+Oyzz7Ru3TqtXbtWu3btkp+fny98+Pn5qVmzZvrss88UERHhdKm1wt///nd95zvf8QU3j8ejwMBA5ebmKjIyUnfccYe++93v6u6771ZMTIzD1QKoCwhvqJVOnjypjz76yOky6r3z589rx44d2rZtmz799FNfQBk0aJAmTZrkcHXOu3z5sqZPn64LFy5I+tfFCO3bt9ctt9yi7t27q3Xr1vJ4PA5XWf/Exsbqtttuc7oMoNoQ3lArpaSkaOTIkU6XAcCFEhIStGbNGqfLAKpNgNMFAGXh/xa105UrV/TZZ5+pZ8+eTpfimEuXLunMmTNq166d06XgaxITE50uAah2hDcAFeb1eut1cJOksLAwfiUBgCO4VQgAAICLEN4AAABchPAGAADgIoQ3AAAAFyG8AQAAuAjhDQAAwEUIbwAAAC5CeAMAAHARwhsAAICLEN4AAABchPAGAADgIoQ3AAAAFyG8AQAAuAjhDfXes88+q8jISHk8Hu3evdvpcsptwoQJ8nq98ng8unLlSp2po0+fPvL391ePHj0qPcaGDRsUGRmpP/7xj6W2mThxosLDw6/7737w4EH99Kc/VefOnRUeHq6AgABFRkbqpptu0tChQ7Vly5ZKjw0AJSG8od574okn9Lvf/c7pMips2bJl+vnPf+50GVVex/bt2zVo0KDrGsPMrtnmtdde06uvvnpd61m6dKm6du2q1NRUvfDCC/riiy90+fJl7dq1S88884zS09O1d+/e61oHAHxTgNMFAFUlOztbgwcP1kcffeR0KagCHo+n0n2HDh2qixcvVmE1xW3dulVJSUn6zne+o7ffflsBAf/+OG3btq3atm2rqKgoHT58uFrruB5Ovmd4vwKVR3hDnbF06VKlpaU5XYYjrifoVKWqrCMwMLDKxirN9dQ7d+5cFRQU6Nlnny0S3L7urrvu0l133VXpdVQ3J98z9fn9ClwvDpuiTnjkkUc0ffp0HT16VB6PR+3bt5f0r8NnL7zwgr71rW8pODhYDRs21PDhw3XgwIEyxztz5oxat26tgIAAfe973/MtLygo0KxZs9SyZUuFhISoW7duSk5OliS9/PLLatCggUJDQ/Xmm29qyJAhioiIUExMjFatWlXp17ZixQr17t1bXq9XDRo0UOvWrfXMM8/4nvfz89Of/vQnDRkyRJGRkWrevLl+//vfFxnjb3/7m26++WZFRkbK6/Wqa9euevvttyVJv/71rxUaGqrw8HClpaVp+vTpatGihQ4ePFihOq9Vx8SJE+XxeOTxeNSuXTvt2rVL0r/OmQsNDVVkZKTWr1/va3/kyBF16tRJDRo0UEhIiAYMGKC///3vvudLq3vp0qVq2bKlPB6PXnrpJV97M9OCBQvUsWNHBQcHKzIyUjNmzCj2OjZt2qSIiAjNmzev1Neam5urd999V40bN9att95a7m1UnvlY0XlU1vwo6+9e2numquZ4Va8bwNcYUAslJydbRadnfHy8tWvXrsiyWbNmWVBQkK1YscLS09MtNTXVevXqZU2aNLHTp0/72q1atcok2a5du8zMLDc31+Lj4+3NN98sMt7Pf/5zCw4OtrVr19qFCxfsySefND8/P9u+fbuZmT311FMmyd599127ePGipaWl2YABA6xBgwaWm5tb4e2wcOFCk2TPPvusnTt3zs6fP2+/+93vbMyYMcXWl56ebufPn7e7777bgoOD7fLly75x1qxZY7Nnz7bz58/buXPnrG/fvta4cWPf81fHmTZtmr344ot233332WeffVbuOstbR3x8vPn7+9uXX35ZpP8DDzxg69ev9/178ODB1rZtW/vHP/5heXl5tm/fPvv2t79tXq/XDh06dM26v/gK3whSAAAgAElEQVTiC5NkL774YpG2Ho/Hnn/+ebtw4YJlZWXZ4sWLi/zdzczeeustCw8Ptzlz5pT6eg8dOmSSrG/fvuXeRmbln4/lnUfXmh/X+ruX9J6pqjleHesuj4SEBEtISCh3e8CNCG+olaoivGVlZVlYWJiNGjWqSLuPP/7YJBX5cv56eMvLy7PRo0fbxo0bi/TLzs620NDQIuNlZWVZcHCwTZkyxcz+/cWWnZ3ta3M1IBw5cqRCryc3N9eioqJs0KBBRZbn5+fbf/3Xf5W6vuXLl5sk27dvX6lj//KXvzRJlpaWVuo4FVHeOt555x2TZHPnzvUtu3jxonXo0MHy8/N9ywYPHmzdu3cvso7U1FSTZD//+c/LXK+ZFQtvWVlZFhoaanfeeWeRdt8M7eW1Y8cOk2R33HFHuftUZD6WZx6VZ3580zf/7t98z1TnHK+KdZcH4Q31AYdNUWft379fly5dUu/evYss79Onj4KCgrRt27ZifQoKCvTAAw+oadOmRQ6XSv+6JURWVpa6dOniWxYSEqJmzZqVeRg2KChIkpSXl1eh+lNTU5Wenl7snCl/f39Nmzat1H5XzxUra31X2xQUFFSopoooqY7/9//+n2666Sb9/ve/910Runr1ao0aNUr+/v5ljte1a1dFRkYqNTW1wrUcOXJEWVlZGjx4cIX7liQsLEySlJWVVe4+lZmPX/fNeVSZ+XGtv3t1zvHqWjdQHxHeUGelp6dL+vcX7ddFRUUpMzOz2PKf/OQnOnz4sF555RV9+umnRZ67fPmyJOnpp5/2nbvl8Xh0/PjxCn2Jl1dGRoav1uv1pz/9SQMHDlR0dLSCg4P12GOPXfeYleHxeDRp0iR9/vnnevfddyVJy5cv1w9/+MNy9Q8MDKxwCJakkydPSpKio6Mr3LckrVu3ltfr1aFDh8rdpzLzsSzlmR8V/btX5Rx3ct1AXUd4Q5119UutpC/F9PR0xcTEFFt+//336y9/+YuioqI0btw45efn+567+sW/cOFC2b9OOfA9quNGrDfeeKMk6auvvrqucU6cOKERI0aoWbNm2rZtmy5evKj58+dXRYmVMn78eHm9Xr322ms6ePCgIiIi1KpVq2v2y8/P1/nz59WyZcsKr9Pr9UqScnJyKty3JMHBwbrrrrv01Vdf6cMPPyy13fnz5zVx4kRJlZuPZbnW/KjM372q5riT6wbqA8Ib6qwuXbooLCxMO3bsKLJ827Ztys3N1S233FKsz6BBg9SkSRMtWbJEn3zyiebOnet7LjY2Vl6vt8Z+haF169Zq1KiR/vznP1/XOHv37lVeXp6mTJmitm3b+n4NwSkNGzbUyJEjtW7dOj333HP60Y9+VK5+7733ngoLC9WrV68Kr7NLly7y8/PT5s2bK9y3NLNnz1ZwcLAeffRRZWdnl9hm3759vtuIVGY+luVa86Myf/eqmuNOrhuoDwhvqDMaNWqkU6dO6dixY8rMzJS/v7+mT5+uN954QytXrlRGRob27t2ryZMnq3nz5kpKSip1rGHDhmn8+PGaN2+ePvnkE0n/2nszYcIErVq1Si+//LIyMjJUUFCgkydP6p///GeVv57g4GA9+eST+uCDDzR16lR9+eWXKiwsVGZmZrFDumW5uqfqnXfe0ZUrV3T48OFrnl9V3SZPnqycnBy99dZbuueee0psk5ubq4sXLyo/P187d+7U1KlT1apVK40fP77C64uOjlZ8fLzWrl2rpUuXKiMjQ6mpqVqyZEmxths3brzmrUIkqUePHvrDH/6gffv2acCAAdqwYYMuXryovLw8/eMf/9Crr76qH/7wh75zvbxeb6XnY0muNT/K83cv6T1TFXPcyXUD9YIjl0kA11CZq0137txprVq1spCQEIuLi7PTp09bYWGhLViwwDp06GCBgYHWsGFDGzFihB08eNDX7/XXX7eGDRuaJGvdurWlpaVZRkaGxcbGmiQLCwuz5cuXm5lZTk6OzZw501q2bGkBAQEWHR1t8fHxtn//flu8eLGFhoaaJOvQoYMdPXrUlixZYhERESbJWrVqVeQ2F+X10ksvWdeuXc3r9ZrX67WePXva4sWLbf78+RYSElJkfStXrvS9lpiYGN+VnjNnzrRGjRpZVFSUJSYm2ksvvWSSrF27dvaTn/zEN05sbKytWLGiQvVVpI6v69mzpz3xxBMljrls2TIbNGiQNW3a1AICAqxx48Y2evRoO378eInr/XrdL774ojVr1swkWWhoqA0bNszMzDIzM23ixInWuHFjCwsLs7i4OJs1a5avxj179piZ2YYNGyw8PLzIFbFlOXHihP385z+3rl27WlhYmPn7+1tUVJT17NnTfvjDH9qHH37oa1ue+VjReVTa/DAr++9+4sSJEt8zVTXHq3rd5cXVpqgPPGbl+BFAoIalpKRo5MiR5fqNSrjT0KFD9dJLL6lNmzZOl4I6JDExUZK0Zs0ahysBqg+HTQHUiK9fJZqamiqv10twA4BKILwBNejAgQNFboNQ2mPUqFF1rs6ZM2fq8OHDOnTokCZMmFDkJ74AAOXHD9MDNahTp06uOBRcHXWGhoaqU6dOatGihRYvXqybb765SscHgPqCPW8AasTcuXNVUFCgEydOlHqFKQDg2ghvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAiAU4XAJQlJSXF6RIAuMjJkycVExPjdBlAtSK8oVYbOXKk0yUAcJmEhASnSwCqlcfMzOkiAKAiPB6PkpOTdf/99ztdCgDUOM55AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcJEApwsAgLIsWbJEFy5cKLb8zTff1D/+8Y8iy8aPH68bbrihpkoDAEd4zMycLgIASpOUlKQlS5YoODjYt8zM5PF4fP/Oz89XZGSkTp8+rcDAQCfKBIAaw2FTALXa6NGjJUk5OTm+R25ubpF/+/n5afTo0QQ3APUCe94A1GqFhYVq3ry50tLSymz397//Xf3796+hqgDAOex5A1Cr+fn56cEHH1RQUFCpbZo3b65+/frVYFUA4BzCG4Bab/To0crNzS3xucDAQI0bN67IOXAAUJdx2BSAK7Rt27bY1aVX7d69W927d6/higDAGex5A+AK48aNK/GChLZt2xLcANQrhDcArvDggw8qLy+vyLLAwEBNmDDBoYoAwBkcNgXgGt26ddO+ffv09Y+tQ4cOqUOHDg5WBQA1iz1vAFxj3Lhx8vf3lyR5PB717NmT4Aag3iG8AXCNBx54QAUFBZIkf39//eAHP3C4IgCoeYQ3AK5x4403ql+/fvJ4PCosLFRiYqLTJQFAjSO8AXCVsWPHysx0++2368Ybb3S6HACocVywADggJSVFI0eOdLoM1FMJCQlas2aN02UAqKQApwsA6rPk5GSnS3Cl559/XklJSQoLC3O6FNdZuHCh0yUAuE6EN8BB999/v9MluFK/fv0UExPjdBmuxB43wP045w2A6xDcANRnhDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCG+BSEydOVHh4uDwej3bv3u10OY55/fXX1bZtW3k8niKPoKAgNW3aVAMHDtSCBQt04cIFp0sFgCpBeANc6rXXXtOrr77qdBmOi4+P1+eff6527dopMjJSZqbCwkKlpaUpJSVFbdq00cyZM9W5c2ft2LHD6XIB4LoR3gDUCtnZ2erXr1+VjOXxeBQVFaWBAwdq2bJlSklJ0ZkzZzR06FBdvHixStbhpKrcVgDch/AGuJjH43G6hCqzdOlSpaWlVcvYCQkJGj9+vNLS0vTKK69UyzpqUnVuKwC1H+ENcAkz04IFC9SxY0cFBwcrMjJSM2bMKNLm17/+tUJDQxUeHq60tDRNnz5dLVq00MGDB2VmeuGFF/Stb31LwcHBatiwoYYPH64DBw74+v/mN7+R1+tV06ZNNWnSJDVv3lxer1f9+vXTtm3bitVzrfGmTp2qoKAgNWvWzLfs4YcfVoMGDeTxePTVV19Jkh555BFNnz5dR48elcfjUfv27SVJmzZtUkREhObNm3fd22/8+PGSpI0bN9bJbQWgHjEANS45Odkq+vZ76qmnzOPx2PPPP28XLlywrKwsW7x4sUmyXbt2FWknyaZNm2Yvvvii3XffffbZZ5/ZrFmzLCgoyFasWGHp6emWmppqvXr1siZNmtjp06d9/ZOSkqxBgwb26aef2pUrV2z//v3Wp08fCw8PtxMnTvjalXe8MWPG2A033FDktSxYsMAk2dmzZ33L4uPjrV27dkXavfXWWxYeHm5z5sy55vZp166dRUZGlvp8RkaGSbLY2Ng6ua3KKyEhwRISEirVF0DtQHgDHFDR8JaVlWWhoaF25513Flm+atWqUsNbdnZ2kf5hYWE2atSoIv0//vhjk1QkHCUlJRULQdu3bzdJ9otf/KLC49VEIDG7dngzM/N4PBYVFeX7d33cVoQ3wP04bAq4wJEjR5SVlaXBgwdXqv/+/ft16dIl9e7du8jyPn36KCgoqNhhvm/q3bu3QkNDfYf5rnc8J1y+fFlmpoiIiDLbsa0A1HaEN8AFTp48KUmKjo6uVP/09HRJUlhYWLHnoqKilJmZec0xgoODdfbs2Sobr6YdOnRIktSpU6cy27GtANR2hDfABbxeryQpJyenUv2joqIkqcSgkJ6erpiYmDL75+XlFWl3veM5YdOmTZKkIUOGlNmObQWgtiO8AS7QpUsX+fn5afPmzZXuHxYWVuwmtdu2bVNubq5uueWWMvu///77MjP17du3wuMFBAQoLy+vUnVXldOnT2vhwoWKiYnRQw89VGbb+r6tANR+hDfABaKjoxUfH6+1a9dq6dKlysjIUGpqqpYsWVKu/l6vV9OnT9cbb7yhlStXKiMjQ3v37tXkyZPVvHlzJSUlFWlfWFioCxcuKD8/X6mpqXrkkUfUsmVL3+02KjJe+/btdf78ea1bt055eXk6e/asjh8/XqzGRo0a6dSpUzp27JgyMzOVl5enjRs3VuhWIWamS5cuqbCwUGams2fPKjk5Wf3795e/v7/WrVt3zXPe3LqtANQjjl4uAdRTlblVSGZmpk2cONEaN25sYWFhFhcXZ7NmzTJJFhMTY3v27LH58+dbSEiI75YYK1as8PUvLCy0BQsWWIcOHSwwMNAaNmxoI0aMsIMHDxZZT1JSkgUGBlqLFi0sICDAIiIibPjw4Xb06NEi7co73rlz52zQoEHm9XqtTZs29tOf/tRmzJhhkqx9+/a+W2rs3LnTWrVqZSEhIRYXF2enT5+2DRs2WHh4uM2dO7fU7bJ+/Xrr1q2bhYaGWlBQkPn5+Zkk35Wlt956q82ZM8fOnTtXpF9d21blxdWmgPt5zMwczI5AvZSSkqKRI0eqNr79Jk2apDVr1ujcuXNOl1LruXFbJSYmSpLWrFnjcCUAKovDpgCKKSgocLoE12BbAahphDcAAAAXIbwB8HnyySe1bNkyXbx4UW3atNHatWudLqnWYlsBcArnvAEOqM3nvKFu45w3wP3Y8wYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8AYAAOAihDcAAAAXIbwBAAC4COENAADARQhvAAAALhLgdAFAfebxeJwuAfVQQkKC0yUAuA4eMzOniwDqm5MnT+qjjz5yugzXGjlypB555BHddtttTpfiSrGxsWw7wMUIbwBcx+PxKDk5Wffff7/TpQBAjeOcNwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CIBThcAAGU5fvy4CgoKii0/c+aMPv/88yLLmjdvrpCQkJoqDQAc4TEzc7oIACjNkCFDtGnTpmu2CwgI0OnTp9W4ceMaqAoAnMNhUwC12qhRo+TxeMps4+fnpzvvvJPgBqBeILwBqNXuu+8+BQYGXrPd2LFja6AaAHAe4Q1ArRYeHq7vf//7ZQa4wMBA3XPPPTVYFQA4h/AGoNYbM2aM8vPzS3wuICBAI0aMUFhYWA1XBQDOILwBqPWGDh2qBg0alPhcQUGBxowZU8MVAYBzCG8Aar3g4GAlJCQoKCio2HNhYWH67ne/60BVAOAMwhsAV3jggQeUm5tbZFlgYKBGjRpVYqgDgLqK+7wBcIXCwkLdcMMN+uqrr4osf++99zRw4EBnigIAB7DnDYAr+Pn56YEHHiiyly06OloDBgxwsCoAqHmENwCuMXr0aN+h06CgII0bN07+/v4OVwUANYvDpgBcw8zUqlUrffHFF5Kk7du3q3fv3g5XBQA1iz1vAFzD4/Fo3LhxkqRWrVoR3ADUSwFOFwCguC1btuiFF15wuoxaKSMjQ5LUoEEDJSYmOlxN7XTbbbfp0UcfdboMANWEPW9ALfTFF19o7dq1TpdRK0VERCgyMlIxMTFOl1Irbd26VVu2bHG6DADViD1vQC22Zs0ap0uold5++23dddddTpdRK7E3Eqj72PMGwHUIbgDqM8IbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q2ooyZOnKjw8HB5PB7t3r3b6XIqZe7cufJ4PMUeXbp0qfBYr7/+utq2bVtsrKCgIDVt2lQDBw7UggULdOHChWp4JQBQdQhvQB312muv6dVXX3W6jFojPj5en3/+udq1a6fIyEiZmQoLC5WWlqaUlBS1adNGM2fOVOfOnbVjxw6nywWAUhHeANRqK1askJkVeezbt69KxvZ4PIqKitLAgQO1bNkypaSk6MyZMxo6dKguXrxYJesAgKpGeAPqMI/H43QJrpKQkKDx48crLS1Nr7zyitPlAECJCG9AHWFmWrBggTp27Kjg4GBFRkZqxowZxdoVFBRo1qxZatmypUJCQtStWzclJydLkl5++WU1aNBAoaGhevPNNzVkyBBFREQoJiZGq1atKjLO5s2bdeuttyo0NFQRERHq2rWrMjIyrrmO6rBp0yZFRERo3rx51z3W+PHjJUkbN270LauL2wyAexHegDriP/7jPzRz5kwlJSXpzJkzOn36tB5//PFi7R5//HH9+te/1sKFC/XPf/5T99xzjx544AHt2LFDU6ZM0c9+9jNlZ2crPDxcycnJOnr0qNq2basf/ehHysvLkyRdvnxZw4YNU0JCgs6fP6/Dhw/rpptuUm5u7jXXUVFPPPGEGjZsqKCgILVp00bDhw/X9u3bi7QpKCiQJBUWFlZ4/G/q0aOHJOnzzz/3LXPbNgNQxxmAWic5Odkq8vbMysqy0NBQu/POO4ssX7VqlUmyXbt2mZlZdna2hYaG2qhRo4r0DQ4OtilTppiZ2VNPPWWSLDs729dm8eLFJsmOHDliZmb79u0zSfbWW28Vq6U86yivEydO2M6dOy0zM9NycnJsy5Yt1rNnTwsJCbF9+/ZVaKyr2rVrZ5GRkWW28Xg8FhUVZWbu22YJCQmWkJBQoT4A3IU9b0AdcOTIEWVlZWnw4MFltjt48KCysrKK3GojJCREzZo104EDB0rtFxQUJEm+vUht27ZV06ZN9eCDD2r27Nk6duzYda+jJLGxserZs6fCwsIUFBSkvn37atmyZcrOztbixYsrNFZ5Xb58WWamiIgISe7bZgDqPsIbUAecPHlSkhQdHV1mu8uXL0uSnn766SL3Ojt+/LiysrLKvb6QkBD99a9/VVxcnObNm6e2bdtq1KhRys7OrrJ1lKZr167y9/fXoUOHrnusklwdt1OnTpLqxjYDULcQ3oA6wOv1SpJycnLKbHc13C1cuLDY7Te2bNlSoXV27txZf/zjH3Xq1CnNnDlTycnJeu6556p0HSUpLCxUYWGhgoODr3uskmzatEmSNGTIEEl1Y5sBqFsIb0Ad0KVLF/n5+Wnz5s1ltouNjZXX673uX1w4deqUPv30U0n/CjfPPvusevXqpU8//bTK1iFJd911V7Fl27dvl5nptttuu+7xv+n06dNauHChYmJi9NBDD0ly3zYDUPcR3oA6IDo6WvHx8Vq7dq2WLl2qjIwMpaamasmSJUXaeb1eTZgwQatWrdLLL7+sjIwMFRQU6OTJk/rnP/9Z7vWdOnVKkyZN0oEDB5Sbm6tdu3bp+PHj6tu3b5WtQ5K+/PJLrV69Wunp6crLy9OWLVs0ceJEtWzZUpMnT/a127hxY4VuFWJmunTpkgoLC2VmOnv2rJKTk9W/f3/5+/tr3bp1vnPe3LbNANQDNXt9BIDyqOjVpmZmmZmZNnHiRGvcuLGFhYVZXFyczZo1yyRZTEyM7dmzx8zMcnJybObMmdayZUsLCAiw6Ohoi4+Pt/3799vixYstNDTUJFmHDh3s6NGjtmTJEouIiDBJ1qpVKzt06JAdO3bM+vXrZw0bNjR/f3+78cYb7amnnrL8/PxrrqMipk+fbu3atbMGDRpYQECAxcTE2I9+9CM7depUkXYbNmyw8PBwmzt3bqljrV+/3rp162ahoaEWFBRkfn5+Jsl3Zemtt95qc+bMsXPnzhXr66ZtxtWmQN3nMTNzMDsCKEFKSopGjhwp3p6oqMTEREnSmjVrHK4EQHXhsCkAAICLEN4A1JgDBw4UuRVGaY9Ro0Y5XSoA1FoBThcAoP7o1KkTh4IB4Dqx5w0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CIBThcAoHSJiYlOlwCX2bp1q/r27et0GQCqEXvegFooNjZWCQkJTpdRa61fv16nTp1yuoxaqW/fvrrtttucLgNANfKYmTldBABUhMfjUXJysu6//36nSwGAGseeNwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABchvAEAALgI4Q0AAMBFCG8AAAAuQngDAABwEcIbAACAixDeAAAAXITwBgAA4CKENwAAABfxmJk5XQQAlGbs2LHavXt3kWXHjh1TdHS0GjRo4FsWGBioP/7xj2rRokVNlwgANSrA6QIAoCwdO3bUypUriy2/dOlSkX936tSJ4AagXuCwKYBabfTo0fJ4PGW2CQwM1Pjx42umIABwGIdNAdR6t9xyi3bv3q3CwsISn/d4PPr888/VunXrmi0MABzAnjcAtd64cePk51fyx5XH49Gtt95KcANQbxDeANR6I0eOLHWvm5+fn8aNG1fDFQGAcwhvAGq9Zs2aacCAAfL39y/x+fj4+BquCACcQ3gD4Apjx44ttszPz0+DBg3SDTfc4EBFAOAMwhsAV0hMTCzxvLeSQh0A1GWENwCuEBERoe9973sKCPj37Sn9/f117733OlgVANQ8whsA13jwwQdVUFAgSQoICNCwYcMUGRnpcFUAULMIbwBcY9iwYQoJCZEkFRQUaMyYMQ5XBAA1j/AGwDW8Xq/uu+8+SVJoaKiGDBnicEUAUPP4bVOgCp08eVIfffSR02XUabGxsZKkPn36aP369Q5XU7fFxsbqtttuc7oMAN/Az2MBVSglJUUjR450ugygSiQkJGjNmjVOlwHgG9jzBlQD/k9UvWbPnq2nn366yJWnqFqJiYlOlwCgFJzzBsB1CG4A6jPCGwDXIbgBqM8IbwAAAC5CeAMAAHARwhsAAICLEN4AAABchPAGAADgIoQ3AAAAFyG8AQAAuAjhDQAAwEUIbwAAAC5CeAMAAHARwhsAAICLEN4AAABchPAG1DITJ05UeHi4PB6Pdu/e7XQ5tUJhYaEWLlyofv36VXqM119/XW3btpXH4ynyCAoKUtOmTTVw4EAtWLBAFy5cqMLKAaDqEd6AWua1117Tq6++6nQZtcbhw4d1++2369FHH1VWVlalx4mPj9fnn3+udu3aKTIyUmamwsJCpaWlKSUlRW3atNHMmTPVuXNn7dixowpfAQBULcIbgGqVnZ1d6T1me/bs0eOPP67JkyerR48eVVyZ5PF4FBUVpYEDB2rZsmVKSUnRmTNnNHToUF28eLHK11fTrmfbA6i9CG9ALeTxeJwuocosXbpUaWlplerbvXt3vf766xozZoyCg4OruLLiEhISNH78eKWlpemVV16p9oPeucUAABB3SURBVPVVt+vZ9gBqL8Ib4DAz04IFC9SxY0cFBwcrMjJSM2bMKNLm17/+tUJDQxUeHq60tDRNnz5dLVq00MGDB2VmeuGFF/Stb31LwcHBatiwoYYPH64DBw74+v/mN7+R1+tV06ZNNWnSJDVv3lxer1f9+vXTtm3bitVzrfGmTp2qoKAgNWvWzLfs4YcfVoMGDeTxePTVV19Jkh555BFNnz5dR48elcfjUfv27atjE2rTpk2KiIjQvHnzrnus8ePHS5I2btwoiW0PoBYyAFUmOTnZKvq2euqpp8zj8djzzz9vFy5csKysLFu8eLFJsl27dhVpJ8mmTZtmL774ot1333322Wef2axZsywoKMhWrFhh6enplpqaar169bImTZrY6dOnff2TkpKsQYMG9umnn9qVK1ds//791qdPHwsPD7cTJ0742pV3vDFjxtgNN9xQ5LUsWLDAJNnZs2d9y+Lj461du3YV2iYl+fa3v23du3cv8bm33nrLwsPDbc6cOdccp127dhYZGVnq8xkZGSbJYmNjfcvq47ZPSEiwhISESvUFUL0Ib0AVqmh4y8rKstDQULvzzjuLLF+1alWp4S07O7tI/7CwMBs1alSR/h9//LFJKhJmkpKSioWW7du3myT7xS9+UeHxalN4q4hrhTczM4/HY1FRUb5/18dtT3gDai8OmwIOOnLkiLKysjR48OBK9d+/f78uXbqk3r17F1nep08fBQUFFTss9029e/dWaGio77Dc9Y5XF1y+fFlmpoiIiDLbse0BOIXwBjjo5MmTkqTo6OhK9U9PT5ckhYWFFXsuKipKmZmZ1xwjODhYZ8+erbLx3O7QoUOSpE6dOpXZjm0PwCmEN8BBXq9XkpSTk1Op/lFRUZJU4hd7enq6YmJiyuyfl5dXpN31jlcXbNq0SZI0ZMiQMtux7QE4hfAGOKhLly7y8/PT5s2bK90/LCys2E1lt23bptzcXN1yyy1l9n///fdlZurbt2+FxwsICFBeXl6l6q6tTp8+rYULFyomJkYPPfRQmW3Z9gCcQngDHBQdHa34+HitXbtWS5cuVUZGhlJTU7VkyZJy9fd6vZo+fbreeOMNrVy5UhkZGdq7d68mT56s5s2bKykpqUj7wsJCXbhwQfn5+UpNTdUjjzyili1b+m6PUZHx2rdvr/Pnz2vdunXKy8vT2bNndfz48WI1NmrUSKdOndKxY8eUmZlZLaFj48aNFbpViJnp0qVLKiwslJnp7NmzSk5OVv/+/eXv769169Zd85w3tj0Axzh6uQRQx1TmViGZmZk2ceJEa9y4sYWFhVlcXJzNmjXLJFlMTIzt2bPH5s+fbyEhIb5bWKxYscLXv7Cw0BYsWGAdOnSwwMBAa9iwoY0YMcIOHjxYZD1JSUkWGBhoLVq0sICAAIuIiLDhw4fb0aNHi7Qr73jnzp2zQYMGmdfrtTZt2thPf/pTmzFjhkmy9u3b+26BsXPnTmvVqpWFhIRYXFxckVteXMuWLVusf//+1rx5c5NkkqxZs2bWr18/27x5s6/dhg0bLDw83ObOnVvqWOvXr7du3bpZaGioBQUFmZ+fn0nyXVl666232pw5c+zcuXNF+tXXbc/VpkDt5TEzcyw5AnVMSkqKRo4cqdr4tpo0aZLWrFmjc+fOOV1KvePGbZ+YmChJWrNmjcOVAPgmDpsC9UhBQYHTJdRbbHsAVYXwBqDGHDhwQB6P55qPUaNGOV0qANRahDegHnjyySe1bNkyXbx4UW3atNHatWsdqaNTp06yf/2yS5mP1atXO1Jfdagt2x5A3cE5b0AVqs3nvAEVwTlvQO3FnjcAAAAXIbwBAAC4COENAADARQhvAAAALkJ4AwAAcBHCGwAAgIsQ3gAAAFyE8Ib/3979hdRd/3Ecf339c46e6TmucE3RSZok6GztQsqtEGIXY3d5NjXMXAwWXUbhaDFqtCLW8KatsLqJwo5bMGvkbgoG/YMCt1rLZP8sMXGJZe6ITn3/LqLz+/nb3LSpXz/6fMC58Hu+5/t5+wXHk+855zsAAOAQ4g0AAMAhxBsAAIBDiDcAAACHEG8AAAAOId4AAAAckuL3AMBy1NbW5vcIwG3p7e1VXl6e32MAuAHiDVgANTU1fo8A3LZoNOr3CABuwDMz83sIAJgLz/MUi8W0Y8cOv0cBgEXHZ94AAAAcQrwBAAA4hHgDAABwCPEGAADgEOINAADAIcQbAACAQ4g3AAAAhxBvAAAADiHeAAAAHEK8AQAAOIR4AwAAcAjxBgAA4BDiDQAAwCHEGwAAgEOINwAAAIcQbwAAAA4h3gAAABxCvAEAADiEeAMAAHAI8QYAAOAQ4g0AAMAhxBsAAIBDiDcAAACHEG8AAAAOId4AAAAcQrwBAAA4hHgDAABwCPEGAADgEOINAADAIcQbAACAQ4g3AAAAhxBvAAAADiHeAAAAHJLi9wAAcDMtLS0aGhq6bnt7e7suXbo0bVtjY6PuuuuuxRoNAHzhmZn5PQQAzGT37t1qaWlRMBhMbDMzeZ6X+HliYkKRSET9/f1KTU31Y0wAWDS8bQpgSaurq5MkjY2NJR7j4+PTfk5KSlJdXR3hBmBF4MobgCVtampKOTk5GhgYuOl+X3zxhTZt2rRIUwGAf7jyBmBJS0pKUn19vQKBwIz75OTkqLKychGnAgD/EG8Alry6ujqNj4/f8LnU1FQ1NDRM+wwcACxnvG0KwAmFhYXXfbv0H6dPn9Z99923yBMBgD+48gbACQ0NDTf8QkJhYSHhBmBFId4AOKG+vl7Xrl2bti01NVU7d+70aSIA8AdvmwJwRnl5uc6ePav//Weru7tbxcXFPk4FAIuLK28AnNHQ0KDk5GRJkud5uv/++wk3ACsO8QbAGY899pgmJyclScnJyXriiSd8nggAFh/xBsAZubm5qqyslOd5mpqa0vbt2/0eCQAWHfEGwCmPP/64zEwPP/ywcnNz/R4HABYdX1gAlqC2tjbV1NT4PQYcFY1GdfToUb/HALBAUvweAMDMYrGY3yMsSYcOHdLu3buVkZHh9yhLTnNzs98jAFhgxBuwhO3YscPvEZakyspK5eXl+T3GksQVN2D54zNvAJxDuAFYyYg3AAAAhxBvAAAADiHeAAAAHEK8AQAAOIR4AwAAcAjxBgAA4BDiDQAAwCHEGwAAgEOINwAAAIcQbwAAAA4h3gAAABxCvAEAADiEeAMAAHAI8QYsU7t27VJmZqY8z9Pp06f9Hudfu3btml555RXdc889CgQCysrKUllZmS5fvjyn43z00UcqLCyU53nTHoFAQGvWrFFVVZUOHjyooaGhhflFAGCeEG/AMvXOO+/o7bff9nuM21ZTU6P33ntPH3zwgeLxuH766ScVFRVpZGRkTseprq7WxYsXVVRUpEgkIjPT1NSUBgYG1NbWprvvvltNTU0qLS3Vd999t0C/DQDcvhS/BwCAmXz44Yc6fvy4zpw5o/Xr10uScnJy1N7ePi/H9zxPWVlZqqqqUlVVlbZt26aamhpt27ZN3d3dikQi87IOAMwnrrwBy5jneX6PcFvefPNNbdy4MRFuCy0ajaqxsVEDAwN66623FmVNAJgr4g1YJsxMBw8e1L333qtgMKhIJKLnnnvuuv0mJye1b98+rVu3Tunp6SovL1csFpMkHTlyRKtWrVIoFFJ7e7u2bt2qcDisvLw8tba2TjvOqVOnVFFRoVAopHA4rPXr12t4ePiWa8zW+Pi4vvnmG23YsOGW+548eVLhcFgHDhyY0xo30tjYKEnq6OhIbHPlnAFYIQzAkhOLxWyuf5579+41z/Ps0KFDNjQ0ZPF43A4fPmySrLOzM7Hfs88+a8Fg0I4dO2ZDQ0P2/PPPW1JSkn377beJ40iyzz77zP78808bGBiwhx56yFatWmXj4+NmZjYyMmLhcNhee+01Gx0dtf7+fnv00UftypUrs1pjNi5dumSSbMOGDVZVVWVr1661YDBoJSUl9sYbb9jU1FRi3xMnTlhmZqbt37//lsctKiqySCQy4/PDw8MmyfLz8507Z2Zm0WjUotHonF4DwC3EG7AEzTXe4vG4hUIh27Jly7Ttra2t0+JtdHTUQqGQ1dbWTnttMBi0p59+2sz+GyKjo6OJff6JwPPnz5uZ2dmzZ02SnThx4rpZZrPGbPzwww8mybZs2WJffvmlDQ4O2h9//GF79uwxSfb+++/P+lj/61bxZmbmeZ5lZWXN+vdZKufMjHgDVgLeNgWWgfPnzysej+uRRx656X4///yz4vG4ysrKEtvS09O1du1adXV1zfi6QCAg6e/bdkhSYWGh1qxZo/r6er344ovTbtvxb9f4f8FgUJJUWlqqyspK3XHHHYpEInrppZcUiUTU0tIy62PNxdWrV2VmCofDktw6ZwBWBuINWAZ6e3slSdnZ2Tfd7+rVq5KkF154Ydq9znp6ehSPx2e9Xnp6uj7//HNt3rxZBw4cUGFhoWprazU6Ojpva+Tk5EiSfv/992nbA4GACgoKdOHChVkfay66u7slSSUlJZLcOmcAVgbiDVgG0tLSJEljY2M33e+fuGtubpb9/bGJxOPrr7+e05qlpaX65JNP1NfXp6amJsViMb3++uvztkZGRoaKi4t17ty5656bmJhYsNt4nDx5UpK0detWSW6dMwArA/EGLANlZWVKSkrSqVOnbrpffn6+0tLSbvt/XOjr60tEVXZ2tl599VVt3LhR586dm7c1pL9v0NvZ2amLFy8mtsXjcfX09CzI7UP6+/vV3NysvLw8Pfnkk5LcO2cAlj/iDVgGsrOzVV1drWPHjundd9/V8PCwvv/+++s+F5aWlqadO3eqtbVVR44c0fDwsCYnJ9Xb26vffvtt1uv19fXpqaeeUldXl8bHx9XZ2amenh498MAD87aGJD3zzDMqKChQY2OjfvnlFw0ODqqpqUmjo6Pas2dPYr+Ojo453SrEzDQyMqKpqSmZma5cuaJYLKZNmzYpOTlZx48fT3zmzbVzBmAFWOQvSACYhX9zq5C//vrLdu3aZXfeeadlZGTY5s2bbd++fSbJ8vLy7MyZM2ZmNjY2Zk1NTbZu3TpLSUmx7Oxsq66uth9//NEOHz5soVDIJFlxcbFduHDBWlpaLBwOmyQrKCiw7u5uu3z5slVWVtrq1astOTnZcnNzbe/evTYxMXHLNebq119/tbq6Olu9erUFg0GrqKiwjo6Oaft8+umnlpmZaS+//PKMx/n444+tvLzcQqGQBQIBS0pKMkmJb5ZWVFTY/v37bXBw8LrXunTO+LYpsPx5ZmY+tiOAG2hra1NNTY3488Rcbd++XZJ09OhRnycBsFB42xQAAMAhxBuARdPV1TXtVhgzPWpra/0eFQCWrBS/BwCwcpSUlPBWMADcJq68AQAAOIR4AwAAcAjxBgAA4BDiDQAAwCHEGwAAgEOINwAAAIcQbwAAAA4h3gAAABxCvAEAADiEeAMAAHAI8QYAAOAQ4g0AAMAhxBsAAIBDiDcAAACHpPg9AICZeZ7n9whwUDQa9XsEAAvIMzPzewgA0/X29uqrr77yeww4Kj8/Xw8++KDfYwBYIMQbAACAQ/jMGwAAgEOINwAAAIcQbwAAAA5JkXTU7yEAAAAwO/8BGfpun4b6HcsAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bG9W0kEP9ve_"
      },
      "source": [
        "# Compile token char model\n",
        "model_4.compile(loss=\"categorical_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(), # section 4.2 of https://arxiv.org/pdf/1612.05251.pdf mentions using SGD but we'll stick with Adam\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_Q9rK8I96lF"
      },
      "source": [
        "### Combining token and character data into a tf.data dataset\n",
        "\n",
        "To keep our experiments fast, we'll fit our token-character-hybrid model on 10% of training and validate on 10% of validation batches. However, the difference with this model is that it requires two inputs, token-level sequences and character-level sequences.\n",
        "\n",
        "We can do this by create a tf.data.Dataset with a tuple as it's first input, for example:\n",
        "\n",
        "* `((token_data, char_data), (label))`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8lmlsJF5_KsT"
      },
      "source": [
        "# Combine chars and tokens into a dataset\n",
        "train_char_token_data = tf.data.Dataset.from_tensor_slices((train_sentences, train_chars)) # make data\n",
        "train_char_token_labels = tf.data.Dataset.from_tensor_slices(train_labels_one_hot) # make labels\n",
        "train_char_token_dataset = tf.data.Dataset.zip((train_char_token_data, train_char_token_labels)) # combine data and labels\n",
        "\n",
        "# Prefetch and batch train data\n",
        "train_char_token_dataset = train_char_token_dataset.batch(32).prefetch(tf.data.AUTOTUNE) \n",
        "\n",
        "# Repeat same steps validation data\n",
        "val_char_token_data = tf.data.Dataset.from_tensor_slices((val_sentences, val_chars))\n",
        "val_char_token_labels = tf.data.Dataset.from_tensor_slices(val_labels_one_hot)\n",
        "val_char_token_dataset = tf.data.Dataset.zip((val_char_token_data, val_char_token_labels))\n",
        "val_char_token_dataset = val_char_token_dataset.batch(32).prefetch(tf.data.AUTOTUNE)"
      ],
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n7u_Zwgf_NON",
        "outputId": "a71cee1c-42ff-41ee-8282-d3f4fe46087e"
      },
      "source": [
        "# Check out training char and token embedding dataset\n",
        "train_char_token_dataset, val_char_token_dataset"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<PrefetchDataset shapes: (((None,), (None,)), (None, 5)), types: ((tf.string, tf.string), tf.float64)>,\n",
              " <PrefetchDataset shapes: (((None,), (None,)), (None, 5)), types: ((tf.string, tf.string), tf.float64)>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "83dkpfH5_PMa"
      },
      "source": [
        "### Fitting a model on token and character-level sequences"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A8omi3OT_eUe",
        "outputId": "15426200-7a52-48fb-e112-d22851b3f752"
      },
      "source": [
        "# Fit the model on tokens and chars\n",
        "model_4_history = model_4.fit(train_char_token_dataset, # train on dataset of token and characters\n",
        "                              steps_per_epoch=int(0.1 * len(train_char_token_dataset)),\n",
        "                              epochs=3,\n",
        "                              validation_data=val_char_token_dataset,\n",
        "                              validation_steps=int(0.1 * len(val_char_token_dataset)))"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "562/562 [==============================] - 36s 51ms/step - loss: 0.9633 - accuracy: 0.6137 - val_loss: 0.7794 - val_accuracy: 0.7048\n",
            "Epoch 2/3\n",
            "562/562 [==============================] - 26s 47ms/step - loss: 0.7847 - accuracy: 0.6947 - val_loss: 0.7074 - val_accuracy: 0.7307\n",
            "Epoch 3/3\n",
            "562/562 [==============================] - 24s 43ms/step - loss: 0.7649 - accuracy: 0.7081 - val_loss: 0.6827 - val_accuracy: 0.7400\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CtKW5j6O_gEg",
        "outputId": "c18456d0-e38f-461e-e593-6c522be6e0da"
      },
      "source": [
        "# Evaluate on the whole validation dataset\n",
        "model_4.evaluate(val_char_token_dataset)"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 21s 23ms/step - loss: 0.6878 - accuracy: 0.7392\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.6878453493118286, 0.739242672920227]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jWbtU3IA_hpz",
        "outputId": "4af93dc2-1c40-4667-aabc-9d1a6c720282"
      },
      "source": [
        "# Make predictions using the token-character model hybrid\n",
        "model_4_pred_probs = model_4.predict(val_char_token_dataset)\n",
        "model_4_pred_probs"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[4.3105549e-01, 3.6402801e-01, 4.6365969e-03, 1.9255179e-01,\n",
              "        7.7281413e-03],\n",
              "       [3.9197549e-01, 4.2148548e-01, 4.7166538e-03, 1.7926705e-01,\n",
              "        2.5554076e-03],\n",
              "       [3.0017719e-01, 1.1882470e-01, 5.9802853e-02, 4.8302776e-01,\n",
              "        3.8167484e-02],\n",
              "       ...,\n",
              "       [4.1736494e-04, 8.0168499e-03, 6.7663439e-02, 1.8327124e-04,\n",
              "        9.2371911e-01],\n",
              "       [6.4267567e-03, 5.8696304e-02, 2.2873038e-01, 2.8747597e-03,\n",
              "        7.0327181e-01],\n",
              "       [3.1206575e-01, 4.2541340e-01, 1.9106378e-01, 3.1199541e-02,\n",
              "        4.0257555e-02]], dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Iskz6lBN_jnU",
        "outputId": "cec39ad6-9665-4daf-fee9-601b55ffd175"
      },
      "source": [
        "# Turn prediction probabilities into prediction classes\n",
        "model_4_preds = tf.argmax(model_4_pred_probs, axis=1)\n",
        "model_4_preds"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(30212,), dtype=int64, numpy=array([0, 1, 3, ..., 4, 4, 1])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rPlHumgd_lfU",
        "outputId": "3cd71c7e-0f05-421a-ba16-f35c4e7aff54"
      },
      "source": [
        "# Get results of token-char-hybrid model\n",
        "model_4_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                    y_pred=model_4_preds)\n",
        "model_4_results"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 73.92426850258175,\n",
              " 'f1': 0.7364952513703975,\n",
              " 'precision': 0.7393450461598946,\n",
              " 'recall': 0.7392426850258176}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yoxYXqHmYRwe"
      },
      "source": [
        "## Model 5: Transfer Learning with pretrained token embeddings + character embeddings + positional embeddings"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "I1wxQygsYfZr",
        "outputId": "ffbfab76-f6ca-496a-acdc-9e71606db72e"
      },
      "source": [
        "# Inspect training dataframe\n",
        "train_df.head()"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>text</th>\n",
              "      <th>line_number</th>\n",
              "      <th>total_lines</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>to investigate the efficacy of @ weeks of dail...</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>a total of @ patients with primary knee oa wer...</td>\n",
              "      <td>1</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>outcome measures included pain reduction and i...</td>\n",
              "      <td>2</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>pain was assessed using the visual analog pain...</td>\n",
              "      <td>3</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>secondary outcome measures included the wester...</td>\n",
              "      <td>4</td>\n",
              "      <td>11</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      target  ... total_lines\n",
              "0  OBJECTIVE  ...          11\n",
              "1    METHODS  ...          11\n",
              "2    METHODS  ...          11\n",
              "3    METHODS  ...          11\n",
              "4    METHODS  ...          11\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7XR-gAGYine"
      },
      "source": [
        "### Create positional embeddings\n",
        "\n",
        "Since our \"`line_number`\" and \"`total_line`\" columns are already numerical, we could pass them as they are to our model.\n",
        "\n",
        "But to avoid our model thinking a line with \"`line_number`\"=5 is five times greater than a line with \"`line_number`\"=1, we'll use one-hot-encoding to encode our \"`line_number`\" and \"`total_lines`\" features.\n",
        "\n",
        "To do this, we can use the `tf.one_hot` utility.\n",
        "\n",
        "`tf.one_hot` returns a one-hot-encoded tensor. It accepts an array (or tensor) as input and the depth parameter determines the dimension of the returned tensor.\n",
        "\n",
        "To figure out what we should set the depth parameter to, let's investigate the distribution of the \"`line_number`\" column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bp1S771WY2Oe",
        "outputId": "7520a10c-75e5-43b3-c80c-4aab838f4d59"
      },
      "source": [
        "# How many different line numbers are there?\n",
        "train_df[\"line_number\"].value_counts()"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0     15000\n",
              "1     15000\n",
              "2     15000\n",
              "3     15000\n",
              "4     14992\n",
              "5     14949\n",
              "6     14758\n",
              "7     14279\n",
              "8     13346\n",
              "9     11981\n",
              "10    10041\n",
              "11     7892\n",
              "12     5853\n",
              "13     4152\n",
              "14     2835\n",
              "15     1861\n",
              "16     1188\n",
              "17      751\n",
              "18      462\n",
              "19      286\n",
              "20      162\n",
              "21      101\n",
              "22       66\n",
              "23       33\n",
              "24       22\n",
              "25       14\n",
              "26        7\n",
              "27        4\n",
              "28        3\n",
              "29        1\n",
              "30        1\n",
              "Name: line_number, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 282
        },
        "id": "wRhZdKemY8Qp",
        "outputId": "b2354fdc-1ba4-490f-c7c1-6482bee3613d"
      },
      "source": [
        "# Check the distribution of \"line_number\" column\n",
        "train_df.line_number.plot.hist()"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<matplotlib.axes._subplots.AxesSubplot at 0x7f1bdbed21d0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 81
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD4CAYAAAAtrdtxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAASwElEQVR4nO3df9CdZX3n8ffHAAVtFShZliHQYM3UTV2rGIGO7a6LIwZphXbVwtQ16zCmM+KMTveH0eks1pYZ3NkWS0fd0pJpcNtGqlayBYeNiv3xBz+CoAiU8hTDkoiQGhCpFjb43T/O9cAxPnlyciXnOc/J837NnHnu+3tf97mva+7kfOb+ce6TqkKSpB7Pm3QHJEnTyxCRJHUzRCRJ3QwRSVI3Q0SS1O2ISXdgoZ1wwgm1cuXKSXdDkqbG7bff/o9VtXyuZUsuRFauXMm2bdsm3Q1JmhpJHtzXMk9nSZK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkrotuW+sH4yVG66fdBcW3PbLz5t0FyQtYh6JSJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbz87SvCb1vDCf2SVNB49EJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1G3sIZJkWZI7kvxlmz8tyS1JZpJ8MslRrf4jbX6mLV859B7vb/X7krxhqL621WaSbBj3WCRJP2ghjkTeA9w7NP9h4IqqegnwGHBxq18MPNbqV7R2JFkNXAj8NLAW+FgLpmXAR4FzgdXARa2tJGmBjDVEkqwAzgP+qM0HOBv4VGuyCbigTZ/f5mnLX9fanw9srqqnqurrwAxwRnvNVNUDVfU0sLm1lSQtkHEfiXwE+K/A99v8jwOPV9WeNr8DOLlNnww8BNCWf7u1f7a+1zr7qv+QJOuTbEuybdeuXQc7JklSM7YQSfILwKNVdfu4tjGqqrqqqtZU1Zrly5dPujuSdNgY5wMYXwO8KckbgaOBFwK/Bxyb5Ih2tLEC2Nna7wROAXYkOQJ4EfCtofqs4XX2VZckLYCxHYlU1furakVVrWRwYfyLVfWrwE3Am1uzdcB1bXpLm6ct/2JVVatf2O7eOg1YBdwK3Aasand7HdW2sWVc45Ek/bBJPAr+fcDmJL8N3AFc3epXA59IMgPsZhAKVNXdSa4F7gH2AJdU1TMASd4N3AgsAzZW1d0LOhJJWuIWJESq6kvAl9r0AwzurNq7zT8Db9nH+pcBl81RvwG44RB2VZJ0APzGuiSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSp29hCJMnRSW5N8pUkdyf5zVY/LcktSWaSfDLJUa3+I21+pi1fOfRe72/1+5K8Yai+ttVmkmwY11gkSXMb55HIU8DZVfUzwCuAtUnOAj4MXFFVLwEeAy5u7S8GHmv1K1o7kqwGLgR+GlgLfCzJsiTLgI8C5wKrgYtaW0nSAhlbiNTAk232yPYq4GzgU62+CbigTZ/f5mnLX5ckrb65qp6qqq8DM8AZ7TVTVQ9U1dPA5tZWkrRAjhjnm7ejhduBlzA4avgH4PGq2tOa7ABObtMnAw8BVNWeJN8GfrzVbx562+F1HtqrfuY++rEeWA9w6qmnHtygtCBWbrh+Ytvefvl5E9u2NG3GemG9qp6pqlcAKxgcObx0nNubpx9XVdWaqlqzfPnySXRBkg5LC3J3VlU9DtwE/CxwbJLZI6AVwM42vRM4BaAtfxHwreH6Xuvsqy5JWiDjvDtreZJj2/QxwOuBexmEyZtbs3XAdW16S5unLf9iVVWrX9ju3joNWAXcCtwGrGp3ex3F4OL7lnGNR5L0w8Z5TeQkYFO7LvI84Nqq+ssk9wCbk/w2cAdwdWt/NfCJJDPAbgahQFXdneRa4B5gD3BJVT0DkOTdwI3AMmBjVd09xvFIkvYythCpqq8Cr5yj/gCD6yN71/8ZeMs+3usy4LI56jcANxx0ZyVJXUY6nZXkX4+7I5Kk6TPqNZGPtW+fvyvJi8baI0nS1BgpRKrq54FfZXA31O1J/jTJ68faM0nSojfy3VlVdT/wG8D7gH8LXJnk75L88rg6J0la3Ea9JvLyJFcwuEX3bOAXq+pftekrxtg/SdIiNurdWb8P/BHwgar63myxqr6R5DfG0jNJ0qI3aoicB3xv6PsZzwOOrqrvVtUnxtY7SdKiNuo1kc8DxwzNP7/VJElL2KghcvTQY91p088fT5ckSdNi1BD5pySnz84keRXwvXnaS5KWgFGvibwX+PMk3wAC/EvgV8bWK0nSVBgpRKrqtiQvBX6qle6rqv83vm5JkqbBgTyA8dXAyrbO6UmoqmvG0itJ0lQYKUSSfAL4SeBO4JlWLsAQkaQlbNQjkTXA6vYjUZIkAaPfnfU1BhfTJUl61qhHIicA9yS5FXhqtlhVbxpLryRJU2HUEPngODshSZpOo97i+1dJfgJYVVWfT/J8Br9rLklawkZ9FPw7gU8Bf9BKJwOfHVenJEnTYdQL65cArwGegGd/oOpfjKtTkqTpMGqIPFVVT8/OJDmCwfdEJElL2Kgh8ldJPgAc035b/c+B/z2+bkmSpsGoIbIB2AXcBfwacAOD31uXJC1ho96d9X3gD9tLkiRg9GdnfZ05roFU1YsPeY8kSVPjQJ6dNeto4C3A8Ye+O5KkaTLSNZGq+tbQa2dVfQQ4b8x9kyQtcqOezjp9aPZ5DI5MDuS3SCRJh6FRg+B3hqb3ANuBtx7y3kiSpsqod2f9u3F3RJI0fUY9nfXr8y2vqt89NN2RJE2TA7k769XAljb/i8CtwP3j6JQkaTqMGiIrgNOr6jsAST4IXF9VbxtXxyRJi9+ojz05EXh6aP7pVpMkLWGjHolcA9ya5C/a/AXApvF0SZI0LUa9O+uyJJ8Dfr6V3lFVd4yvW5KkaTDq6SyA5wNPVNXvATuSnDZf4ySnJLkpyT1J7k7ynlY/PsnWJPe3v8e1epJcmWQmyVeHv+CYZF1rf3+SdUP1VyW5q61zZZIc0OglSQdl1J/HvRR4H/D+VjoS+F/7WW0P8J+qajVwFnBJktUMHiv/hapaBXyhzQOcC6xqr/XAx9u2jwcuBc4EzgAunQ2e1uadQ+utHWU8kqRDY9QjkV8C3gT8E0BVfQP4sflWqKqHq+rLbfo7wL0Mfpv9fJ67nrKJwfUVWv2aGrgZODbJScAbgK1VtbuqHgO2AmvbshdW1c1VVQyu28y+lyRpAYwaIk+3D+oCSPKCA9lIkpXAK4FbgBOr6uG26Js8d5fXycBDQ6vtaLX56jvmqM+1/fVJtiXZtmvXrgPpuiRpHqOGyLVJ/oDB0cE7gc8z4g9UJflR4NPAe6vqieFlw8E0TlV1VVWtqao1y5cvH/fmJGnJ2O/dWe1i9SeBlwJPAD8F/Leq2jrCukcyCJA/qarPtPIjSU6qqofbKalHW30ncMrQ6itabSfw2r3qX2r1FXO0lyQtkP0eibSjhRuqamtV/Zeq+s8jBkiAq4F793q21hZg9g6rdcB1Q/W3t7u0zgK+3U573Qick+S4dkH9HODGtuyJJGe1bb196L0kSQtg1C8bfjnJq6vqtgN479cA/wG4K8mdrfYB4HIGp8cuBh7kuUfK3wC8EZgBvgu8A6Cqdif5LWB22x+qqt1t+l3AHwPHAJ9rL0nSAhk1RM4E3pZkO4M7tMLgIOXl+1qhqv62tZvL6+ZoX8Al+3ivjcDGOerbgJftr/OSpPGYN0SSnFpV/5fBbbaSJP2A/R2JfJbB03sfTPLpqvr3C9EpSdJ02N+F9eHTUS8eZ0ckSdNnfyFS+5iWJGm/p7N+JskTDI5IjmnT8NyF9ReOtXeSpEVt3hCpqmUL1RFJ0vQ5kEfBS5L0AwwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN0NEktTNEJEkdTNEJEndDBFJUjdDRJLUzRCRJHUzRCRJ3QwRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN0NEktTNEJEkdTNEJEndjph0B6TFZuWG6yey3e2XnzeR7UoHwyMRSVI3Q0SS1M0QkSR1M0QkSd0MEUlSN0NEktTNEJEkdTNEJEndDBFJUrexhUiSjUkeTfK1odrxSbYmub/9Pa7Vk+TKJDNJvprk9KF11rX29ydZN1R/VZK72jpXJsm4xiJJmts4j0T+GFi7V20D8IWqWgV8oc0DnAusaq/1wMdhEDrApcCZwBnApbPB09q8c2i9vbclSRqzsYVIVf01sHuv8vnApja9CbhgqH5NDdwMHJvkJOANwNaq2l1VjwFbgbVt2Qur6uaqKuCaofeSJC2Qhb4mcmJVPdymvwmc2KZPBh4aarej1ear75ijPqck65NsS7Jt165dBzcCSdKzJnZhvR1B1AJt66qqWlNVa5YvX74Qm5SkJWGhQ+SRdiqK9vfRVt8JnDLUbkWrzVdfMUddkrSAFjpEtgCzd1itA64bqr+93aV1FvDtdtrrRuCcJMe1C+rnADe2ZU8kOavdlfX2ofeSJC2Qsf0oVZI/A14LnJBkB4O7rC4Hrk1yMfAg8NbW/AbgjcAM8F3gHQBVtTvJbwG3tXYfqqrZi/XvYnAH2DHA59pLkrSAxhYiVXXRPha9bo62BVyyj/fZCGyco74NeNnB9FGSdHD8xrokqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSep2xKQ7IGlg5YbrJ7Ld7ZefN5Ht6vDgkYgkqZshIknqZohIkroZIpKkboaIJKmbISJJ6maISJK6GSKSpG6GiCSpmyEiSepmiEiSuhkikqRuhogkqZtP8ZWWuEk9PRh8gvDhYOqPRJKsTXJfkpkkGybdH0laSqY6RJIsAz4KnAusBi5KsnqyvZKkpWPaT2edAcxU1QMASTYD5wP3TLRXkkbiD3FNv2kPkZOBh4bmdwBn7t0oyXpgfZt9Msl9nds7AfjHznUXm8NlLIfLOMCxLJh8eOSmi3ocB+hgxvIT+1ow7SEykqq6CrjqYN8nybaqWnMIujRxh8tYDpdxgGNZjA6XccD4xjLV10SAncApQ/MrWk2StACmPURuA1YlOS3JUcCFwJYJ90mSloypPp1VVXuSvBu4EVgGbKyqu8e4yYM+JbaIHC5jOVzGAY5lMTpcxgFjGkuqahzvK0laAqb9dJYkaYIMEUlSN0NkBIfTo1WSbE9yV5I7k2ybdH8ORJKNSR5N8rWh2vFJtia5v/09bpJ9HNU+xvLBJDvbvrkzyRsn2cdRJDklyU1J7klyd5L3tPrU7Zd5xjKN++XoJLcm+Uoby2+2+mlJbmmfZZ9sNyQd3La8JjK/9miVvwdez+DLjLcBF1XVVH4rPsl2YE1VTd0XqJL8G+BJ4Jqqelmr/Xdgd1Vd3gL+uKp63yT7OYp9jOWDwJNV9T8m2bcDkeQk4KSq+nKSHwNuBy4A/iNTtl/mGctbmb79EuAFVfVkkiOBvwXeA/w68Jmq2pzkfwJfqaqPH8y2PBLZv2cfrVJVTwOzj1bRAquqvwZ271U+H9jUpjcx+E+/6O1jLFOnqh6uqi+36e8A9zJ4ksTU7Zd5xjJ1auDJNntkexVwNvCpVj8k+8UQ2b+5Hq0ylf+wmgL+T5Lb2+Ngpt2JVfVwm/4mcOIkO3MIvDvJV9vprkV/CmhYkpXAK4FbmPL9stdYYAr3S5JlSe4EHgW2Av8APF5Ve1qTQ/JZZogsPT9XVaczePLxJe20ymGhBudmp/n87MeBnwReATwM/M5kuzO6JD8KfBp4b1U9Mbxs2vbLHGOZyv1SVc9U1SsYPMnjDOCl49iOIbJ/h9WjVapqZ/v7KPAXDP5xTbNH2rns2XPaj064P92q6pH2H//7wB8yJfumnXP/NPAnVfWZVp7K/TLXWKZ1v8yqqseBm4CfBY5NMvsl80PyWWaI7N9h82iVJC9oFwxJ8gLgHOBr86+16G0B1rXpdcB1E+zLQZn90G1+iSnYN+0C7tXAvVX1u0OLpm6/7GssU7pflic5tk0fw+DGoHsZhMmbW7NDsl+8O2sE7Za+j/Dco1Uum3CXuiR5MYOjDxg88uZPp2ksSf4MeC2DR1o/AlwKfBa4FjgVeBB4a1Ut+gvW+xjLaxmcMilgO/BrQ9cVFqUkPwf8DXAX8P1W/gCDawlTtV/mGctFTN9+eTmDC+fLGBwsXFtVH2qfAZuB44E7gLdV1VMHtS1DRJLUy9NZkqRuhogkqZshIknqZohIkroZIpKkboaIJKmbISJJ6vb/AVwSphAAsBgmAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2LRqNfRfY_SM"
      },
      "source": [
        "# Use TensorFlow to create one-hot-encoded tensors of our \"line_number\" column \n",
        "train_line_numbers_one_hot = tf.one_hot(train_df[\"line_number\"].to_numpy(), depth=15)\n",
        "val_line_numbers_one_hot = tf.one_hot(val_df[\"line_number\"].to_numpy(), depth=15)\n",
        "test_line_numbers_one_hot = tf.one_hot(test_df[\"line_number\"].to_numpy(), depth=15)"
      ],
      "execution_count": 82,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpotHQuMY_O0"
      },
      "source": [
        "Setting the depth parameter of tf.one_hot to 15 means any sample with a \"`line_number`\" value of over 15 gets set to a tensor of all 0's, where as any sample with a \"`line_number`\" of under 15 gets turned into a tensor of all 0's but with a 1 at the index equal to the \"`line_number`\" value.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "87xBjEeHY_MM",
        "outputId": "d4b599dc-2c5e-4521-caf6-666100acce3f"
      },
      "source": [
        "# Check one-hot encoded \"line_number\" feature samples\n",
        "train_line_numbers_one_hot.shape, train_line_numbers_one_hot[:20]"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([180040, 15]), <tf.Tensor: shape=(20, 15), dtype=float32, numpy=\n",
              " array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0.],\n",
              "        [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.]],\n",
              "       dtype=float32)>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-T4lhMV6Y_Js",
        "outputId": "8831e93e-9218-41df-c5b3-48cf32d63690"
      },
      "source": [
        "# How many different numbers of lines are there?\n",
        "train_df[\"total_lines\"].value_counts()"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "11    24468\n",
              "10    23639\n",
              "12    22113\n",
              "9     19400\n",
              "13    18438\n",
              "14    14610\n",
              "8     12285\n",
              "15    10768\n",
              "7      7464\n",
              "16     7429\n",
              "17     5202\n",
              "6      3353\n",
              "18     3344\n",
              "19     2480\n",
              "20     1281\n",
              "5      1146\n",
              "21      770\n",
              "22      759\n",
              "23      264\n",
              "4       215\n",
              "24      200\n",
              "25      182\n",
              "26       81\n",
              "28       58\n",
              "3        32\n",
              "30       31\n",
              "27       28\n",
              "Name: total_lines, dtype: int64"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        },
        "id": "EuoXIpdUY_Hn",
        "outputId": "4d3b50a8-9e94-4e3a-99d1-ac35c4f4411c"
      },
      "source": [
        "# Check the distribution of total lines\n",
        "train_df.total_lines.plot.hist();"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAD6CAYAAABgZXp6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXpUlEQVR4nO3df7BfdX3n8efLRCpSkVDSLJNgg21Gl7r+gCvg1HatjCHg1tBdl4WtS5ZhiDNgV8f9QXQ6i8Uyk+5spdJatqlkTVwV8SfZEppGxHb7Bz+CIAjo5IqwJAJJDRDRFhZ97x/fz5Wv4ebyzbn53i/35vmY+c49530+55zPZ74TXpxzPt/vN1WFJEldvGjUHZAkzV6GiCSpM0NEktSZISJJ6swQkSR1ZohIkjobWogkeVWSO/tee5O8L8nRSbYm2d7+Lmjtk+TKJONJ7kpyYt+xVrX225Os6quflOTuts+VSTKs8UiSnisz8TmRJPOAncApwMXAnqpam2QNsKCqLklyJvC7wJmt3Uer6pQkRwPbgDGggNuBk6rqsSS3Av8BuAXYDFxZVTdM1Zdjjjmmli5dOpRxStJcdPvtt/99VS2cbNv8GerDacB3qurBJCuBt7T6BuBrwCXASmBj9VLt5iRHJTm2td1aVXsAkmwFViT5GnBkVd3c6huBs4ApQ2Tp0qVs27bt4I5OkuawJA/ub9tMPRM5B/hMW15UVQ+35UeARW15MfBQ3z47Wm2q+o5J6pKkGTL0EElyGPAO4HP7bmtXHUO/n5ZkdZJtSbbt3r172KeTpEPGTFyJnAF8vaoebeuPtttUtL+7Wn0ncFzffktabar6kknqz1FV66pqrKrGFi6c9LaeJKmDmQiRc3n2VhbAJmBihtUq4Lq++nltltapwBPtttcWYHmSBW0m13JgS9u2N8mpbVbWeX3HkiTNgKE+WE9yBPA24N195bXAtUkuAB4Ezm71zfRmZo0DPwLOB6iqPUk+DNzW2l028ZAduAj4BHA4vQfqUz5UlyQdXDMyxfeFZGxsrJydJUmDS3J7VY1Nts1PrEuSOjNEJEmdGSKSpM5m6hPrmqWWrrl+JOd9YO3bR3JeSQfGKxFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSps6GGSJKjknw+ybeS3JfkTUmOTrI1yfb2d0FrmyRXJhlPcleSE/uOs6q1355kVV/9pCR3t32uTJJhjkeS9LOGfSXyUeCvqurVwOuA+4A1wI1VtQy4sa0DnAEsa6/VwFUASY4GLgVOAU4GLp0Intbmwr79Vgx5PJKkPkMLkSQvB34DuBqgqp6uqseBlcCG1mwDcFZbXglsrJ6bgaOSHAucDmytqj1V9RiwFVjRth1ZVTdXVQEb+44lSZoBw7wSOR7YDfzPJHck+XiSI4BFVfVwa/MIsKgtLwYe6tt/R6tNVd8xSV2SNEOGGSLzgROBq6rqDcAPefbWFQDtCqKG2AcAkqxOsi3Jtt27dw/7dJJ0yBhmiOwAdlTVLW398/RC5dF2K4r2d1fbvhM4rm//Ja02VX3JJPXnqKp1VTVWVWMLFy6c1qAkSc8aWohU1SPAQ0le1UqnAfcCm4CJGVargOva8ibgvDZL61TgiXbbawuwPMmC9kB9ObClbdub5NQ2K+u8vmNJkmbA/CEf/3eBTyU5DLgfOJ9ecF2b5ALgQeDs1nYzcCYwDvyotaWq9iT5MHBba3dZVe1pyxcBnwAOB25oL0nSDBlqiFTVncDYJJtOm6RtARfv5zjrgfWT1LcBr5lmNyVJHfmJdUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOjNEJEmdGSKSpM4MEUlSZ4aIJKkzQ0SS1JkhIknqzBCRJHVmiEiSOhtqiCR5IMndSe5Msq3Vjk6yNcn29ndBqyfJlUnGk9yV5MS+46xq7bcnWdVXP6kdf7ztm2GOR5L0s2biSuQ3q+r1VTXW1tcAN1bVMuDGtg5wBrCsvVYDV0EvdIBLgVOAk4FLJ4Kntbmwb78Vwx+OJGnCKG5nrQQ2tOUNwFl99Y3VczNwVJJjgdOBrVW1p6oeA7YCK9q2I6vq5qoqYGPfsSRJM2DYIVLAXye5PcnqVltUVQ+35UeARW15MfBQ3747Wm2q+o5J6s+RZHWSbUm27d69ezrjkST1mT/k47+5qnYm+UVga5Jv9W+sqkpSQ+4DVbUOWAcwNjY29PNJ0qFiqFciVbWz/d0FfIneM41H260o2t9drflO4Li+3Ze02lT1JZPUJUkzZGghkuSIJC+bWAaWA98ENgETM6xWAde15U3AeW2W1qnAE+221xZgeZIF7YH6cmBL27Y3yaltVtZ5fceSJM2AYd7OWgR8qc26nQ98uqr+KsltwLVJLgAeBM5u7TcDZwLjwI+A8wGqak+SDwO3tXaXVdWetnwR8AngcOCG9pIkzZChhUhV3Q+8bpL694HTJqkXcPF+jrUeWD9JfRvwmml3VpLUiZ9YlyR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktTZQCGS5J8NuyOSpNln0CuRP0tya5KLkrx8qD2SJM0aA4VIVf068DvAccDtST6d5G1D7Zkk6QVv4GciVbUd+D3gEuCfA1cm+VaSfzmszkmSXtgGfSby2iRXAPcBbwV+q6r+aVu+Yoj9kyS9gM0fsN2fAB8HPlhV/zBRrKrvJfm9ofRMkvSCN+jtrLcDn54IkCQvSvJSgKr65FQ7JpmX5I4kf9nWj09yS5LxJJ9Nclir/1xbH2/bl/Yd4wOt/u0kp/fVV7TaeJI1BzJwSdL0DRoiXwEO71t/aasN4r30boNN+EPgiqr6FeAx4IJWvwB4rNWvaO1IcgJwDvCrwAp6M8XmJZkHfAw4AzgBOLe1lSTNkEFvZ72kqp6cWKmqJyeuRKaSZAm9q5jLgfcnCb3nKP+2NdkAfAi4CljZlgE+D/xpa78SuKaqngK+m2QcOLm1G6+q+9u5rmlt7x1wTHoBW7rm+pGd+4G1bx/ZuaXZZtArkR8mOXFiJclJwD9M0X7CHwP/BfhJW/8F4PGqeqat7wAWt+XFwEMAbfsTrf1P6/vss7+6JGmGDHol8j7gc0m+BwT4J8C/mWqHJP8C2FVVtyd5y7R6OU1JVgOrAV7xileMsiuSNKcMFCJVdVuSVwOvaqVvV9X/e57dfg14R5IzgZcARwIfBY5KMr9dbSwBdrb2O+l9mHFHkvnAy4Hv99Un9O+zv/q+/V8HrAMYGxur5+m3JGlAB/IFjG8EXgucSO8h9nlTNa6qD1TVkqpaSu/B+Fer6neAm4B3tmargOva8qa2Ttv+1aqqVj+nzd46HlgG3ArcBixrs70Oa+fYdADjkSRN00BXIkk+CfwycCfw41YuYGOHc14CXJPkD4A7gKtb/Wrgk+3B+R56oUBV3ZPkWnoPzJ8BLq6qH7d+vQfYAswD1lfVPR36I0nqaNBnImPACe3K4IBV1deAr7Xl+3l2dlV/m38E/vV+9r+c3gyvfeubgc1d+iRJmr5Bb2d9k97DdEmSfmrQK5FjgHuT3Ao8NVGsqncMpVeSpFlh0BD50DA7IUmanQad4vs3SX4JWFZVX2mfVp833K5Jkl7oBv0q+AvpfRXJn7fSYuDLw+qUJGl2GPTB+sX0Pjy4F376A1W/OKxOSZJmh0FD5KmqenpipX2i3E9+S9IhbtAQ+ZskHwQOb7+t/jngfw+vW5Kk2WDQEFkD7AbuBt5N7wN+/qKhJB3iBp2d9RPgL9pLkiRg8O/O+i6TPAOpqlce9B5JkmaNA/nurAkvofcdV0cf/O5IkmaTgZ6JVNX3+147q+qP6f3srSTpEDbo7awT+1ZfRO/KZNCrGEnSHDVoEPxR3/IzwAPA2Qe9N5KkWWXQ2Vm/OeyOSJJmn0FvZ71/qu1V9ZGD0x1J0mxyILOz3sizv2H+W/R+53z7MDoljdLSNdeP5LwPrHWuimafQUNkCXBiVf0AIMmHgOur6l3D6pgk6YVv0K89WQQ83bf+dKtJkg5hg16JbARuTfKltn4WsGE4XZIkzRaDzs66PMkNwK+30vlVdcfwuiVJmg0GvZ0F8FJgb1V9FNiR5PipGid5SZJbk3wjyT1Jfr/Vj09yS5LxJJ9Nclir/1xbH2/bl/Yd6wOt/u0kp/fVV7TaeJI1BzAWSdJBMOjP414KXAJ8oJVeDPyv59ntKeCtVfU64PXAiiSnAn8IXFFVvwI8BlzQ2l8APNbqV7R2JDkBOAf4VWAF8GdJ5iWZB3wMOAM4ATi3tZUkzZBBr0R+G3gH8EOAqvoe8LKpdqieJ9vqi9urgLfS+7126D1XOastr+TZ5yyfB05Lkla/pqqeqqrvAuPAye01XlX3t19dvKa1lSTNkEFD5OmqKtrXwSc5YpCd2hXDncAuYCvwHeDxqnqmNdkBLG7Li4GHANr2J4Bf6K/vs8/+6pKkGTJoiFyb5M+Bo5JcCHyFAX6gqqp+XFWvp/c5k5OBV3fu6TQkWZ1kW5Jtu3fvHkUXJGlOet7ZWe2W0mfpBcBe4FXAf62qrYOepKoeT3IT8CZ6QTS/XW0sAXa2ZjuB4+g9tJ8PvBz4fl99Qv8++6vve/51wDqAsbGx5/y4liSpm+e9Emm3sTZX1daq+s9V9Z8GCZAkC5Mc1ZYPB94G3AfcBLyzNVsFXNeWN7V12vavtnNvAs5ps7eOB5bR+8qV24BlbbbXYfQevk98LYskaQYM+mHDryd5Y1XddgDHPhbY0GZRvQi4tqr+Msm9wDVJ/gC4A7i6tb8a+GSScWAPvVCgqu5Jci1wL72vob+4qn4MkOQ9wBZgHrC+qu45gP5JkqZp0BA5BXhXkgfozdAKvYuU1+5vh6q6C3jDJPX76T0f2bf+j/R+dneyY10OXD5JfTOwebAhSJIOtilDJMkrqur/AqdP1U6SdGh6viuRL9P79t4Hk3yhqv7VTHRKkjQ7PN+D9fQtv3KYHZEkzT7PFyK1n2VJkp73dtbrkuyld0VyeFuGZx+sHznU3kmSXtCmDJGqmjdTHZEkzT4H8lXwkiT9DENEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktTZoD9KpRFauub6UXdBkibllYgkqTNDRJLUmSEiSerMEJEkdWaISJI6G1qIJDkuyU1J7k1yT5L3tvrRSbYm2d7+Lmj1JLkyyXiSu5Kc2HesVa399iSr+uonJbm77XNlkjy3J5KkYRnmlcgzwH+sqhOAU4GLk5wArAFurKplwI1tHeAMYFl7rQaugl7oAJcCpwAnA5dOBE9rc2HffiuGOB5J0j6GFiJV9XBVfb0t/wC4D1gMrAQ2tGYbgLPa8kpgY/XcDByV5FjgdGBrVe2pqseArcCKtu3Iqrq5qgrY2HcsSdIMmJFnIkmWAm8AbgEWVdXDbdMjwKK2vBh4qG+3Ha02VX3HJPXJzr86ybYk23bv3j2tsUiSnjX0EEny88AXgPdV1d7+be0Koobdh6paV1VjVTW2cOHCYZ9Okg4ZQw2RJC+mFyCfqqovtvKj7VYU7e+uVt8JHNe3+5JWm6q+ZJK6JGmGDHN2VoCrgfuq6iN9mzYBEzOsVgHX9dXPa7O0TgWeaLe9tgDLkyxoD9SXA1vatr1JTm3nOq/vWJKkGTDML2D8NeDfAXcnubPVPgisBa5NcgHwIHB227YZOBMYB34EnA9QVXuSfBi4rbW7rKr2tOWLgE8AhwM3tJckaYYMLUSq6u+A/X1u47RJ2hdw8X6OtR5YP0l9G/CaaXRTkjQNfmJdktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnQ0tRJKsT7IryTf7akcn2Zpke/u7oNWT5Mok40nuSnJi3z6rWvvtSVb11U9Kcnfb58okGdZYJEmTmz/EY38C+FNgY19tDXBjVa1NsqatXwKcASxrr1OAq4BTkhwNXAqMAQXcnmRTVT3W2lwI3AJsBlYANwxxPNJQLV1z/UjO+8Dat4/kvJobhnYlUlV/C+zZp7wS2NCWNwBn9dU3Vs/NwFFJjgVOB7ZW1Z4WHFuBFW3bkVV1c1UVvaA6C0nSjJrpZyKLqurhtvwIsKgtLwYe6mu3o9Wmqu+YpC5JmkEje7DeriBqJs6VZHWSbUm27d69eyZOKUmHhJkOkUfbrSja312tvhM4rq/dklabqr5kkvqkqmpdVY1V1djChQunPQhJUs9Mh8gmYGKG1Srgur76eW2W1qnAE+221xZgeZIFbSbXcmBL27Y3yaltVtZ5fceSJM2Qoc3OSvIZ4C3AMUl20JtltRa4NskFwIPA2a35ZuBMYBz4EXA+QFXtSfJh4LbW7rKqmnhYfxG9GWCH05uV5cwsSZphQwuRqjp3P5tOm6RtARfv5zjrgfWT1LcBr5lOHyVJ0+Mn1iVJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSpM0NEktSZISJJ6swQkSR1ZohIkjozRCRJnRkikqTODBFJUmeGiCSps/mj7oCk0Vq65vqRnfuBtW8f2bl1cHglIknqbNZfiSRZAXwUmAd8vKrWDutco/w/NmkuGtW/Ka+ADp5ZfSWSZB7wMeAM4ATg3CQnjLZXknTomNUhApwMjFfV/VX1NHANsHLEfZKkQ8Zsv521GHiob30HcMqI+iJplnAywcEz20NkIElWA6vb6pNJvj3K/kziGODvR92JIZvrY3R8s9+MjDF/OOwz7Nd0xvdL+9sw20NkJ3Bc3/qSVvsZVbUOWDdTnTpQSbZV1dio+zFMc32Mjm/2m+tjHNb4ZvszkduAZUmOT3IYcA6wacR9kqRDxqy+EqmqZ5K8B9hCb4rv+qq6Z8TdkqRDxqwOEYCq2gxsHnU/pukFe6vtIJrrY3R8s99cH+NQxpeqGsZxJUmHgNn+TESSNEKGyIgleSDJ3UnuTLJt1P05GJKsT7IryTf7akcn2Zpke/u7YJR9nI79jO9DSXa29/HOJGeOso/TkeS4JDcluTfJPUne2+pz4j2cYnxz6T18SZJbk3yjjfH3W/34JLckGU/y2TYhaXrn8nbWaCV5ABirqjkzBz/JbwBPAhur6jWt9t+APVW1NskaYEFVXTLKfna1n/F9CHiyqv77KPt2MCQ5Fji2qr6e5GXA7cBZwL9nDryHU4zvbObOexjgiKp6MsmLgb8D3gu8H/hiVV2T5H8A36iqq6ZzLq9EdNBV1d8Ce/YprwQ2tOUN9P7Rzkr7Gd+cUVUPV9XX2/IPgPvofTvEnHgPpxjfnFE9T7bVF7dXAW8FPt/qB+U9NERGr4C/TnJ7+2T9XLWoqh5uy48Ai0bZmSF5T5K72u2uWXmrZ19JlgJvAG5hDr6H+4wP5tB7mGRekjuBXcBW4DvA41X1TGuyg4MQnobI6L25qk6k903EF7dbJXNa9e6hzrX7qFcBvwy8HngY+KPRdmf6kvw88AXgfVW1t3/bXHgPJxnfnHoPq+rHVfV6et/kcTLw6mGcxxAZsara2f7uAr5E782eix5t96In7knvGnF/DqqqerT9o/0J8BfM8vex3Uf/AvCpqvpiK8+Z93Cy8c2193BCVT0O3AS8CTgqycTnAyf9mqgDZYiMUJIj2oM9khwBLAe+OfVes9YmYFVbXgVcN8K+HHQT/3FtfptZ/D62h7JXA/dV1Uf6Ns2J93B/45tj7+HCJEe15cOBt9F79nMT8M7W7KC8h87OGqEkr6R39QG9bw/4dFVdPsIuHRRJPgO8hd63hj4KXAp8GbgWeAXwIHB2Vc3Kh9P7Gd9b6N0GKeAB4N19zw9mlSRvBv4PcDfwk1b+IL3nBrP+PZxifOcyd97D19J7cD6P3sXCtVV1WftvzjXA0cAdwLuq6qlpncsQkSR15e0sSVJnhogkqTNDRJLUmSEiSerMEJEkdWaISJI6M0QkSZ0ZIpKkzv4/2LyLCkd/AwYAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vL87NPSgY-5K",
        "outputId": "139440b7-918a-4f80-f26e-dd428c335920"
      },
      "source": [
        "# Check the coverage of a \"total_lines\" value of 20\n",
        "np.percentile(train_df.total_lines, 98) # a value of 20 covers 98% of samples"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "20.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gNafwWXDZPce",
        "outputId": "4cef5df3-c2df-4bd4-dfdb-acd59167db7d"
      },
      "source": [
        "# Use TensorFlow to create one-hot-encoded tensors of our \"total_lines\" column \n",
        "train_total_lines_one_hot = tf.one_hot(train_df[\"total_lines\"].to_numpy(), depth=20)\n",
        "val_total_lines_one_hot = tf.one_hot(val_df[\"total_lines\"].to_numpy(), depth=20)\n",
        "test_total_lines_one_hot = tf.one_hot(test_df[\"total_lines\"].to_numpy(), depth=20)\n",
        "\n",
        "# Check shape and samples of total lines one-hot tensor\n",
        "train_total_lines_one_hot.shape, train_total_lines_one_hot[:10]"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([180040, 20]), <tf.Tensor: shape=(10, 20), dtype=float32, numpy=\n",
              " array([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0.,\n",
              "         0., 0., 0., 0.]], dtype=float32)>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ejQ1zA2nZXtm"
      },
      "source": [
        "### Building a tribid embedding model\n",
        "\n",
        "We're going to go through the following steps:\n",
        "\n",
        "1. Create a token-level model (similar to `model_1`)\n",
        "2. Create a character-level model (similar to `model_3` with a slight modification to reflect the paper)\n",
        "3. Create a \"`line_number`\" model (takes in one-hot-encoded \"`line_number`\" tensor and passes it through a non-linear layer)\n",
        "4. Create a \"`total_lines`\" model (takes in one-hot-encoded \"`total_lines`\" tensor and passes it through a non-linear layer)\n",
        "5. Combine (using `layers.Concatenate`) the outputs of 1 and 2 into a token-character-hybrid embedding and pass it series of output to Figure 1 and section 4.2 of Neural Networks for Joint Sentence Classification in Medical Paper Abstracts\n",
        "6. Combine (using `layers.Concatenate`) the outputs of 3, 4 and 5 into a token-character-positional tribrid embedding\n",
        "7. Create an output layer to accept the tribrid embedding and output predicted label probabilities\n",
        "8. Combine the inputs of 1, 2, 3, 4 and outputs of 7 into a tf.keras.Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6PPo_dYnZXah"
      },
      "source": [
        "# 1. Token inputs\n",
        "token_inputs = layers.Input(shape=[], dtype=\"string\", name=\"token_inputs\")\n",
        "token_embeddings = tf_hub_embedding_layer(token_inputs)\n",
        "token_outputs = layers.Dense(128, activation=\"relu\")(token_embeddings)\n",
        "token_model = tf.keras.Model(inputs=token_inputs,\n",
        "                             outputs=token_embeddings)\n",
        "\n",
        "# 2. Char inputs\n",
        "char_inputs = layers.Input(shape=(1,), dtype=\"string\", name=\"char_inputs\")\n",
        "char_vectors = char_vectorizer(char_inputs)\n",
        "char_embeddings = char_embed(char_vectors)\n",
        "char_bi_lstm = layers.Bidirectional(layers.LSTM(32))(char_embeddings)\n",
        "char_model = tf.keras.Model(inputs=char_inputs,\n",
        "                            outputs=char_bi_lstm)\n",
        "\n",
        "# 3. Line numbers inputs\n",
        "line_number_inputs = layers.Input(shape=(15,), dtype=tf.int32, name=\"line_number_input\")\n",
        "x = layers.Dense(32, activation=\"relu\")(line_number_inputs)\n",
        "line_number_model = tf.keras.Model(inputs=line_number_inputs,\n",
        "                                   outputs=x)\n",
        "\n",
        "# 4. Total lines inputs\n",
        "total_lines_inputs = layers.Input(shape=(20,), dtype=tf.int32, name=\"total_lines_input\")\n",
        "y = layers.Dense(32, activation=\"relu\")(total_lines_inputs)\n",
        "total_line_model = tf.keras.Model(inputs=total_lines_inputs,\n",
        "                                  outputs=y)\n",
        "\n",
        "# 5. Combine token and char embeddings into a hybrid embedding\n",
        "combined_embeddings = layers.Concatenate(name=\"token_char_hybrid_embedding\")([token_model.output, \n",
        "                                                                              char_model.output])\n",
        "z = layers.Dense(256, activation=\"relu\")(combined_embeddings)\n",
        "z = layers.Dropout(0.5)(z)\n",
        "\n",
        "# 6. Combine positional embeddings with combined token and char embeddings into a tribrid embedding\n",
        "z = layers.Concatenate(name=\"token_char_positional_embedding\")([line_number_model.output,\n",
        "                                                                total_line_model.output,\n",
        "                                                                z])\n",
        "# 7. Create output layer\n",
        "output_layer = layers.Dense(5, activation=\"softmax\", name=\"output_layer\")(z)\n",
        "\n",
        "# 8. Put together model\n",
        "model_5 = tf.keras.Model(inputs=[line_number_model.input,\n",
        "                                 total_line_model.input,\n",
        "                                 token_model.input, \n",
        "                                 char_model.input],\n",
        "                         outputs=output_layer)"
      ],
      "execution_count": 88,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "548OtDhVZXXN",
        "outputId": "d374c4ae-ccc6-4115-ccaf-c160f089d8c1"
      },
      "source": [
        "# Get a summary of our token, char and positional embedding model\n",
        "model_5.summary()"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_8\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "char_inputs (InputLayer)        [(None, 1)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "char_vectorizer (TextVectorizat (None, 290)          0           char_inputs[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "token_inputs (InputLayer)       [(None,)]            0                                            \n",
            "__________________________________________________________________________________________________\n",
            "char_embed (Embedding)          (None, 290, 25)      1750        char_vectorizer[2][0]            \n",
            "__________________________________________________________________________________________________\n",
            "universal_sentence_encoder (Ker (None, 512)          256797824   token_inputs[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_1 (Bidirectional) (None, 64)           14848       char_embed[2][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "token_char_hybrid_embedding (Co (None, 576)          0           universal_sentence_encoder[2][0] \n",
            "                                                                 bidirectional_1[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "line_number_input (InputLayer)  [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "total_lines_input (InputLayer)  [(None, 20)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "dense_10 (Dense)                (None, 256)          147712      token_char_hybrid_embedding[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "dense_8 (Dense)                 (None, 32)           512         line_number_input[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "dense_9 (Dense)                 (None, 32)           672         total_lines_input[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "dropout_2 (Dropout)             (None, 256)          0           dense_10[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "token_char_positional_embedding (None, 320)          0           dense_8[0][0]                    \n",
            "                                                                 dense_9[0][0]                    \n",
            "                                                                 dropout_2[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "output_layer (Dense)            (None, 5)            1605        token_char_positional_embedding[0\n",
            "==================================================================================================\n",
            "Total params: 256,964,923\n",
            "Trainable params: 167,099\n",
            "Non-trainable params: 256,797,824\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 707
        },
        "id": "0Y4xa9pDZXUQ",
        "outputId": "90597d89-2c71-494b-fb52-073ef360cbb6"
      },
      "source": [
        "# Plot the token, char, positional embedding model\n",
        "from tensorflow.keras.utils import plot_model\n",
        "plot_model(model_5)"
      ],
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAA/sAAANHCAYAAABpcE00AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nOzdaXxU5f3///dknUwgk7AGCGEJKqBoFbAUcUGrAlYEghABFSoKboCCUnEpCqioBb6yWK2IP6HFJGBRwa1aFa2IK6IIiFhBBAxbEkyALHz+N/wzdcg2JJNMcvJ6Ph65kXPOXOdznTlz5bwzZ3GZmQkAAAAAADhFZlioKwAAAAAAAMFF2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAh4kIdQFOd+WVV4a6BKDOyszMDHUJAAAAQJ3k4m781cvlcqlHjx5KSkoKdSlAnbFjxw59+OGHYngCAAAAKiWTb/ZrwG233aYhQ4aEugygzsjIyNDQoUNDXQYAAABQZ3HNPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4DGEfAAAAAACHIewDAAAAAOAwhH0AAAAAAByGsA8AAAAAgMMQ9gEAAAAAcBjCPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4DGEfAAAAAACHIewDAAAAAOAwhH0AAAAAAByGsF+HjBo1Sm63Wy6XS4cPHw5pLa+88oq8Xq9efvnlkNZRXT788EN16tRJYWFhcrlcat68uaZPnx7qsvwsX75c7du3l8vlksvlUmJiokaMGBHqsgAAAADUAhGhLgCBW7RokVq1aqUZM2aEuhSZWahLqFY9evTQxo0b1adPH73++uvavHmz4uPjQ12Wn9TUVKWmpqpDhw7au3evdu/eHeqSAAAAANQSfLOPSrnsssuUk5Ojyy+/PNSl6NChQ+rZs2eoy6h29aWfAAAAAKqOsF9HuVyuUJdQayxcuFBZWVmhLqPa1Zd+AgAAAKg6wn4ttHjxYnXr1k1ut1uxsbFq27atpk2b5psfFhamVatWqW/fvvJ6vWrRooWeeeYZvzbee+89de7cWV6vV263W126dNHrr78uSXrkkUfk8XjUsGFDZWVlaeLEiWrVqpU2b94cUH3vv/++kpOT5XK5NG/ePEnSggULFBsbK4/HoxdffFF9+/ZVXFyckpKStHTpUt9rH3/8cbndbjVr1kxjx45VixYt5Ha71bNnT61du9a33Lhx4xQVFaXExETftJtvvlmxsbFyuVzau3evJGnChAmaOHGitm7dKpfLpQ4dOkiS3n33XZ199tnyeDyKi4tTly5dlJubK0l67bXXFBcXV6nLIWpbP09UefvF6NGjfdf/p6Sk6PPPP5f0y70iPB6PvF6vXnrpJUlScXGx7rvvPiUnJysmJkann3660tPTJVV9/wIAAAAQBIZqJcnS09MDXn727NkmyR566CHbt2+f7d+/35588kkbPny4mZndfffdJsneeusty87Otv3791u/fv0sOjra8vLyfO1kZmba1KlTbf/+/bZv3z7r0aOHNW7c2Df/WDvjx4+3uXPn2qBBg2zjxo0B1/nDDz+YJJs7d26JNt966y3LycmxrKwsO/fccy02NtYKCgp8y40ZM8ZiY2Pt66+/tsOHD9uGDRuse/fu1rBhQ9u+fbtvueHDh1vz5s391vvoo4+aJNuzZ49vWmpqqqWkpPh+//nnny0uLs5mzpxphw4dst27d9ugQYN8r1m5cqU1bNjQHnjggQr7eemll5okO3DgQK3r5zEpKSnm9Xor7ItZxftFamqqhYeH248//uj3umHDhtlLL73k+33SpEkWHR1ty5YtswMHDtiUKVMsLCzMPv74Y79tVNn9Kz093RieAAAAgErL4Jv9WqSwsFD333+/evfurT/96U9q1KiREhISdN1116l79+5+y/bs2VNer1cJCQlKS0vTkSNH9N///tc3f/Dgwfrzn/+shIQENWrUSP3799e+ffu0Z88ev3Yefvhh3XLLLVq+fLk6duwYlH707NlTcXFxatq0qdLS0pSXl6ft27f7LRMREaFOnTopOjpanTt31oIFC3Tw4EEtWrSoyuv//vvvlZubq1NPPVVut1vNmzfX8uXL1aRJE0m/3G8gNzdX9957b5XWE+p+VkZF+8WNN96o4uJiv/pyc3P18ccfq1+/fpKkw4cPa8GCBRo4cKBSU1MVHx+ve+65R5GRkSX6VR37FwAAAICKEfZrkfXr1ys7O1uXXnqp3/Tw8HCNHz++zNdFRkZK+uWfBRUtU1xcHIRKAxcVFSWp/NokqVu3bvJ4PNq0aVOV19m+fXs1a9ZMI0aM0NSpU/X9999Xuc2KhKKfwXD8fnHhhRfq5JNP1jPPPON74sLzzz+vtLQ0hYeHS5I2b96s/Px8nXbaab52YmJilJiYWGv6BQAAANR3hP1a5Ng15cF4xNuqVat0wQUXqGnTpoqOjtadd95Z5TarW3R0dIkzDyojJiZG//73v9WrVy/NmDFD7du3V1pamg4dOhSEKqsuWP2sjIr2C5fLpbFjx+q7777TW2+9JUl67rnndN111/mWycvLkyTdc889vmv8XS6Xtm3bpvz8/JrrDAAAAIAyEfZrkZYtW0qS76ZslbV9+3YNHDhQiYmJWrt2rXJycjRz5sxglFhtCgsLlZ2draSkpKC0d+qpp+rll1/Wzp07NXnyZKWnp+uxxx4LSttVEex+VmT16tWaPXu2pMD3i5EjR8rtduvpp5/W5s2bFRcXpzZt2vjmN23aVJI0e/ZsmZnfz5o1a2qkXwAAAADKR9ivRdq2batGjRrpjTfeqFI7X375pQoLC3XTTTepffv2crvdtf5Rfe+8847MTD169PBNi4iIqPC0+NLs3LlTX3/9taRfgulDDz2ks846yzctlILZz0B8+umnio2NlRT4fpGQkKChQ4dqxYoVeuyxx3T99df7zW/durXcbrfWrVtXLTUDAAAAqDrCfi0SHR2tKVOmaPXq1Ro3bpx+/PFHHT16VAcPHjyhoJqcnCxJevPNN3X48GFt2bLF73FvtcHRo0d14MABFRUVaf369ZowYYKSk5M1cuRI3zIdOnTQ/v37tWLFChUWFmrPnj3atm1bibYaNWqknTt36vvvv9fBgwe1bds2jR07Vps2bVJBQYE+//xzbdu2zRewX3311Uo/eq829bO8fxAUFhbqp59+0jvvvOML+yeyX9x44406cuSIVq5cqcsvv9xvntvt1qhRo7R06VItWLBAubm5Ki4u1o4dO7Rr164T3UQAAAAAqkMIHwVQL+gEH71nZjZv3jzr0qWLud1uc7vdduaZZ9r8+fNt5syZFhMTY5LspJNOsq1bt9qSJUssISHBJFlSUpJ99dVXZmY2efJka9SokcXHx9uVV15p8+bNM0mWkpJit9xyi6+d1q1b2+LFi0+ovrlz51piYqJJMo/HY/3797f58+ebx+Pxq+2pp56yuLg4k2Rt2rSxb775xsx+eSRdZGSktWrVyiIiIiwuLs4GDBhgW7du9VvPvn37rHfv3uZ2u61du3Z266232h133GGSrEOHDr7H13322WfWpk0bi4mJsV69etnatWutZ8+elpCQYOHh4dayZUu7++67raioyMzMXnnlFWvYsKFNnz69zD5++OGHduqpp1pYWJhJssTERJsxY0at6ucTTzxhKSkpJqncnxdeeMG3rvL2i18/DtDM7Mwzz7S77rqr1O1z5MgRmzx5siUnJ1tERIQ1bdrUUlNTbcOGDX77aWX2LzMevQcAAABUUYbL7P+/5TaqhcvlUnp6uoYMGRLqUmqNsWPHKjMzU/v27Qt1KdWqrvfzsssu07x589SuXbsaX3dGRoaGDh0qhicAAACgUjI5jR8hUdOPAAyVutTPX18WsH79ernd7pAEfQAAAABVR9iHJGnTpk1+j1Er6yctLS3UpaKaTJ48WVu2bNE333yjUaNGadq0aaEuCQAAAEAlEfYhSerYsWOJx6iV9vP8889XaT1TpkzRokWLlJOTo3bt2mnZsmVB6kHtUhf76fF41LFjR/3+97/X1KlT1blz51CXBAAAAKCSuGa/mnHNPnDiuGYfAAAAqBKu2QcAAAAAwGkI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAcxmVmFuoinMzlcqlHjx5KSkoKdSlAnbFjxw59+OGHYngCAAAAKiUzItQVON3gwYNDXQLK8Mknn0iSunXrFuJKcLykpCQ+OwAAAEAV8M0+6q0hQ4ZIkjIyMkJcCQAAAAAEVSbX7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DARoS4AqAnPPvus5syZo+LiYt+0PXv2SJK6dOnimxYeHq4JEyZo5MiRNV0iAAAAAASNy8ws1EUA1W3z5s3q2LFjQMtu3Lgx4GUBAAAAoBbK5DR+1AunnHKKunTpIpfLVeYyLpdLXbp0IegDAAAAqPMI+6g3rrnmGoWHh5c5PyIiQtdee20NVgQAAAAA1YPT+FFv7Ny5U0lJSSprl3e5XNq+fbuSkpJquDIAAAAACCpO40f90bJlS/Xs2VNhYSV3+7CwMPXs2ZOgDwAAAMARCPuoV66++upSr9t3uVy65pprQlARAAAAAAQfp/GjXtm/f7+aN2+uoqIiv+nh4eH66aef1Lhx4xBVBgAAAABBw2n8qF8aNWqkiy++WBEREb5p4eHhuvjiiwn6AAAAAByDsI96Z8SIETp69KjvdzPT1VdfHcKKAAAAACC4OI0f9U5eXp6aNGmiw4cPS5Kio6O1d+9eNWjQIMSVAQAAAEBQcBo/6p/Y2Fj1799fkZGRioiI0IABAwj6AAAAAByFsI96afjw4SoqKlJxcbGGDRsW6nIAAAAAIKgiKl6kbsjIyAh1CahDiouL5Xa7ZWb6+eef2X9wQoYMGRLqEgAAAIByOeaa/dKenQ4A1cEhwyYAAACcK9Mx3+xLUnp6Ot+4IWBvv/22XC6XLrjgglCXgjoiIyNDQ4cODXUZAAAAQIUcFfaBE3H++eeHugQAAAAAqBaEfdRbYWHcnxIAAACAM5F2AAAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYb8Uo0aNktvtlsvl0uHDh0NdTp31yiuvyOv16uWXXw51KZKktLQ0uVyugH5WrlxZbXWMGTNGsbGxcrlcioyM1BlnnKGNGzf6LfPMM88oOTlZLpdLzZs317PPPltt9VRWTb2/tW0/AgAAAOoCwn4pFi1apEmTJoW6jDrPzEJdQglvvPGGsrOzVVhYqF27dkmS+vfvr4KCAuXl5SkrK0vXX399tdbw5JNPas2aNZKkrl276osvvlCnTp38lvnjH/+o9957Ty1bttSOHTs0cuTIaq2pMmrq/a2N+xEAAABQ2xH2IUk6dOiQevbsGdQ2L7vsMuXk5Ojyyy8ParuV5XK5dM4558jr9SoiIsJvemRkpDwej5o2baquXbsGdb2lbdvTTz9dvXr10tq1a/XZZ5+V+rq//vWv+uMf/6jIyMhqqaGqquP9La3O2rYfAQAAAHUBYb8CLpcr1CXUiIULFyorKyvUZVSrpUuXyuPxVLjcmDFj9Ic//CFo6y1r295yyy2SpPnz55eYV1BQoOeee05jxoyp1hpqm7pSJwAAAFDb1euwv3jxYnXr1k1ut1uxsbFq27atpk2b5psfFhamVatWqW/fvvJ6vWrRooWeeeYZvzbee+89de7cWV6vV263W126dNHrr78uSXrkkUfk8XjUsGFDZWVlaeLEiWrVqpU2b94cUH2dOnWSy+VSWFiYunbtqvz8fEnSnXfe6VvfsWu5i4uLdd999yk5OVkxMTE6/fTTlZ6eHlB/J0yYoIkTJ2rr1q1yuVzq0KGDpF9On541a5Y6deqk6OhoJSQkaMCAAdq0aZOvzbL6uHDhQt815/PmzZMkffvtt2VeI/+vf/2rwn6Utz1fe+01xcXFacaMGQFt20CUV8uzzz6rBg0ayOVyKSEhQStWrNAnn3yiNm3aKDw8XMOGDZOkMretJKWmpqply5Z6/vnnlZ2d7bfuZcuW6be//a2SkpIc9f6W93kprc7333+/xHoCrX3BggWKjY2Vx+PRiy++qL59+youLk5JSUlaunRpFfYMAAAAoA4wh5Bk6enpAS8/e/Zsk2QPPfSQ7du3z/bv329PPvmkDR8+3MzM7r77bpNkb731lmVnZ9v+/futX79+Fh0dbXl5eb52MjMzberUqbZ//37bt2+f9ejRwxo3buybf6yd8ePH29y5c23QoEG2cePGgGosKiqytm3bWnJyshUVFfnNu+2222z27Nm+3ydNmmTR0dG2bNkyO3DggE2ZMsXCwsLs448/Dqi/qamplpKS4reO++67z6Kiomzx4sWWnZ1t69evt7POOsuaNGliu3fvrrCPP/zwg0myuXPnmpnZli1b7E9/+pNv++3atcsSEhKsZ8+eVlxcHFA/ylrXypUrrWHDhvbAAw8EtG2PrV+SXXHFFaXOr6iWr7/+2jwej1177bW+19x111329NNP+7VT2rY9ZurUqSbJZs2a5Te9V69e9uabbwZcS115fyv6vJRW5/HrqUztb731luXk5FhWVpade+65FhsbawUFBaW+J+VJT083Bw2bAAAAcK4Mxxy1nkjYLygosPj4eOvdu7ff9KKiIpszZ46Z/S8kHDp0yDf/ueeeM0n21Vdfldn2gw8+aJIsKyurzHZOxLEQl5GR4ZuWl5dnycnJlpOTY2Zmhw4dMo/HY2lpab5l8vPzLTo62m666aaA+nt8yMrPz7cGDRr4tWlm9tFHH5kkv1BdVh9LC2m/NnDgQHO73bZp06aA+lHeuiqjvLAfSC1mZk8++aRJsiVLltg//vEPu/3220u0VV7Y37Vrl0VGRtrJJ59sR48eNTOz9evXW8eOHQOupa68v6U5/vMSSNivau3z5883Sfbtt9+WWVdZCPsAAACoIzLq5Wn869evV3Z2ti699FK/6eHh4Ro/fnyZrzt2o7TCwsIKlykuLg5CpdLo0aPl9Xo1Z84c37QlS5ZowIABiouLkyRt3rxZ+fn5Ou2003zLxMTEKDExUZs2bapUfzds2KCff/5Z3bp185vevXt3RUVFae3atVXqV0ZGhv75z3/q/vvv1ymnnBJQP2pSoLXccMMNGjx4sMaOHauMjAw98sgjJ7SexMREpaam6ptvvtGbb74pSXriiSd04403BlxLXXl/S1OZz0tVa4+KipJU/ucYAAAAqOvqZdjPzc2VJMXHx1e5rVWrVumCCy5Q06ZNFR0drTvvvLPKbf5agwYNdMMNN+iDDz7QRx99JOmXMDhu3DjfMnl5eZKke+65x+866W3btik/P79S/T12DXmDBg1KzIuPj9fBgwcr3ad9+/bp1ltvVffu3TVx4sSA+1GTTqSWGTNm6Oeff670jeWO3ahvwYIFOnjwoP75z3/q2muvDbiWuvL+SsH5vFRn7QAAAIBT1Muw37JlS0nS3r17q9TO9u3bNXDgQCUmJmrt2rXKycnRzJkzg1Gin3HjxikyMlKzZ8/W6tWr1bp1a6WkpPjmN23aVJI0e/ZsmZnfz5o1ayrV32PBsbTglJ2d7btxXGWMHz9e2dnZWrRokcLDwwPuR00KtJbCwkKNHz9es2bN0po1azR9+vQTXtc555yjM888Uy+//LIeeughXXHFFfJ6vQHXUlfe32B9XqqzdgAAAMAp6mXYb9u2rRo1aqQ33nijSu18+eWXKiws1E033aT27dvL7XZXy6P6kpKSNGTIEC1btkz33nuvJkyY4De/devWcrvdWrduXamvr0x/TzvtNDVo0ECffPKJ3/S1a9eqoKCg0s+iX7Vqlf7+97/r3nvv1amnnuqbfscdd1TYj5oUaC233nqrrr/+et122226/fbbNW3atEr9Y+Lmm29WcXGxHn74Yd10000nVEtdeX+D9XmprtoBAAAAJ6mXYT86OlpTpkzR6tWrNW7cOP344486evSoDh48qK+//jrgdpKTkyVJb775pg4fPqwtW7ZU+VrnskycOFFFRUU6cOCALrzwQr95brdbo0aN0tKlS7VgwQLl5uaquLhYO3bs0K5duwLqb6NGjbRz5059//33OnjwoMLDwzVx4kS98MILWrJkiXJzc/Xll1/qxhtvVIsWLSr1/Pfc3FyNHTtWv/nNb/SnP/1JknT48GF98sknWrduXYX9KM+rr74a1EfvBVLL/Pnz1apVKw0aNEiS9OCDD6pz584aPny479R6qeS2Le1a8WHDhqlRo0Y655xzdPrpp59QLXXl/Q3k8xLItnK73UGvHQAAAHCcmr8pYPXQCT56z8xs3rx51qVLF3O73eZ2u+3MM8+0+fPn28yZMy0mJsYk2UknnWRbt261JUuWWEJCgkmypKQk3x35J0+ebI0aNbL4+Hi78sorbd68eSbJUlJS7JZbbvG107p1a1u8eHGV+ti7d+8Sj3U75siRIzZ58mRLTk62iIgIa9q0qaWmptqGDRsq7K+Z2WeffWZt2rSxmJgY69Wrl+3evduOHj1qjz76qJ100kkWGRlpCQkJNnDgQNu8ebOvzV9vq1/3ce7cuZaYmGiSzOPxWP/+/e2xxx4zSaX+9OvXr8J+lLUuM7NXXnnFGjZsaNOnT69wO+bm5tp5551njRo1MkkWFhZmHTp0sBkzZgS8TS+//HJzuVzWqFEj++CDD8zsl8chhoWFmSTzer32ySeflLltS3PHHXfYP/7xD0e/v+V9XrZv316iznvuuafEeswsoNrnz59vHo/H73P81FNPWVxcnEmyNm3a2DfffFPh/vJr3I0fAAAAdUSGy8ysRv6rUM1cLpfS09M1ZMiQUJcCwKEyMjI0dOhQOWTYBAAAgHNl1svT+AEAAAAAcDLCfg3btGmT3+PTyvpJS0sLdakAAAAAgDoqItQF1DcdO3bkFGAAAAAAQLXim30AAAAAAByGsA8AAAAAgMMQ9gEAAAAAcBjCPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4DGEfAAAAAACHIewDAAAAAOAwhH0AAAAAAByGsA8AAAAAgMMQ9gEAAAAAcBjCPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4TESoCwimNWvWhLoEAA7GGAMAAIC6wmVmFuoigsHlcoW6BAD1hEOGTQAAADhXpmO+2efgGydqyJAhkqSMjIwQVwIAAAAAwcU1+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOExEqAsAasK7776rDz/80G/apk2bJEkzZ870m96jRw+df/75NVYbAAAAAASby8ws1EUA1e1f//qXLrnkEkVGRiosrPQTWo4eParCwkK98cYbuvjii2u4QgAAAAAImkzCPuqF4uJiNW/eXPv27St3uYSEBGVlZSkigpNeAAAAANRZmVyzj3ohPDxcw4cPV1RUVJnLREVF6eqrryboAwAAAKjzCPuoN6666ioVFBSUOb+goEBXXXVVDVYEAAAAANWD0/hRr7Rp00bbt28vdV5SUpK2b98ul8tVw1UBAAAAQFBxGj/qlxEjRigyMrLE9KioKF177bUEfQAAAACOQNhHvTJixAgVFhaWmF5QUKC0tLQQVAQAAAAAwUfYR73SqVMnderUqcT0jh076rTTTgtBRQAAAAAQfIR91DvXXHON36n8kZGRuvbaa0NYEQAAAAAEFzfoQ72zfft2tW3bVsd2fZfLpe+++05t27YNbWEAAAAAEBzcoA/1T3Jysrp166awsDC5XC51796doA8AAADAUQj7qJeuueYahYWFKTw8XFdffXWoywEAAACAoOI0ftRLe/bsUYsWLSRJP/74o5o3bx7iigAAAAAgaDIjQl2B0/Cc9ronMTEx1CUgQPxvEqHE+A5Uv8GDByszMzPUZQCAIxD2q8GECRP0u9/9LtRloALvvvuuXC6XzjvvvFCXggqsWbNGc+bMCXUZAOM7UI1mz54d6hIAwFEI+9Xgd7/7nYYMGRLqMlCBPn36SJLi4uJCXAkCQdhHbcD4DlQfvtEHgOAi7KPeIuQDAAAAcCruxg8AAAAAgMMQ9gEAAAAAcBjCPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4DGEfAAAAAACHIewDAAAAAOAwhH0AAAAAAByGsA8AAAAAgMMQ9gEAAAAAcBjCPgAAAAAADkPYBwAAAADAYQj7AAAAAAA4DGG/Fhs1apTcbniA62UAACAASURBVLdcLpcOHz4c6nKqRffu3RUeHq7f/OY3QW979OjRatiwoVwul9atW3dCr12+fLnat28vl8tV5k/btm2DUmdt2AZlLffKK6/I6/Xq5ZdfDnptAEr30EMPyev1Vmrsqm5OHxM+/PBDderUSWFhYXK5XGrevLmmT58e6rL8HP/3KTExUSNGjAh1WQCAWoiwX4stWrRIkyZNCnUZ1erjjz9W7969q6Xtp59+Wn/7298q9drU1FR99913SklJkdfrlZnJzFRUVKT8/Hz99NNP8ng8QamzNmyDspYzs+ooC0A57rrrLj355JOhLqNUTh8TevTooY0bN+qSSy6RJG3evFn33HNPiKvyd/zfp927d2vJkiWhLgsAUAsR9lEruFyuUJcQkPDwcMXExKhZs2Y6+eSTg9p2bdwGl112mXJycnT55ZeHuhSgzjl06JB69uwZ6jKCqjaNCU7cvqWpL/0EAAQfYb+OqI1BMJgiIyOrpd3q3G4rVqwIanuh3gY1sY+ZmTIzM/XUU09V+7qAUFu4cKGysrJCXYZj1ZftW1/6CQAIPsJ+LbB48WJ169ZNbrdbsbGxatu2raZNm+abHxYWplWrVqlv377yer1q0aKFnnnmGb823nvvPXXu3Fler1dut1tdunTR66+/Lkl65JFH5PF41LBhQ2VlZWnixIlq1aqVNm/eHHCNxcXFuu+++5ScnKyYmBidfvrpSk9PlyTNmTNHsbGxCgsLU9euXdW8eXNFRkYqNjZWZ511ls4991y1bt1abrdb8fHxuvPOO0u0/+2336pjx46KjY1VTEyMzj33XL3//vsB1yD9EiQfffRRnXLKKYqOjpbX69Udd9xRYl2vvfaa4uLiNGPGjID7X5G6tg0CWe79999XcnKyXC6X5s2bJ0lasGCBYmNj5fF49OKLL6pv376Ki4tTUlKSli5dWqLWBx98UKeccopiYmLUpEkTtWvXTg8++KCGDBlS6W0N1AUTJkzQxIkTtXXrVrlcLnXo0EHSL5+9WbNmqVOnToqOjlZCQoIGDBigTZs2ldveTz/9pLZt2yoiIkJ9+vTxTS9vTDiRz2sgqjImPP7443K73WrWrJnGjh2rFi1ayO12q2fPnlq7dq1vuXHjxikqKkqJiYm+aTfffLNiY2Plcrm0d+/ecrfvu+++q7PPPlsej0dxcXHq0qWLcnNzJVVt7K9t/TxR5R0jjB492nf9f0pKij7//HNJv9w3yOPxyOv16qWXXpJU/v4WjGMNAECQGYJKkqWnpwe8/OzZs02SPfTQQ7Zv3z7bv3+/PfnkkzZ8+HAzM7v77rtNkr311luWnZ1t+/fvt379+ll0dLTl5eX52snMzLSpU6fa/v37bd++fdajRw9r3Lixb/6xdsaPH29z5861QYMG2caNGwOuc9KkSRYdHW3Lli2zAwcO2JQpUywsLMw+/vhjMzP785//bJJs7dq1lpeXZ3v37rU+ffqYJFu1apXt2bPH8vLybNy4cSbJ1q1b52v7oosusvbt29t///tfKywstK+++sp++9vfmtvttm+++SbgGu6++25zuVz2l7/8xQ4cOGD5+fk2f/58k2Sff/65r52VK1daw4YN7YEHHqiw3ykpKeb1ev2mjR8/3r788ssSy9albRDocj/88INJsrlz5/q99tg+mZOTY1lZWXbuuedabGysFRQU+JabMWOGhYeH24svvmj5+fn26aefWvPmze2CCy6ocLsfLz093RiuEGonOr6npqZaSkqK37T77rvPoqKibPHixZadnW3r16+3s846y5o0aWK7d+/2Lbd06VK/z2NBQYGlpqbaiy++6NdeIGNCIJ/XQFVlTBgzZozFxsba119/bYcPH7YNGzZY9+7drWHDhrZ9+3bfcsOHD7fmzZv7rffRRx81SbZnz54yt+/PP/9scXFxNnPmTDt06JDt3r3bBg0a5HvNiYz9l156qUmyAwcO1Lp+HlPa36eyVHSMkJqaauHh4fbjjz/6vW7YsGH20ksv+X4PdH+r7LHG4MGDbfDgwQEvDwAoVwZHz0F2IgeDBQUFFh8fb7179/abXlRUZHPmzDGz//3hPHTokG/+c889Z5Lsq6++KrPtBx980CRZVlZWme0E6tChQ+bxeCwtLc03LT8/36Kjo+2mm24ys/8F3YMHD/qW+X//7/+ZJL9g/NFHH5kke/75533TLrroIjvjjDP81rl+/XqTZJMmTQqohvz8fPN4PHbxxRf7tXP8AfOJSklJMUklfsoL+7V9G5zItirvwP7X+9KxfxR8++23vmndu3e3s88+228dN9xwg4WFhdmRI0dKbL/yEPZRG1Q17Ofn51uDBg38PsNm/xsTfh1Cf/15LCwstKuuuspeffVVv9cFMjYH+nkNVFXGhDFjxpQIpx9//LFJsvvvv983rbIh+KuvvjJJtnLlyhPu1/HKC/uh7ucxJxL2j3f8McKbb75pkmz69Om+ZXJycuykk06yoqIiM6v8/nYiCPsAEFQZnMYfQuvXr1d2drYuvfRSv+nh4eEaP358ma87dm13YWFhhcsUFxdXuc7NmzcrPz9fp512mm9aTEyMEhMTyz31NCoqSpJUVFRUoq7yapekLl26yOv1av369QHV8O233yo/P18XXXTRiXewAr++G7+ZlfveHK82boPq2FbH+vnrPh0+fLjEnbuLi4sVGRmp8PDwoK0bqCs2bNign3/+Wd26dfOb3r17d0VFRfmd5n1McXGxhg0bpmbNmvmdvi9VfWyuaAyqikDX0a1bN3k8ngovYwhE+/bt1axZM40YMUJTp07V999/X+U2KxKKfgbD8ccIF154oU4++WQ988wzvnH7+eefV1pamm+8ruz+BgAIHcJ+CB27jjA+Pr7Kba1atUoXXHCBmjZtqujo6FKvCa+svLw8SdI999zj95z5bdu2KT8/P2jrOV5kZKTvAKqiGnbs2CFJatq0abXVc8ycOXP8DnaqU3Vsg5raVv369dOnn36qF198UYcOHdInn3yiFStW6A9/+ANhH/VSdna2JKlBgwYl5sXHx+vgwYMlpt9yyy3asmWL/vrXv+rrr7/2mxeqsTnYoqOjtWfPniq3ExMTo3//+9/q1auXZsyYofbt2ystLU2HDh0KQpVVF6x+VkZFxwgul0tjx47Vd999p7feekuS9Nxzz+m6667zLeOU/Q0A6hPCfgi1bNlSknw34qms7du3a+DAgUpMTNTatWuVk5OjmTNnBqNESf8LhbNnz/b7htvMtGbNmqCt59eKioq0f/9+JScnB1SD2+2WJB05cqRa6gmF6toGNbWtpk6dqgsvvFAjR45UXFycBg0apCFDhuhvf/tbta4XqK2O/WO3tFCfnZ2tpKSkEtOHDBmif/3rX4qPj9c111zjd5ZQKMbmYCssLCyz75Vx6qmn6uWXX9bOnTs1efJkpaen67HHHgtK21UR7H5WZPXq1Zo9e7akwI8RRo4cKbfbraefflqbN29WXFyc2rRp45vvhP0NAOobwn4ItW3bVo0aNdIbb7xRpXa+/PJLFRYW6qabblL79u3ldruD+hi1Y3eRX7duXdDarMjbb7+to0eP6qyzzgqohtNOO01hYWF69913a6zGXbt2adSoUdXWfnVtg5raVhs2bNDWrVu1Z88eFRYWavv27VqwYIESEhKqdb1AbXXaaaepQYMG+uSTT/ymr127VgUFBeratWuJ1/Tu3VtNmjTRU089pU8//VTTp0/3zQvF2Bxs77zzjsxMPXr08E2LiIio1CUGO3fu9J390LRpUz300EM666yzSpwREQrB7GcgPv30U8XGxkoK/BghISFBQ4cO1YoVK/TYY4/p+uuv95vvhP0NAOobwn4IRUdHa8qUKVq9erXGjRunH3/8UUePHtXBgwdP6ODk2De/b775pg4fPqwtW7aUeu1nZbndbo0aNUpLly7VggULlJubq+LiYu3YsUO7du0KyjoKCgqUk5OjoqIiffbZZxo3bpzatGmjkSNHBlRD06ZNlZqaqmXLlmnhwoXKzc3V+vXrS32e+6uvvlqlR++ZmQ4dOqTly5crLi6uKt32U1Pb4ES2VVXccsstSk5O1s8//xzUdoG6olGjRtq5c6e+//57HTx4UOHh4Zo4caJeeOEFLVmyRLm5ufryyy914403qkWLFhozZkyZbfXv318jR47UjBkz9Omnn0qqmbE52I4ePaoDBw6oqKhI69ev14QJE5ScnOwb5ySpQ4cO2r9/v1asWKHCwkLt2bNH27ZtK9HW8dt327ZtGjt2rDZt2qSCggJ9/vnn2rZtmy9gV3Xsry39LO8fBIWFhfrpp5/0zjvv+ML+iRwj3HjjjTpy5IhWrlypyy+/3G9eXdzfAKDeq5kbAdYfOsG7NZuZzZs3z7p06WJut9vcbredeeaZNn/+fJs5c6bFxMSYJDvppJNs69attmTJEktISDBJlpSU5Lsj/+TJk61Ro0YWHx9vV155pc2bN88kWUpKit1yyy2+dlq3bm2LFy8+4X4dOXLEJk+ebMnJyRYREWFNmza11NRU27Bhg82ZM8c8Ho9JsrZt29p7771nDz/8sHm9XpNkzZs3t7///e/2/PPPW/PmzU2SJSQk2NKlS83MbNGiRda7d29r1qyZRUREWOPGje2qq66ybdu2BVyDmdnBgwdt9OjR1rhxY2vQoIH16tXL7rvvPt+2+uKLL8zM7JVXXrGGDRv63XX4eC+88EKZd+L/9c8999xjZlbntkEgy82dO9cSExNNknk8Huvfv7/Nnz/f189j++RTTz1lcXFxJsnatGnje1Tgv//9b2vcuLHf9oqMjLROnTrZ8uXLT2j/4278qA1OdHz/7LPPrE2bNhYTE2O9evWy3bt329GjR+3RRx+1k046ySIjIy0hIcEGDhxomzdv9r1u+fLlvnG+bdu2lpWVZbm5uda6dWuTZA0aNLDnnnvOzMofE07k8xqIqo4JY8aMscjISGvVqpVFRERYXFycDRgwwLZu3eq3nn379lnv3r3N7XZbu3bt7NZbb7U77rjDJFmHDh18j687fvuuXbvWevbsaQkJCRYeHm4tW7a0u+++23c3+UDG/g8//NBOPfVUCwsLM0mWmJhoM2bMqFX9fOKJJwL6+/TCCy/41lXeMcKvHwdoZnbmmWfaXXfdVer2KW9/+/UxS2WPNbgbPwAEVYbL7LjbZaNKXC6X0tPTNWTIkFCXAoTUggULtGXLFt91o9IvZy/86U9/0oIFC3TgwAHFxMQE1FZGRoaGDh1a4u7+QE1ifK+asWPHKjMzU/v27Qt1KdWqrvfzsssu07x589SuXbsaX/eVV14pScrMzKzxdQOAA2VGhLoCAM6ze/dujRs3rsS1nVFRUUpOTlZhYaEKCwsDDvsAnCEYj4OtC+pSPwsLC32P4lu/fr3cbndIgj4AIPi4Zr+e2rRpk9+jc8r6SUtLC3WpqINiYmIUGRmphQsX6qefflJhYaF27typp59+Wvfdd5/S0tKCer8DAJXD3wJMnjxZW7Zs0TfffKNRo0Zp2rRpoS4JABAkfLNfT3Xs2JFTolFtvF6v3njjDT3wwAM6+eSTlZeXpwYNGujUU0/Vww8/rBtuuCHUJQJQzf0tmDJlihYtWqSCggK1a9dOjz76qAYPHlzt661pdbGfHo9HHTt2VKtWrTR//nx17tw51CUBAIKEa/aDjGs6geDjmn3UBozvQPXimn0ACKpMTuMHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHIawDwAAAACAwxD2AQAAAABwGMI+AAAAAAAOQ9gHAAAAAMBhCPsAAAAAADgMYR8AAAAAAIch7AMAAAAA4DCEfQAAAAAAHMZlZhbqIpzE5XKFugTAsRiuEEqM70D1Gzx4sDIzM0NdBgA4QWZEqCtwmvT09FCXgADNnj1bknTbbbeFuBIAdQHje92yZs0azZkzh/etjmndunWoSwAAx+CbfdRbQ4YMkSRlZGSEuBIAQLBlZGRo6NChnBEEAKivMrlmHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOExEqAsAasLevXuVm5vrNy0vL0+S9N133/lNj4uLU5MmTWqsNgBA1Rw6dEi7du3ym/bTTz9JKjnGh4eHq02bNjVWGwAAoeIyMwt1EUB1W7hwoUaPHh3Qsk8//bSuu+66aq4IABAs+/btU2JiooqKiipctk+fPnr11VdroCoAAEIqk9P4US8MGjRIkZGRFS4XGRmpQYMG1UBFAIBgady4sS6++GKFhZV/WONyuZSWllZDVQEAEFqEfdQLCQkJ6tOnjyIiyr5yJSIiQn379lVCQkINVgYACIYRI0aoopMVIyIiNGDAgBqqCACA0CLso94YMWKEiouLy5xfXFysESNG1GBFAIBgueKKKxQdHV3m/IiICPXv319er7cGqwIAIHQI+6g3+vfvr5iYmDLnu91uXXbZZTVYEQAgWGJjY3XFFVeUeclWcXGxhg8fXsNVAQAQOoR91Btut1sDBw4s9UAwMjJSqamp8ng8IagMABAMw4cPV2FhYanzYmJi1Ldv3xquCACA0CHso14ZNmxYqQeChYWFGjZsWAgqAgAES58+fRQXF1diemRkpIYOHSq32x2CqgAACA3CPuqVSy65pNQb8MXHx+v3v/99CCoCAARLZGSkhgwZUuIMLv6hCwCojwj7qFciIiKUlpamqKgo37TIyEgNGzYsoEfzAQBqt9LO4GrcuLF69+4doooAAAgNwj7qnauuukoFBQW+3wsLC3XVVVeFsCIAQLCcf/75atasme/3qKgojRgxQuHh4SGsCgCAmkfYR73Tq1cvtWzZ0vd7YmKizjnnnBBWBAAIlrCwMI0YMcJ3BldBQQH/0AUA1EuEfdQ7LpfLdyAYGRmpa665Ri6XK9RlAQCC5NdncCUlJenss88OcUUAANQ8wj7qpWMHgty0CQCcp1u3bmrXrp0kaeTIkfxDFwBQL0UcP2HNmjWaNWtWKGoBalSDBg0kSdOnTw9xJUD1u/322/W73/2uWtrm7wZqo5iYGEnSRx99pCuvvDLE1QD+MjMzQ10CgHqgxDf7P/zwg5YtWxaKWoAa1aZNG7Vp0ybUZQDVbtmyZfrhhx+qrX3+bqA2at26tbxer+Li4kJdCuCzY8cOxksANabEN/vH8B9HON3WrVslSSkpKSGuBKheNXUKM383UNu8/vrruvTSS0NdBuCTkZGhoUOHhroMAPVEmWEfcDpCPgA4G0EfAFCfcYM+AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGEI+wAAAAAAOAxhHwAAAAAAhyHsAwAAAADgMIR9AAAAAAAchrAPAAAAAIDDEPYBAAAAAHAYwj4AAAAAAA5D2AcAAAAAwGGqHPa7d++u8PBw/eY3v6lw2VdeeUVer1cvv/xymcuMHj1aDRs2lMvl0rp1607otdUp1Ot/7LHH1KxZM7lcLv31r3+tdDtHjx7V7Nmz1bNnzwqXffPNN3XXXXcFbd01qbx+vvTSS5o5c6aKi4sr1fby5cvVvn17uVwuv5+IiAg1adJEv//97/XCCy+UeB37f+UFsg8e21+Pf38SExM1YsSICtfxxRdfKC0tTe3atVN0dLSaNGmiM844Q9OnT/ctk5aWVuJ9L+tn5cqVJWq59957y61h1qxZcrlcCgsLU8eOHbV69eoq76/1Raj30RNRF8dUBF9Z4311On5Mat26tRYuXOib/+6776pVq1a+sfOpp56qkboCqTXQsRwA8D9VDvsff/yxevfuHdCyZlbhMk8//bT+9re/Veq11SnU6580aZI++OCDKrWxZcsWnXfeebr99tuVn59f7rJ//vOf9fjjj2vKlClBWXdNqqif/fv3l9vt1kUXXaTs7OwTbj81NVXfffedUlJS5PV6ZWYyM+3Zs0fp6en68ccflZqaqvT0dL/Xsf9XXkX74K/31+Pfn927d2vJkiXltv/ll1+qZ8+eSkxM1Ntvv62cnBx98MEH6tOnj9555x2/Zd944w1lZ2ersLBQu3btkvTLPlVQUKC8vDxlZWXp+uuvl+S/r0i/vL+FhYWl1lBcXKzHH39cknThhRdq06ZNOu+886q8v9YXod5HT0RdG1NRPcoa76vT8ePjDz/8oOuuu843/7zzzlO/fv10ww03aNeuXbrhhhtqtL7yag1kLAcA+IsIVkMul6vCZS677DLl5ORUqv2qvPZEHTp0SBdddJHfwVhNrr86fPHFF3rggQd04403Ki8vr9wD44cffljPP/+8vvjiC7nd7kqtr7RtWBMC7ef48eP13XffqV+/flq9erUiIqr+UUhISNBFF12k//u//9Oll16qjIwMDR061Def/b96BGN/feyxxxQfH685c+b4pp188smaNm2aUlNTfdNcLpfOOecceTwev9e7XC5FRkYqMjJSHo9HXbt2LbGOrl276tNPP9WKFSt05ZVXlpi/fPlytWrVStu2bSsxrzr2V6epzftoXRSqMRyhc/ToUY0ePVput1vz588P6LgOAFC7Be2a/cjIyGA1FfI/MAsXLlRWVlZIawi2M844Q8uXL9fw4cMVHR1d5nLffvut7r33Xt1///2VDk5S6LZhoP2UpKlTp2rdunV+AS8Y2rZtK0mV/haW/T9wwdpf9+3bp5ycHO3fv99velRUlN9p4UuXLi0R9EszZswY/eEPf/CbdtNNN0mSnnjiiVJfM2vWLE2cOLHMNqtrfwVKU5fGgbos1OP9MUePHtUf//hHeTweLViwoNbUBQComqCF/W+//VYdO3ZUbGysYmJidO655+r999/3zX///feVnJwsl8ulefPm+aabmR599FGdcsopio6Oltfr1R133OHXdmmvfeSRR+TxeNSwYUNlZWVp4sSJatWqlTZv3qzi4mLdd999Sk5OVkxMjE4//fQSp1QvXrxY3bp1k9vtVmxsrNq2batp06ZpwoQJmjhxorZu3SqXy6UOHTqUW/usWbPUqVMnRUdHKyEhQQMGDNCmTZt8yyxYsECxsbHyeDx68cUX1bdvX8XFxSkpKUlLly71q+m9995T586d5fV65Xa71aVLF73++utVf3NOwOOPPy4zU//+/Stc9t1339XZZ58tj8ejuLg4denSRbm5uaVuwzlz5ig2NlZhYWHq2rWrmjdvrsjISMXGxuqss87Sueeeq9atW8vtdis+Pl533nlntfc1ISFB559/vubMmeM7A+C1115TXFycZsyYUel2169fL0k6//zzfdPY/6tn/z+R/bU83bt3V15eni688EL95z//qVJbZbnwwgvVqVMnvf3229q8ebPfvP/85z/Kz8/XJZdcUubrS9tf66px48YpKipKiYmJvmk333yzYmNj5XK5tHfvXkmB7z+l7aOdOnXy3f+ga9euvkt67rzzTt8+9uyzz0pSuZ+Z8j5rZY2BUvWO5+Wtt7y+BLo9SxsHgtX2MWWNQRWt50TUVL2BjIXHlqtovK+o7vL2x8r+/Tp69KhGjhwpr9frN8YHq66KPgvl7c8nqrx1jR492nf9f0pKij7//HNJ0qhRo+TxeOT1evXSSy9Vqa8AUOvYcdLT062UyeW66KKLrH379vbf//7XCgsL7auvvrLf/va35na77ZtvvvEt98MPP5gkmzt3rm/a3XffbS6Xy/7yl7/YgQMHLD8/3+bPn2+S7PPPP6/wtZJs/PjxNnfuXBs0aJBt3LjRJk2aZNHR0bZs2TI7cOCA/X/s3XlcVOX+B/DPAAPDDgKCiqDgrrhvuZSmtnn1pghier16M7cSzSXUzKw05WphbjfNpayuCuZ1zZtL5VJoairmgltCRgqCLALCAN/fH/2Y68g2AwMHhs/79eIPzpxznu/znOc8M9+Zc54zd+5csbCwkFOnTomISEREhACQxYsXS3JysqSkpMjatWtl5MiRIiISGBgo/v7+enUsrvz58+eLtbW1fP7555KamioxMTHSsWNHcXd3lzt37hSJ8/Dhw5KWliaJiYnSu3dvsbe3l9zcXN16UVFRsmDBAklJSZHk5GTp3r27uLm56V6/du2aAJB//etfRh2fx3Xr1k3atWtX7Gt+fn7SqlWrIssfL/vBgwfi5OQk4eHhkp2dLXfu3JGhQ4dKUlKSiBTfhm+//bYAkJMnT0pmZqbcu3dPnnvuOQEg+/btk6SkJMnMzJTQ0FABIOfOnau0ehaaM2eOXl/bu3evODo6yrvvvlvm/v39/cXZ2Vn3f1ZWluzfv198fX3lmWeekQcPHuitz/5v+v5fUn8t7viUJisrSzp37iwABIC0atVKwsPDJTk5udTt/vjjDwEgf/3rX0tdz9/fX3799Vf56KOPBIBMmzZN7/UhQ4bIpk2bJCMjQwBIv379it3P4/3VUABk27ZtRm1jjPK8b4wcOVI8PT31li1dulQA6MYREcP7z+N9NC8vTxo1aiQ+Pj6Sl5enV87rr78uERERuv/LOmeKO9dOnz5d6hhYWeN5WWOvoXUpqz2LGwdMte+yxqCyyjFUVcVrzFhoyHhfnv54+fLlcr1/5eXlyciRI0WtVktsbKxJ2vPxuEo7F8rqEmZccAAAIABJREFUz4/GaoiyzrvAwECxtLSU33//XW+7l156SXbv3l3huhqiPOMlEVE5RZos2X88qYqJiREAMnPmTN2yxz+MZWVliZ2dnQwYMEBv2y1bthiV7GRnZ+uWZWdni52dnYSEhOiWZWVliY2NjUyePFlyc3PFxcVF+vbtq1dmXl6eLF++XEQMS3aysrLEwcFBrxwRkZ9++kkA6L3ZFhdn4Rv89evXi7Rnoffff18ASGJioohUfrL/4MEDUalUMmjQoCKvPV72L7/8IgBk7969xZZRWrKfkZGhW/bZZ58JALlw4YJuWWEbbt26tVz1K2RIsr9x40YBIJs3bzZ6//7+/rrk8NG/gIAA+eyzzyQnJ0dvffZ/0/b/0vqriHEfEEVEcnNz5aOPPpIWLVrojmXdunXl+++/L3EbY5P91NRUsbe3F1dXV8nKyhIRkRs3boi3t7fk5OSUmeyXt7+aQ7JfVv8p7hwpTNIiIyN1yzIzM8XHx0fS0tJEpOxzpqQYyhoDH2eq8by0cstbl+La8/FxwFT7LmsMMqQcQ1RVvIaOhYaO9+WN21j+/v7i6OgoI0aMkI4dOwoAad26dZEvqQuZMq5HzwVDziNjx/KSyhIROXTokACQhQsX6tZJS0uTpk2b6r4UrOxjwGSfiKpQpMku439cQEAAnJ2ddZc0F+f69evIyspCv379TFZubGwssrKy0KZNG90yW1tbeHl54cqVK4iJiUFqaiqeffZZve0sLS0xdepUg8u5ePEiHjx4gM6dO+st79KlC6ytrXHy5MlSt7e2tgaAEmfmBv43D0JVPXIrMTERImLQPcl+fn6oW7cuRo0ahQULFuDWrVvlKrOwHfLy8nTLCutdWtuYSmFd7969W67tH52NX6vV4vbt23j99dcRGhqKtm3b6i5HLg77f8X6vzH91RBqtRqhoaG4fPkyTpw4gRdffBGJiYkICgrC/fv3TVKGs7MzXnrpJdy/fx9bt24FAERERGDy5Mm6NilNRfuruTCk/wB/Xrbr7OysN8/BF198gRdffBFOTk4Ayj5nSmLsGGiq8by0cstbF0Pa01T7LmsMKm85SsVr6Fho6HhvqvobIisrC0899RTOnDmDIUOG4OLFixg3blylx/XouWCqzxKGlAX8eTtVs2bNsHHjRt3tUFu3bkVISAgsLS0BVO0xICKqbJWW7AN/DrKlfXi4ffs2AMDDw8NkZWZmZgIA5s2bp/fM67i4OGRlZenuA3NxcalQOYWTrzk4OBR5zcXFBRkZGUbvc9++fejTpw88PDxgY2NTJfetP+rhw4cAUObEdsCfb3zffvstevXqhUWLFsHPzw8hISHIzs6u7DBNytbWFsD/6l4RVlZWaNCgAcaOHYtly5YhNjYWixcvLnF99n99xvZ/Y/qrsbp164b//Oc/mDRpEpKSkvDdd9+ZbN+FE/V9/PHHSE1NRVRUFCZOnGjQtqbsr7WBg4MDxo8fjx9//BE//fQTgD8nSAwNDdWtU9Y5U5KyxsDKGs9LK7e8dTGEqfZd1hhkqnKqKl5Dx0JDx/vKPIaPc3BwwIQJEwAAmzZtgp+fH7Zu3YqIiAiTxlXauWDqzxJlnXcqlQoTJ07EzZs3cfjwYQDA5s2b9R4/WJXHgIioslVasp+Xl4eUlBT4+PiUuE7h7Nk5OTkmK7fwjTQiIkL3i2vhX3R0NOrXrw8Apf7iaojCN/7ikprU1FR4e3sbtb/4+HgMGTIEXl5eOHnyJNLS0hAeHl6hGI1VmEgY+stT69atsWfPHiQkJCAsLAzbtm3DsmXLKjNEk8vNzQXwv7qbSkBAAADg0qVLJa7D/v8/5en/xvbXRx09elTvA21gYKDe1SWF/va3vwGAST/gtW/fHt27d8dPP/2ECRMmICgoCK6urgZtW1n91ZyFhoZCrVYjIiICR48eRcOGDeHv7697vaxzpjQljYGVPZ6XVG5F6lIWU+27rDHIVOVUVbyGjoWGjveVeQxL4+zsjKioKF2CfPToUZPEZci5UJHPEo+O5Yaed2PGjIFGo8H69esRGxsLJycn+Pr6VriuRETVUaUl+9999x0KCgrQsWPHEtdp06YNLCwscOTIEZOVWzij+7lz54p9vVGjRqhTpw4OHDhQoXLatGkDBwcHnD59Wm/5yZMnkZubW+xztktz4cIFaLVaTJ48GX5+ftBoNFX+6Ju6detCpVIZ9KzqhIQEXSLr4eGBxYsXo2PHjqUmt9VRYV09PT1Nut8zZ84AAJo3b17iOuz//1Oe/m9Mf33cmTNnYG9vr/s/Jyen2L5bOLty27ZtjS6jNIW/7m/fvh2vv/66wdtVVn9VgpWVVZXcquPt7Y3g4GBs374db731FqZNm6b3elnnTElKGwMrczwvrdzy1sUQptp3WWOQqcqpqngNHQsNHe8r8xiWpWPHjoiIiEBeXh6Cg4ORkJBQ4bjKOhcq+lni0bHc0PPO1dUVw4cPx86dO7Fs2TK88soreq8reQyIiEzNZMl+bm4u0tLSkJeXh59//hmhoaHw9fXFmDFjStzGw8MDgYGB2L59OzZs2ID09HTExMRg3bp15Y5Do9Fg7Nix2LJlC9asWYP09HTk5+fj9u3b+OOPP2BjY4O5c+fi6NGjCA0Nxe+//46CggJkZGTo3lzq1KmDhIQE3Lp1CxkZGcV+INVoNJgxYwZ27NiBL774Aunp6bhw4QImTZqEevXq6S6NM1ThFRCHDh3Cw4cPce3atTLvezY1Ozs7+Pn56S43LE1CQgImTpyIK1euIDc3F2fPnkVcXBy6d+8OwLA2rA4K61r4S/z+/fuNfnRRdnY2CgoKICJISEjApk2bMG/ePLi7u5eayLH//095+r8x/bWQVqvF3bt38f333+sl+wAwZMgQREZGIjU1FWlpadi1axdmz56Nv/71ryZP9oODg+Hu7o4hQ4bAz8/P4O0e7681WZMmTZCSkoKdO3dCq9UiKSkJcXFxlVLWjBkzkJeXh/v37+Ppp5/We62sc6YkpY2BlTmel1ZueetSnMfHAUtLS5Psu6wxyFR1MNV+DInXkLHQ0PG+InGX5/3rcZMmTcKIESNw9+5dBAUF6cb/8sZV1rlQ1meJkhQ3lhtz3k2aNAk5OTnYu3cvBg0apPeaKc8jIiLFPT5lX3lmCd20aZP07dtX6tatK1ZWVuLm5iYjRoyQuLg43TorV64ULy8vASB2dnYyePBgERHJyMiQcePGiZubmzg4OEivXr1k/vz5AkC8vb3l/PnzxW4bHh4utra2AkAaNmwon3/+ua6snJwcCQsLEx8fH7GyshIPDw8JDAyUixcv6tZZtWqVBAQEiEajEY1GIx06dJDVq1eLiMjPP/8svr6+YmtrK7169ZJ58+YVG3tBQYEsXbpUmjZtKmq1WlxdXWXIkCF6j69ZvXq12NnZCQBp2rSp3LhxQ9atWydOTk4CQHx9fXWPJwwLC5M6deqIi4uLBAUFyapVqwSA+Pv7y7Rp08TT01MAiL29vQwdOtSoYxQdHS09e/aUevXq6WYa9/Lykh49esiRI0d064WGhopardbNFC4i8sEHHxQp+9atW9KjRw9xdXUVS0tLqV+/vrz55pu62Wwfb8M5c+bo2qFRo0Zy7NgxWbJkiTg7OwsA8fT0lC+//FK2bt2qK8vV1VW2bNlSKfUsNHDgQGnQoIEUFBSIiMjXX38tjo6OejP1Pm7Hjh0lzsRvY2MjTZs2lcmTJ0t8fLxuG/b/yun/xfXX0o7Po387duzQbXPgwAEZPny4+Pv7i42NjVhbW0vz5s1lwYIF8vDhwyJ9ID09XZ588kmpU6eOABALCwtp0qSJLFq0qMS+4u7uLq+99prutTfeeEN+/PFH3f+PtrOFhYW0atVKjh07pre/x/uroVANZ+NPTk6Wvn37ikajkcaNG8uUKVNk1qxZAkCaNGki8fHxBvefks6vR/Xt21fWr19fbCylnTMlnWtljYGVNZ6XVW5pdTHmfHx8HLhz547J9i1S+hhkyBhmiKqK15CxUMSw8b68/VGkfO9f3t7eMnfu3CJxNm/eXPdEkg0bNlQortLOhWPHjpXYn8szlpdW1qPvySIiHTp0kDlz5hjdd0qrqyE4Gz8RVaFIlcj/T0f6/yIjIzF8+HA8tphqievXr6Nly5bYtGkTRo0apXQ4lSo5ORne3t5YuHAhZsyYoXQ4VA7sr4ZRqVTYtm0bgoODKyU2vm8QUU0zcOBArFq1Co0bN67ScjleElEViqrU2fip5mnSpAneffddvPvuu3jw4IHS4VSqBQsWoH379nozc1PNwv5KRESGePSWtJiYGGg0mipP9ImIqhqT/RrqypUreo+EKekvJCTE6H3PmTMHQUFBCAkJKdfkZ6ZUWfX88MMPce7cOXz99de65/BSzVSd+mtlYX81b5U5npsTthNVRFhYGK5du4arV69i7NixeO+995QOiYio0lkpHQCVT4sWLSr1ErBFixbhwIEDWLx4MZYsWVJp5ZSlMuq5a9cu5OTk4Pvvv4elpaVJ903KqC79tTKwv5q/yh7PzQXbiSrCzs4OLVq0QIMGDbB69Wq0atVK6ZCIiCod79knIjJzvGefiKh64HhJRFWI9+wTERERERERmRsm+0RERERERERmhsk+ERERERERkZlhsk9ERERERERkZpjsExEREREREZkZJvtEREREREREZobJPhEREREREZGZYbJPREREREREZGaY7BMRERERERGZGSb7RERERERERGaGyT4RERERERGRmWGyT0RERERERGRmmOwTERERERERmRmrkl4ICgqqyjiIyAhpaWlQqVRwcnJSOhQiHb5vmJ/09HSICJydnZUOhcgs3L59W+kQiKgWKZLsN2zYEMOGDVMiFiIy0PXr13Hr1i00bNgQrVq1goODg9IhUTU2bNgwNGzYsNL2z/cN85ORkYHLly/jt99+Q6NGjdCpUyelQyIyC97e3hwviajKqERElA6CiIwjIti+fTveeust3LhxAyNGjMCCBQvg5+endGhEVIPFx8dj0aJF2LhxI5o0aYI5c+Zg5MiRsLS0VDo0IiIiMk4U79knqoFUKhWCgoJw6dIl/Pvf/0Z0dDRatmyJCRMmICEhQenwiKiGuX37NqZOnYpmzZrhm2++werVq3HhwgWMHj2aiT4REVENxWSfqAazsLDQJf0rV67Evn370LRpU0ydOhV3795VOjwiqubu3buH2bNno1mzZvjPf/6D8PBwxMbGYvz48bCyKnFaHyIiIqoBeBk/kRnJzc3Fp59+irfffhsPHjzAq6++itmzZ8PFxUXp0IioGklJScGKFSsQEREBGxsbzJgxA1OnToVGo1E6NCIiIjKNKCb7RGYoMzMTq1atQnh4OFQqFaZMmYLp06dz9n6iWu7BgwdYvXo1lixZAktLS8yaNQtTpkyBnZ2d0qERERGRaTHZJzJnGRkZWLNmDd5//31YW1tj5syZ/PWOqBZ69AvAvLw8TJ48GXPmzOEj9YiIiMwXk32i2uDevXtYtmwZVqxYAXd3d8yYMQMTJ06EjY2N0qERUSXirT1ERES1FpN9otokMTERH374IZYvXw4vLy/MnTsXL7/8MmfbJjIzhUn+O++8g9TUVIwbNw5z586Fp6en0qERERFR1eCj94hqk7p162LJkiW4evUqnn32Wbz66qsICAhAVFQU+L0fUc2n1WqxefNmtGzZElOmTMFf/vIXXL9+HR999BETfSIiolqGyT5RLeTj44O1a9fiwoUL6Ny5M0aMGIF27dohKipK6dCIqBwKCgoQFRWF1q1bY9y4cejZsyeuXLmCtWvXol69ekqHR0RERApgsk9Ui7Vo0QKbN2/G+fPn0aJFCwwfPhxPPPEEDh8+rHRoRGQAEcGePXvQsWNHhISEoH379rh06RI2b96Mxo0bKx0eERERKYjJPhGhdevWiIyMRHR0NNzd3dG/f3/06tULR48eVTo0IirBoUOH0LlzZ7z44oto1qwZLl26hMjISDRp0kTp0IiIiKgaYLJPRDrdunXDnj17cPz4cajVajz11FMYMGAAzpw5o3RoRPT/Dh06hK5du2LAgAGoU6cOTp8+jcjISDRv3lzp0IiIiKgaYbJPREX07NkT3333HQ4ePIjU1FR06dIFgwYNQkxMjNKhEdVaP/zwA55++mkMGDAAzs7OOHXqFA4ePIgOHTooHRoRERFVQ0z2iahE/fv3x6lTp3DgwAHcvn0bHTp0QHBwMK5fv650aES1xokTJzBo0CD06tULubm5+P7773Hw4EF07txZ6dCIiIioGmOyT0Rl6t+/P86cOYOtW7fi/PnzaNWqFUaPHo1ff/1V6dCIzNaFCxcQHByMJ554AsnJyTh06BCOHz+Op556SunQiIiIqAZgsk9EBrGwsEBQUBAuX76ML7/8Ej/88ANatGiBCRMm4M6dO0qHR2Q2Ll26hODgYLRr1w5xcXHYvXs3fvzxR/Tr10/p0IiIiKgGYbJPREZ5NOlfuXIl9u7diyZNmmD27Nm4f/++0uER1Vi//vorJkyYgLZt2+Ly5cvYtm2b7hJ+IiIiImMx2SeicrG2tsb48eNx7do1LFq0CJ9++il8fX0xe/ZspKWlKR0eUY0RFxeHCRMmoFmzZjh27Bg2btyI8+fPIygoCCqVSunwiIiIqIZSiYgoHQQR1XwPHjzA6tWrsWTJElhaWmLWrFkIDQ2Fra2t0qERVUu3b9/G0qVLsXbtWtSrVw9z5szByy+/DEtLS6VDIyIiopovisk+EZlUSkoKVqxYgYiICNjY2GDGjBmYOnUqNBqN0qERVQtJSUn44IMPsGLFCri7u2PevHn4xz/+ASsrK6VDIyIiIvPBZJ+IKse9e/ewbNkyfPTRR6hbty7efPNNJjRUqyUnJ2PlypX48MMPYWtri+nTp/OLMCIiIqosTPaJqHL99ttvWLZsGdauXYv69etj9uzZvFSZapWMjAysWbMGixcvhlqtxsyZM3mLCxEREVU2JvtEVDXi4uLw/vvvY8OGDWjRogXefvttDBs2jBOQkdnKzMzEqlWrEB4eDpVKhSlTpmD69OlwcnJSOjQiIiIyf1GcjZ+IqoSvry/Wrl2LCxcuoFWrVhg+fDi6d++OPXv2KB0akUllZWXho48+QpMmTbBw4UKMHz8eN27cwIIFC5joExERUZVhsk9EVaply5aIjIzE+fPn4evri8GDB6Nnz5747rvvlA6NqEJyc3Oxbt06NG3aFG+++SaCg4Nx48YNLFmyBC4uLkqHR0RERLUMk30iUkRAQAAiIyMRHR0NW1tbPP300xgwYABOnTqldGhERtFqtdi8eTNatmyJKVOm4C9/+QuuXbumm5ySiIiISAlM9olIUd27d8ehQ4dw7Ngx5ObmomvXrhgwYADOnj2rdGhEpSooKEBUVBRat26NcePGoX///rh58ybWrl2LevXqKR0eERER1XJM9omoWujVqxeOHDmCgwcP4v79++jcuTOCg4Nx9epVpUMj0iMiiIqKQqtWrRASEoL27dvj8uXLWLt2LRo0aKB0eEREREQAmOwTUTXTv39/nDp1Cjt37kRsbCxatmypu/eZSGmHDh1Cp06dEBISgrZt2+Ly5cuIjIyEv7+/0qERERER6WGyT0TVjkqlwqBBg3D27Fls3boVZ8+eRcuWLTFhwgQkJCQoHR7VQocOHULXrl3xzDPPoEGDBjhz5gwiIyPRrFkzpUMjIiIiKhaTfSKqtiwsLBAUFIRLly5h/fr1OHjwIPz8/DBhwgTcvXtX6fCoFjh+/Dj69u2LAQMGwNnZGadOncKePXvQvn17pUMjIiIiKhWTfSKq9tRqNUaPHo0rV65gxYoV2L17N5o0aYLZs2cjNTVV6fDIDEVHR6N///7o3bs3tFqtbj6JTp06KR0aERERkUGY7BNRjWFtbY3x48fj+vXrmDdvHtatWwd/f38sWLAA6enpSodHZiAmJgbBwcHo0aMHsrOzcfjwYRw/fhxPPvmk0qERERERGYXJPhHVOPb29ggLC0NcXBzeeOMNLF++HP7+/ggPD0d2drbS4VENdPHiRQQHB6N9+/aIj4/H7t278cMPP+Dpp59WOjQiIiKicmGyT0Q1lqOjI8LCwnDjxg28/PLLeOedd9CsWTN89NFHyMnJUTo8qgGuXLmC0aNHo127drh8+TK2bduG6OhoDBo0SOnQiIiIiCqEyT4R1Xhubm5YsmQJ4uLiMHLkSMyePRvNmzfHunXrkJ+fb9A+Hj58WMlRUmXLzc01eN24uDhMmDABAQEBOH36NDZu3Ijz588jKCgIKpWqEqMkIiIiqhpM9onIbHh4eGDJkiWIjY3Fs88+i1dffRUBAQHYvHkzCgoKStxORNC/f398//33VRcsmVRGRgb69OmDixcvlrreb7/9hqlTp6J58+Y4cOAAVq9ejQsXLmD06NGwsOBbIhEREZkPfrIhIrPj4+ODtWvX4urVq+jduzf+8Y9/oF27doiKioKIFFl/x44d+OGHH/DCCy/gp59+UiBiqojs7Gy88MILiI6OxltvvVXsOklJSZg9ezaaNWuGnTt3YsWKFbh27RrGjx8PS0vLKo6YiIiIqPKppLhPvkREZuTixYt45513sH37dnTr1g1z587V3ZOdn5+Pli1b4saNG1CpVLC3t8fx48cREBCgcNRkiNzcXAwaNAjffvst8vLyoFKp8PPPP6N9+/YAgOTkZCxduhQrV66Eg4MDpk+fjqlTp0Kj0SgcOREREVGliuIv+0Rk9lq3bo3IyEicOHEC7u7uGDx4MHr16oUjR47g3//+N65fv46CggLk5+cjKysLffr0wZUrV5QOm8qQn5+PUaNG4fDhw8jLywMAWFlZ4c0330RGRgbCw8Ph7++PDRs2YP78+bh16xbCwsKY6BMREVGtwF/2iajWOXr0KObNm4djx47BxcUF6enpevf0q9VquLm5ITo6Go0aNVIuUCqRiOCVV17Bp59+WuwkjA4ODrCxscGsWbPw2muvwd7eXoEoiYiIiBQTxWSfiGqtqVOnYuXKlcXex69Wq1G/fn2cOHECXl5eCkRHpZk+fTo++uijYideVKvVaNy4MU6dOgUnJycFoiMiIiJSHC/jJ6LaKScnB5GRkSW+rtVqkZCQgL59+yIlJaUKI6OyzJkzB8uXLy/xCQtarRZXr17F2bNnqzgyIiIiouqDyT4R1Uoff/wxkpKSiv1Vv5BWq8WNGzcwYMAAZGRkVGF0VJJFixZhyZIlpR434M9792fPnl1FURERERFVP7yMn4hqnczMTPj6+iI5Odmg9a2srNCzZ0/897//5eRuClqxYgWmTp1q1DYHDx5E//79KykiIiIiomqLl/ETUe2zf/9+qFQq3f8qlQrW1tawsrIqdv28vDz88MMPGDZsGLRabVWFSY/YuHEjpk2bVuLrxR1DKysrfPbZZ1URHhEREVG1w1/2iR5R2j3cZH6ys7Nx9+5d3LlzB3fu3MEff/yBhIQE/PHHH7rL9lUqFaysrJCXlwcRwRNPPIGpU6fqfVlAlevHH3/EihUrICKwsrJCQUGB7n59KysruLm5oUGDBqhfvz48PT3h5eUFT09PuLu7w9LSUuHoqaoEBwcrHQIREVF1wtn4iR7FBI6IqGbixxkiIiI9UcVfs0pUi23bto2/EFGpsrKyoFaroVarlQ7F7KWnp/PxeVSqyMhIDB8+XOkwiIiIqh0m+0RERrKzs1M6hFqDiT4RERFR+XCCPiIiIiIiIiIzw2SfiIiIiIiIyMww2SciIiIiIiIyM0z2iYiIiIiIiMwMk30iIiIiIiIiM8Nkn4iIiIiIiMjMMNknIiIiIiIiMjNM9omIiIiIiIjMDJN9IiIiIiIiIjPDZJ+IiIiIiIjIzDDZJyIiIiIiIjIzTPaJiIiIiIiIzAyTfSIiIiIiIiIzw2SfSAGLFy+Gs7MzVCoVzp07p3Q4Bhs7diw0Gg1UKhUePnxoNnF06dIFlpaWaN++fbn38fXXX8PZ2Rl79uwpcZ1x48bB0dGx2h13U9S/JIbWuaT1DGnXqhYbG4spU6agdevWcHR0hJWVFZydndGsWTMMHDgQ0dHRSodIRERExGSfSAlz5szB2rVrlQ7DaJs2bcLMmTOVDsPkcZw6dQp9+/at0D5EpMx11q9fj08++aRC5VQGU9S/JIbWuaT1DGnXqrRhwwYEBAQgJiYGH374IX777TdkZmbi7NmzeO+995CamooLFy4oHSYRERERrJQOgKgmy87ORr9+/fDjjz8qHQqZgEqlKve2AwcORFpamgmjqXoVqX9lqU7teuLECUyYMAFPPfUUvvnmG1hZ/e8t1M/PD35+fnBxccG1a9cUjLJ0So5ZHC+JiIiqFpN9ogrYsGEDEhMTlQ5DEdUlMTRlHGq12mT7Kkl1abfiVFb9Da1zVbSNiGD79u24f/8+xo8fb9S2CxcuRH5+PhYvXqyX6D/q2WefxbPPPmuKUCuFkmNWbR4viYiIlMDL+InKadq0aZgxYwZu3LgBlUqFJk2aAPgzmfjwww/RsmVL2NjYwNXVFS+++CKuXLlS6v7u3r2LRo0awcrKCs8995xueX5+PubPnw8fHx/Y2tqibdu22LZtGwBgzZo1sLe3h52dHXbt2oXnn38eTk5O8Pb2xpYtW8pdt88//xydO3eGRqOBvb09GjVqhPfee0/3uoWFBfbt24fnn38ezs7OqFevHjZu3Ki3j2PHjqFVq1ZwdnaGRqNBQEAAvvnmGwDAP//5T9jZ2cHR0RGJiYmYMWMGGjRogNjYWKPiLCuOcePGQaVSQaVSwd/fH2fPngXw5z3/dnZ2cHZ2xu7du3XrX79+HS1atIC9vT1sbW3Ru3dvHD9+XPd6SXFv2LABPj4+UKlUWLVqlW59EcHSpUvRvHlz2NjYwNnZGbNmzTKqjo8qrS8sX74c9vb2sLD8II+SAAAgAElEQVSwQKdOneDp6Qm1Wg17e3t07NgRvXv3RsOGDaHRaODi4oI33nijyP7Lqn9ZMRhTZ0PWO378eJF2NabP5+fn4/3330fz5s1ha2sLd3d3NG7cGO+//z6Cg4N16/33v/+Fk5MTFi1aVGLb5+bm4vDhw3Bzc0PXrl1LXK+4epY1Hhh7Hpd2fpZ23pU0ZplqjDF12URERFRBQkQ6AGTbtm0Grx8YGCj+/v56y+bPny/W1tby+eefS2pqqsTExEjHjh3F3d1d7ty5o1tvy5YtAkDOnj0rIiK5ubkSGBgou3bt0tvfzJkzxcbGRrZv3y7379+XuXPnioWFhZw6dUpERN58800BIIcPH5a0tDRJTEyU3r17i729veTm5hrdBhEREQJAFi9eLMnJyZKSkiJr166VkSNHFikvNTVVUlJS5IUXXhAbGxvJzMzU7ScqKkoWLFggKSkpkpycLN27dxc3Nzfd64X7mTp1qqxcuVKGDh0qly9fNjhOQ+MIDAwUS0tL+f333/W2f+mll2T37t26//v16yd+fn7y66+/ilarlV9++UW6desmGo1Grl69Wmbcv/32mwCQlStX6q2rUqnkgw8+kPv370tWVpasXr1a77gbo6y+8PbbbwsAOXnypGRmZsq9e/fkueeeEwCyb98+SUpKkszMTAkNDRUAcu7cOaPrb0h/NKTOhq5XUrsa0ucXLVoklpaWsmvXLsnKypIzZ86Ip6en9OnTR69d9+7dK46OjvLuu++W2PZXr14VANK9e3ejjpmh44GhdSrr/CzrvCtuzDLVGFMZZRti27Ztwo8zRERERUTy3ZHoERVN9rOyssTBwUFCQkL01vvpp58EgF4y8Wiyr9VqZcSIEbJ//3697bKzs8XOzk5vf1lZWWJjYyOTJ08Wkf99EM/OztatU5g0Xb9+3eC6iPz5hYOLi4v07dtXb3leXp4sX768xPI2b94sAOSXX34pcd/vv/++AJDExMQS92MMQ+M4dOiQAJCFCxfqlqWlpUnTpk0lLy9Pt6xfv37Srl07vTJiYmIEgMycObPUckWKJqVZWVliZ2cnAwYM0Fvv8S95DGVIXyhM9jMyMnTrfPbZZwJALly4oFtW2B+3bt1qVP3LisHQOhvTNqUl+2X1+S5dukjXrl31yhg/frxYWFhITk6OGOP06dMCQPr372/wNsaMB4bUyZDz83GPn3ePj1mVOcaYomxDMNknIiIqViQv4ycyoYsXL+LBgwfo3Lmz3vIuXbrA2toaJ0+eLLJNfn4+XnrpJdStW1fv8n3gz0d8ZWVloU2bNrpltra28PLyKvW2AGtrawCAVqs1Kv6YmBikpqYWuefY0tISU6dOLXG7wnu9SyuvcJ38/HyjYjJGcXE8/fTTaNasGTZu3Kib2X3r1q0ICQmBpaVlqfsLCAiAs7MzYmJijI7l+vXryMrKQr9+/YzetjgV7Qt5eXm6ZYYcL6Bo/cuKwdA6m7ptgOL7/MOHD4vM5p+fnw+1Wl3msX+cg4MDACArK8vgbcozHjzq8TqV5/ws67yrzDGmssomIiIiwzDZJzKh1NRUAP9LDB7l4uKCjIyMIstfe+01XLt2DR9//DEuXbqk91pmZiYAYN68ebp7z1UqFeLi4oxKOgyVnp6ui7Wi9u3bhz59+sDDwwM2NjbF3iNeFVQqFSZOnIibN2/i8OHDAIDNmzfj5ZdfNmh7tVpt9JcmAHD79m0AgIeHh9HbFqeq+0KhR+tfVgyG1tnUbVOSF154AWfOnMGuXbuQnZ2N06dPY+fOnfjLX/5idLLfqFEjaDQaXL161eBtyjMelMaQ89PY886U/UrJsomIiKgoJvtEJlT4Iby4D/Gpqanw9vYusjw4OBgHDx6Ei4sLRo8erfcLbGEyFBERARHR+4uOjjZ5/PXr1wcA3Lt3r0L7iY+Px5AhQ+Dl5YWTJ08iLS0N4eHhpgixXMaMGQONRoP169cjNjYWTk5O8PX1LXO7vLw8pKSkwMfHx+gyNRoNACAnJ8fobYtT1X0BKFr/smIwtM6mbpuSLFiwAE8//TTGjBkDJycnDB06FMHBwfjkk0+M3peNjQ2effZZ3Lt3Dz/88EOJ66WkpGDcuHEAyjcelKas87M8552p+pWSZRMREVHxmOwTmVCbNm3g4OCA06dP6y0/efIkcnNz0alTpyLb9O3bF+7u7li3bh3OnDmDhQsX6l4rnD393LlzlR478Oevl3Xq1MGBAwcqtJ8LFy5Aq9Vi8uTJ8PPzg0ajUfSRc66urhg+fDh27tyJZcuW4ZVXXjFou++++w4FBQXo2LGj0WW2adMGFhYWOHLkiNHbFqeq+wJQtP5lxWBonU3dNiW5ePEibty4gaSkJGi1WsTHx2PNmjVwdXUt1/4WLFgAGxsbTJ8+HdnZ2cWu88svv+gey1ee8aA0ZZ2f5TnvTNWvlCybiIiIisdkn6gC6tSpg4SEBNy6dQsZGRmwtLTEjBkzsGPHDnzxxRdIT0/HhQsXMGnSJNSrVw8TJkwocV+DBw/GmDFjsGjRIpw5cwbAn7+Ajh07Flu2bMGaNWuQnp6O/Px83L59G3/88YfJ62NjY4O5c+fi6NGjCA0Nxe+//46CggJkZGQUucWgNIW/BB86dAgPHz7EtWvXyrw/ubJNmjQJOTk52Lt3LwYNGlTsOrm5uUhLS0NeXh5+/vlnhIaGwtfXF2PGjDG6PA8PDwQGBmL79u3YsGED0tPTERMTg3Xr1pUr/qroC2XVv6wYDK2zqdumJK+99hp8fHzw4MGDUtfbv39/mY/eA4D27dvjyy+/xC+//ILevXvj66+/RlpaGrRaLX799Vd88sknePnll3X3qms0mnKPB8Up6/w05LwrbswyRb9SsmwiIiIqQVVOB0hU3cHI2fh//vln8fX1FVtbW+nVq5fcuXNHCgoKZOnSpdK0aVNRq9Xi6uoqQ4YMkdjYWN12X331lbi6ugoAadSokSQmJkp6ero0bNhQAIiDg4Ns3rxZRERycnIkLCxMfHx8xMrKSjw8PCQwMFAuXrwoq1evFjs7OwEgTZs2lRs3bsi6devEyclJAIivr6/eY9MMtWrVKgkICBCNRiMajUY6dOggq1evlvDwcLG1tdUr74svvtDVxdvbWzcTflhYmNSpU0dcXFwkKChIVq1aJQDE399fXnvtNd1+GjZsKJ9//rlR8RkTx6M6dOggc+bMKXafmzZtkr59+0rdunXFyspK3NzcZMSIERIXF1dsuY/GvXLlSvHy8hIAYmdnJ4MHDxYRkYyMDBk3bpy4ubmJg4OD9OrVS+bPn6+L8fz580bVu7S+sHz5cl1faNSokRw7dkyWLFkizs7OAkA8PT3lyy+/lK1bt4qnp6cAEFdXV9myZYvB9S8rBmPqbMh6xbWrMX3+22+/FTc3NwGg+1Or1dKyZUv56quvdHX6+uuvxdHRUe+JDaWJj4+XmTNnSkBAgDg4OIilpaW4uLhIhw4d5OWXX5YffvhBt64h44Gx53FJ56dI6eddfHx8sWOWqcYYU5dtKM7GT0REVKxIlchjUxUT1WIqlQrbtm1DcHCw0qFQJRg4cCBWrVqFxo0bKx0KVYE1a9bg2rVriIiI0C3Lzc3F7NmzsWbNGty/fx+2trYKRkimEBkZieHDhxd58gIREVEtF2WldARERJVFq9XqLqmOiYmBRqNhol9L3LlzB6GhoUXuB7e2toaPjw+0Wi20Wi2TfSIiIjJbvGefyMxduXJF77FWJf2FhISYXZxhYWG4du0arl69irFjx+K9996rxBoYr6Ycm5rI1tYWarUaGzZswN27d6HVapGQkID169dj/vz5CAkJgZOTk9JhEhEREVUa/rJPZOZatGhRIy5vrYw47ezs0KJFCzRo0ACrV69Gq1atTLr/iqopx6YmcnZ2xoEDB/Duu++iWbNmyMzMhIODA1q3bo0lS5Zg/PjxSodIREREVKl4zz7RI3jPPhFRzcJ79omIiIoVxcv4iYiIiIiIiMwMk30iIiIiIiIiM8Nkn4iIiIiIiMjMMNknIiIiIiIiMjNM9omIiIiIiIjMDJN9IiIiIiIiIjPDZJ+IiIiIiIjIzDDZJyIiIiIiIjIzTPaJiIiIiIiIzAyTfSIiIiIiIiIzw2SfiIiIiIiIyMww2SciIiIiIiIyM0z2iYiIiIiIiMyMldIBEFU30dHRSodAREQG4phNRERUPJWIiNJBEFUXKpVK6RCIiKgc+HGGiIhITxR/2Sd6BD8skrkKDg4GAERGRiocCRERERFVBd6zT0RERERERGRmmOwTERERERERmRkm+0RERERERERmhsk+ERERERERkZlhsk9ERERERERkZpjsExEREREREZkZJvtEREREREREZobJPhEREREREZGZYbJPREREREREZGaY7BMRERERERGZGSb7RERERERERGaGyT4RERERERGRmWGyT0RERERERGRmmOwTERERERERmRkm+0RERERERERmhsk+ERERERERkZlhsk9ERERERERkZpjsExEREREREZkZJvtEREREREREZobJPhEREREREZGZYbJPREREREREZGaY7BMRERERERGZGSb7RERERERERGaGyT4RERERERGRmWGyT0RERERERGRmmOwTERERERERmRkm+0RERERERERmhsk+ERERERERkZlhsk9ERERERERkZpjsExEREREREZkZJvtEREREREREZobJPhEREREREZGZYbJPREREREREZGaslA6AiIhM68iRIzhx4oTesitXrgAAwsPD9ZZ3794dTz31VJXFRkRERERVQyUionQQRERkOgcPHsQzzzwDtVoNC4viL+AqKCiAVqvFgQMHMGDAgCqOkIiIiIgqWRSTfSIiM5Ofnw9PT08kJyeXup6rqysSExNhZcWLvIiIiIjMTBTv2SciMjOWlpYYOXIkrK2tS1zH2toaf/vb35joExEREZkpJvtERGZoxIgRyM3NLfH13NxcjBgxogojIiIiIqKqxMv4iYjMlK+vL+Lj44t9zdvbG/Hx8VCpVFUcFRERERFVAV7GT0RkrkaNGgW1Wl1kubW1Nf7+978z0SciIiIyY0z2iYjM1KhRo6DVaossz83NRUhIiAIREREREVFVYbJPRGSmWrZsiZYtWxZZ3qJFC7Rp00aBiIiIiIioqjDZJyIyY6NHj9a7lF+tVuPvf/+7ghERERERUVXgBH1ERGYsPj4ejRo1QuFQr1KpcPPmTTRq1EjZwIiIiIioMnGCPiIic+bj44POnTvDwsICKpUKXbp0YaJPREREVAsw2SciMnOjR4+GhYUFLC0t8be//U3pcIiIiIioCvAyfiIiM5eUlIR69eoBAH7//Xd4enoqHBERERERVbIoK6UjICIqj6CgIGzfvl3pMGocLy8vpUOoMYYNG4aoqKhK2Tf7LxGVhuMPERmruN/wmewTUY3VvXt3vP7660qHUSMcOXIEKpUKTz75pNKh1AgRERGVXgb7LxEVh+MPERkjOjoay5cvL/Y1JvtEVGN5e3sjODhY6TBqhOeeew4A4OTkpHAkNUNl/aL2KPZfIioOxx8iMhaTfSKiWoxJPhEREVHtwtn4iYiIiIiIiMwMk30iIiIiIiIiM8Nkn4iIiIiIiMjMMNknIiIiIiIiMjNM9omIiIiIiIjMDJN9IiIiIiIiIjPDZJ+IiIiIiIjIzDDZJyIiIiIiIjIzTPaJiIiIiIiIzAyTfSIiIiIiIiIzw2SfiIiIiIiIyMww2SciIiIiIiIyM0z2iYiIiIiIiMwMk30iqrXGjRsHR0dHqFQqnDt3TulwKqSgoAARERHo0aNHiescP34cPXv2hJ2dHerVq4ewsDDk5OQYXdZXX30FPz8/qFQqvT9ra2vUrVsXffr0wdKlS3H//v2KVIkesWzZMtStWxcqlQoff/yxbvnXX38NZ2dn7NmzR8HoqocuXbrA0tIS7du3VzoUszguJ06cQMuWLWFhYQGVSgVPT08sXLhQ6bD0PD4WeXl5YdSoUUqHRRVU0nhXkW2r+zlZ3eMzBMeM6onJPhHVWuvXr8cnn3yidBgVdu3aNTz55JOYPn06srKyil3n4sWLeOaZZ9CvXz8kJSVhx44d2LhxIyZNmmR0eYGBgbh58yb8/f3h7OwMEUFBQQESExMRGRmJxo0bIywsDK1bt8bp06crWj0CMHPmTPz4449FlouIAtFUT6dOnULfvn2VDgOAeRyX7t274/Lly3jmmWcAALGxsZg3b57CUel7fCy6c+cOvvjiC6XDogoqabyryLbV/Zys7vEZgmNG9cRkn4ioBjt//jxmz56NSZMmlfqL5nvvvQcvLy+88847sLe3xxNPPIGwsDB8+umnuHLlSoXjUKlUcHFxQZ8+fbBp0yZERkbi7t27GDhwINLS0iq8fypeYfsOGjRI6VCqDZVKpXQI1eq4ZGdnl3rFT01iTnWpaSrS9tXhuFWnc7I41Sm+6nC8TMWc6lJeTPaJqFarDolBRbRr1w5fffUVRo4cCRsbm2LXycvLw759+/DUU0/p1ff555+HiGDXrl0mj2vYsGEYM2YMEhMTjb4Mk6gi1Gq10iFUKxs2bEBiYqLSYZiEOdWlpqlI2/O41SzmdLzMqS7lxWSfiGoNEcHSpUvRvHlz2NjYwNnZGbNmzSqyXn5+PubPnw8fHx/Y2tqibdu22LZtGwBgzZo1sLe3h52dHXbt2oXnn38eTk5O8Pb2xpYtW/T2c+TIEXTt2hV2dnZwcnJCQEAA0tPTyyzD1G7evIkHDx7Ax8dHb7m/vz8AICYmRrfsv//9L5ycnLBo0aIKlztmzBgAwP79+3XLzK1tlXT8+HH4+PhApVJh1apVAIxrQ1O0k6HlhYaGwtraGl5eXrplr776Kuzt7aFSqXDv3j0AwPLly2Fvbw8LCwt06tQJnp6eUKvVsLe3R8eOHdG7d280bNgQGo0GLi4ueOONN4rEdP36dbRo0QL29vawtbVF7969cfz4cYPr/s9//hN2dnZwdHREYmIiZsyYgQYNGiA2NtagNqnIcVmxYgU0Gg3q1q2LiRMnol69etBoNOjRowdOnjxpdHtOmzYNM2bMwI0bN6BSqdCkSRMAFTvPq1tdjHXs2DG0atUKzs7O0Gg0CAgIwDfffAPgz3lcCu/l9ff3x9mzZwEAY8eOhZ2dHZydnbF7924AlduHqpuS2l5E8OGHH6Jly5awsbGBq6srXnzxRb2rxUratrTjYGqVPVaW9n5U2fFxzDC+Lsaq8WOGEBHVQMOGDZNhw4YZtc2bb74pKpVKPvjgA7l//75kZWXJ6tWrBYCcPXtWt97MmTPFxsZGtm/fLvfv35e5c+eKhYWFnDp1SrcfAHL48GFJS0uTxMRE6d27t9jb20tubq6IiDx48ECcnJwkPDxcsrOz5c6dOzJ06FBJSkoyqIzy6Natm7Rr167I8iNHjggAWbp0aZHXbG1tpV+/frr/9+7dK46OjvLuu++WWZ6/v784OzuX+Hp6eroAkIYNG+qW1ZS2LU//MkZ59n/t2jUBIP/61790y3777TcBICtXrtQtM6QNRUzXBw0tb+TIkeLp6am37dKlSwWA7tiJiLz99tsCQE6ePCmZmZly7949ee655wSA7Nu3T5KSkiQzM1NCQ0MFgJw7d063bb9+/cTPz09+/fVX0Wq18ssvv0i3bt1Eo9HI1atXDa57YZ2mTp0qK1eulKFDh8rly5cNbpOKHJcJEyaIvb29XLp0SR4+fCgXL16ULl26iKOjo8THxxvdnoGBgeLv76+3njHn+bPPPisA5P79+9WyLiJlj0WPioqKkgULFkhKSookJydL9+7dxc3NTa8MS0tL+f333/W2e+mll2T37t26/yuzD1XH8ae4tp8/f75YW1vL559/LqmpqRITEyMdO3YUd3d3uXPnTqnblnUcihvvDFWVY2VZ70eG4phRdXURMb8xY9u2bVJCWh/JX/aJqFbIzs5GREQE+vfvj+nTp8PFxQW2traoU6eO3noPHz7EmjVrMGTIEAQGBsLFxQXz5s2DWq3Gpk2b9Nbt0aMHnJyc4OHhgZCQEGRmZiI+Ph4AcOvWLaSnp6N169bQaDTw9PTEV199BXd3d6PKMIXCGfctLS2LvKZWq5Gdna37f+DAgUhPT8dbb71V4XILn3SQkZEBwDzbtjorrQ0ro51KK688WrVqBTs7O7i5uWHEiBEAAB8fH7i7u8POzk43g/Ljc044OjqiUaNGsLKyQuvWrfHJJ5/g4cOHWLdundF1X7JkCV577TV89dVXaNGiRbnr8ihD2snKykr3a2mrVq2wZs0aZGRkmKwPm+o8rw51MdawYcPw9ttvw9XVFXXq1MHgwYORnJyMpKQkAMCkSZOQn5+vF196ejpOnTqFF154AYDyfag6yM7OxocffoihQ4di1KhRcHZ2RkBAAD7++GPcu3dPd76VpKzjUJUqMlaW9n5UFfEV4phReWr6mMFkn4hqhevXryMrKwv9+vUrdb3Y2FhkZWWhTZs2umW2trbw8vIqdSI7a2trAIBWqwUA+Pn5oW7duhg1ahQWLFiAW7duVbiM8tJoNAD+vHf/cbm5ubC1tTV5mQCQmZkJEYGTkxMA82zbmuLxNqzsdnq8PFPt79E+XHhvflllBAQEwNnZWXe7SnXqI4a2U+fOnWFnZ1et+3BNrUthP8rPzwcAPP3002jWrBk2btyomyF969atCAkJ0X1hWp36kFIuXryIBw8eoHPnznrLu3TpAmtra73Lrg3x+HFQirFjZWnvR1URX0mq23lWnJpal5o2ZjDZJ6Ja4fbt2wAADw+PUtfLzMwEAMybN0/vGfJxcXElPtauOLa2tvj222/Rq1cvLFq0CH5+fggJCUF2drbJyjBU4X1tj99DmJWVhYcPH6JevXomLxMArl69CgC6b6bNsW1rqtrWTmq1WveBsqbW3cbGRpFfPSuDknXZt28f+vTpAw8PD9jY2BSZ90GlUmHixIm4efMmDh8+DADYvHkzXn75Zd06NbUPmVJqaioAwMHBochrLi4uuiu6SlLWcaguyjrWpb0fKY1jhmnU9DGDyT4R1QqFv24XXtJeksIvAyIiIiAien/R0dFGldm6dWvs2bMHCQkJCAsLw7Zt27Bs2TKTlmGIxo0bw9HREXFxcXrLr1+/DgBo27atycsE/pzQB/hz1n/APNu2pqpN7ZSXl4eUlBTdBJU1se5arRapqanw9vZWOpQKq+q6HD16FBEREQCA+Ph4DBkyBF5eXjh58iTS0tIQHh5eZJsxY8ZAo9Fg/fr1iI2NhZOTE3x9fXWv18Q+ZGouLi4AUGxSX9bxNfQ4VAeGHOuS3o+UxDGj/MxtzGCyT0S1Qps2bWBhYYEjR46Uul7hTN/nzp2rUHkJCQm4dOkSgD8H+cWLF6Njx464dOmSycowlJWVFV544QUcPXoUBQUFuuX79++HSqXC4MGDTV7mnTt3EBERAW9vb/zjH/8AYJ5tW1Mp0U5WVlYmu6zfGN999x0KCgrQsWNHAMrUvaK+//57iAi6d++uW6ZUe1ZUVdflzJkzsLe3BwBcuHABWq0WkydPhp+fHzQaTbGPX3V1dcXw4cOxc+dOLFu2DK+88ore6zWxD5lamzZt4ODggNOnT+stP3nyJHJzc9GpU6cStzX0OFQHZR3r0t6PlMQxo/zMbcxgsk9EtYKHhwcCAwOxfft2bNiwAenp6YiJiSkyiZBGo8HYsWOxZcsWrFmzBunp6cjPz8ft27fxxx9/GFxeQkICJk6ciCtXriA3Nxdnz55FXFwcunfvbrIyjPHWW2/h7t27ePvtt5GZmYno6GgsXboUY8aMQfPmzXXr7d+/36jH64gIHjx4gIKCAogIkpKSsG3bNvTs2ROWlpbYuXOn7p59c23bmkiJdmrSpAlSUlKwc+dOaLVaJCUlFbnaxBRyc3ORlpaGvLw8/PzzzwgNDYWvr6/uUZA1oY8UFBTg/v37yMvLQ0xMDKZNmwYfHx9dHQDD27NOnTpISEjArVu3kJGRAa1Wa/R5Xp3rUhKtVou7d+/i+++/131wL7y649ChQ3j48CGuXbtW4r3lkyZNQk5ODvbu3YtBgwbpvVYT+pCpPd72lpaWmDFjBnbs2IEvvvgC6enpuHDhAiZNmoR69ephwoQJJW5beOuYIcdBaWUd69Lej6oSxwyOGSUyaD5/IqJqpjyPDsrIyJBx48aJm5ubODg4SK9evWT+/PkCQLy9veX8+fMiIpKTkyNhYWHi4+MjVlZW4uHhIYGBgXLx4kVZvXq12NnZCQBp2rSp3LhxQ9atWydOTk4CQHx9feXq1aty69Yt6dGjh7i6uoqlpaXUr19f3nzzTcnLyyuzDGNER0dLz549pV69egJAAIiXl5f06NFDjhw5orfukSNHpGvXrmJjYyP16tWTWbNmycOHD/XW+frrr8XR0VEWLlxYYpm7d++Wtm3bip2dnVhbW4uFhYUAEJVKJS4uLtK1a1d59913JTk5uci2NaVtq9ujrz744APx9PQUAGJvby9Dhw6VlStXipeXlwAQOzs7GTx4sMFtKGKadjKmvOTkZOnbt69oNBpp3LixTJkyRWbNmiUApEmTJhIfHy/Lly/X7a9Ro0Zy7NgxWbJkiTg7OwsA8fT0lC+//FK2bt2qaw9XV1fZsmWLiIhs2rRJ+vbtK3Xr1hUrKytxc3OTESNGSFxcnF7cpdU9PDxcbG1tdY+N/Pzzzw1uDxGp8HGZMGGCqNVqadCggVhZWYmTk5O8+OKLcuPGDb1yDGlPEZGff/5ZfH19xdbWVnr16iV37twx6Dw/ceKEtG7dWnd+e3l5yaJFi6pVXf71r3+Jv7+/buwr6W/Hjh26ssLCwqROnTri4uIiQUFBsmrVKgEg/v7+eo/2EhHp0KGDzJkzp9j2qcw+VN3GH5Hi+1FBQYEsXbpUmjZtKgezNowAACAASURBVGq1WlxdXWXIkCESGxtb5ralHYdp06YVGe8MVdVjZVnvR4bgmMExo6JjRmmP3lOJ/P+0gURENUhQUBAAICoqSuFIyBxVdv9i/6WSTJw4EVFRUUhOTlY6lAqr6XUZOHAgVq1ahcaNG/8fe/cdXkWZuH38Puk9hN5DQq8CCwoIBpa1C6IUUTCCsIJlCQoKgstGEBCj1EWRIj+ju0CCCoptUVZERaQaOhikC6EHSCDtef/w5awHAiSkTM7k+7mu/MFkzsydOfMccufMmadY98vrD/LD3cfZH7n7z2LVa0ZCQoIeeugh5VLrE7mMHwAAoASxevqxwuROP8sfL/FNSkqSn59fsf/SDtwIdxpn1+NOP4s7vGZQ9gGgBNmxY4fLtCxX++rdu7fVUWFTnINX4piUDiNGjNDu3bu1a9cu9e/fX+PGjbM6EuSe488dMyP/3OE1w8vqAACA/2nQoEFul2EBxYZz8ErFdUxGjRql+fPnKyMjQxEREYqLi1OPHj2KfL9FwR1/loCAADVo0EDVqlXTzJkz1ahRI6sjQe75msRrRv6548/iDq8ZfGYfgFviM4coSnxmFoBVeP0BkB98Zh8AAAAAgFKEsg8AAAAAgM1Q9gEAAAAAsBnKPgAAAAAANkPZBwAAAADAZij7AAAAAADYDGUfAAAAAACboewDAAAAAGAzlH0AAAAAAGyGsg8AAAAAgM1Q9gEAAAAAsBnKPgAAAAAANkPZBwAAAADAZrysDgAAN2rx4sVyOBxWx4BN9ejRo0i3z/kL4Gp4/QFQGBzGGGN1CADIr9WrV+vAgQNWx3B77777rrZt26ZJkyZZHaXEqVGjhtq2bVsk2+b8xR9NmTJFkvTss89anAQlBa8/uGTFihWaO3eu4uLiVK1aNavjoATr1avX5YsSKfsAUIp99NFH6tGjh44dO6ayZctaHQcolS79gpaQkGBxEgAlydmzZ1W/fn317NlT06ZNszoO3E8in9kHgFIsKipKkvT9999bnAQAAPzR2LFjdeHCBY0ZM8bqKHBTlH0AKMXKli2rRo0aaeXKlVZHAQAA/19ycrJmzJihcePGqVy5clbHgZui7ANAKRcVFUXZBwCgBHnuuedUu3ZtPfHEE1ZHgRuj7ANAKRcVFaWNGzfq9OnTVkcBAKDUW7FihT7++GNNnjxZ3t7eVseBG6PsA0ApFxUVpZycHP3www9WRwEAoFTLzs7W0KFD1bVrV915551Wx4Gbo+wDQClXsWJFNWjQgEv5AQCw2KxZs7Rz507FxcVZHQU2QNkHAPC5fQAALHbq1CnFxsYqJiZG9erVszoObICyDwBQVFSU1q9fr7Nnz1odBQCAUukf//iHPDw8NHr0aKujwCYo+wAARUVFKSsrS6tXr7Y6CgAApc727ds1a9YsjR8/XqGhoVbHgU1Q9gEAqlKliurWrcul/AAAWOC5555T48aN1b9/f6ujwEa8rA4AACgZ+Nw+AADFb9myZfriiy/0zTffyNPT0+o4sBHe2QcASPq97K9du1bnz5+3OgoAAKVCZmamhg8frp49eyoqKsrqOLAZyj4AQJLUsWNHZWRk6Mcff7Q6CgAApcKMGTO0b98+vfbaa1ZHgQ1R9gEAkqTq1asrMjKSS/kBACgGx44d07hx4zR8+HDVqlXL6jiwIco+AMCJz+0DAFA8XnrpJQUGBmrEiBFWR4FNUfYBAE5RUVFas2aN0tPTrY4CAIBtbd26Ve+8844mTpyooKAgq+PApij7AACnqKgoXbx4UT/99JPVUQAAsK1nn31WN910k/r06WN1FNgYZR8A4FSrVi2Fh4dzKT8AAEXkgw8+0FdffaWpU6fKw4M6hqLD2QUAcHHbbbdR9gEAKAIZGRkaOXKk+vTpo/bt21sdBzZH2QcAuIiKitLq1at18eJFq6MAAGArb7zxhg4dOqTx48dbHQWlAGUfAOAiKipK6enpWrt2rdVRAACwjaNHj+rVV1/Viy++qJo1a1odB6UAZR8A4KJOnTqqUaMGl/IDAFCIRo4cqdDQUA0bNszqKCglKPsAgCt06NCBsg8AQCHZsGGD4uPj9frrrysgIMDqOCglKPsAgCtERUXp+++/V0ZGhtVRAABwa8YYDR06VG3atFHPnj2tjoNShLIPALhCVFSU0tLStGHDBqujAADg1hYuXKjvv/9eU6dOlcPhsDoOShHKPgDgCvXr11fVqlW5lB8AgAJIT0/Xiy++qP79+6t169ZWx0EpQ9kHAOSqffv2lH0AAApg0qRJOnnypMaNG2d1FJRClH0AQK6ioqL03XffKSsry+ooAAC4nYMHD+r111/XSy+9pCpVqlgdB6UQZR8AkKuoqCidPXtWGzdutDoKAABu54UXXlClSpUUExNjdRSUUpR9AECuGjVqpIoVK3IpPwAA+bR69WotXLhQkydPlq+vr9VxUEpR9gEAuXI4HOrQoQNlHwCAfMjJydHQoUPVqVMn3X///VbHQSlG2QcAXFVUVJRWrVql7OzsK76Xk5NjQSIAAEoGY0yuy+Pj47V+/XpNmTKlmBMBrrysDgAAKLmioqJ05swZJSUlyc/PTytXrnR+bdy4UZUqVbI6IuBWjh8/rtTUVJdl58+flyTt2bPHZXlISIjKly9fbNkA5M/MmTO1efNmvfLKK6pQoYIk6dy5cxo9erQGDRqkZs2aWZwQpR1lHwBwBWOMtm7dqm+//VYhISHOm/V5eXkpJydHDoeDEgLcgKVLl2rgwIG5fu+zzz5z+ffcuXM1YMCA4ogF4AasW7dO7777rhYuXKgJEyZo0KBBGj9+vNLT0/Xyyy9bHQ+Qw1zt+hMAQKnzySefaP78+frvf/+r06dPy8vr978JXz79Xrly5XT8+HErIgJu7dSpU6pUqZIyMzOvuZ63t7eOHj2qsLCwYkoGIL8aNWqk7du3S5I8PDwUGRmpY8eOKTY2VkOHDrU4HaBEPrMPAHCqXr26Pv74Y50+fVrS7yX/8qIvSZUrVy7uaIAthIWF6a677nL+IS03Xl5euvvuuyn6QAmWkZGh3bt3O/+dk5OjX3/9VWfOnNGSJUu0bds2C9MBv6PsAwCcWrRooRdeeEGenp7XXK9GjRrFlAiwn759++Z608tLsrOz1bdv32JMBCC/tmzZcsUfwy+N6x9++EHNmjVTTEyMzpw5Y0U8QBJlHwBwmX/84x+qXbv2VQu/l5eXqlWrVsypAPvo2rWr/P39r/p9Pz8/3XvvvcWYCEB+JSUlycMj9yqVmZmp7OxsTZ8+XQ0aNNChQ4eKOR3wO8o+AMCFr6+v4uPjrzqlkKenp6pUqVLMqQD78PPz0wMPPCBvb+8rvuft7a3u3bsrICDAgmQA8mrTpk3X/DiOp6enfH199dZbb/EHcliGsg8AuMItt9yiZ555JtdfZHJycphyDyigRx55JNeb9GVmZuqRRx6xIBGA/Pjpp5+UkZGR6/e8vb1VpkwZrVq1St26dSvmZMD/cDd+AECu0tLS1LBhQx06dMjl88UOh0OJiYnq3r27hekA95aVlaWKFSvq1KlTLsvLlCmjlJSUXN/1B1AyGGMUEhKic+fOXfE9b29vRURE6Msvv1StWrWKPxzwP9yNHwCQu4CAAP3f//2fcnJyXJYbY7gbP1BAXl5e6t27t3x8fJzLvL299cgjj1D0gRLu119/zbXoe3l5qV27dlqzZg1FHyUCZR8AcFWdOnXSgAEDrricn8/sAwX38MMPu1wGnJmZqYcfftjCRADyYtOmTXI4HC7LPDw89Mgjj2j58uUqU6aMRckAV5R9AMA1vfHGGypfvrzLXYf5zD5QcO3bt1fVqlWd/65cubJuvfVWCxMByIuff/7Z5aocSfr73/+ud999lytzUKJQ9gEA1xQSEqI5c+Y4L+f39/dXYGCgxakA9+dwONS3b1/5+PjI29tb0dHRV7xbCKDk2bBhgzIzM+Xh4SFvb2+9//77io2NtToWcAXKPgDguu677z7n5cUVK1a0OA1gH5cu5ecu/ID7WL9+vXJychQUFKQVK1aoT58+VkcCcsXd+AHYXs+ePa2OYAsXL17Ul19+qeDgYHXq1MnqOKVWYmKi1RFuCOPw6r744gtJ0l133WVxkpLLXc97q61evVqTJ0+2OoatZGRk6OOPP1ZAQIBuu+02BQUFWR2pyLRt21bPPfec1TFw47gbPwD7W7x4sQ4ePGh1DLfn6+urFi1aKCAgwOoopdLBgwe1ePFiq2PcMMbh1YWHhys8PNzqGCWSu5/3Vjtw4ADHr5CdPn1aZcuWVefOnW1d9H/88UetXr3a6hgoIK/rrwIA7u/ZZ59Vr169rI5hC19//bU6d+5sdYxSJyEhQQ899JDVMQqEcZi75ORkSVLt2rUtTlLy2OG8Lwm4MqLwbN++XREREfLz87M6SpHiaix7oOwDAPKFog8ULko+4D4aNmxodQQgz7iMHwAAAAAAm6HsAwAAAABgM5R9AAAAAABshrIPAAAAAIDNUPYBAAAAALAZyj4AAAAAADZD2QcAAAAAwGYo+wAAAAAA2AxlHwAAAAAAm6HsAwAAAABgM5R9AAAAAABshrIPAAAAAIDNUPYBAAAAALAZyj4AXMfAgQMVHBwsh8OhTZs2WR3HUmPHjlWjRo0UEhIiX19f1alTRy+88ILOnTuX72198MEHioyMlMPhcPny8fFRxYoV1bFjR8XFxenUqVNF8JPA3dhlHP773/9W69atFRwcrPDwcPXv319HjhzJ93YYPygudhh7mZmZGjNmjCIjI+Xj46Nq1app+PDhSk9Pz/e2GHtwJ5R9ALiOuXPnas6cOVbHKBFWrFihZ555Rnv37tXx48c1YcIETZ06VT179sz3trp37649e/aodu3aCg0NlTFGOTk5SklJUUJCgiIiIjRixAg1btxY69atK4KfBu7EDuNw0aJF6tOnj3r27KmDBw9q6dKl+vbbb3X33XcrKysrX9ti/KC42GHsDR06VHFxcZowYYJOnDihf/3rX5ozZ44GDhyY720x9uBOKPsAUMqkp6erXbt2N/TYoKAgDRo0SGXLllVwcLB69eqlBx54QF988YUOHDhQ4GwOh0NlypRRx44dNX/+fCUkJOjo0aO69957debMmQJv32oFOfZwf2+//baqVq2q559/XqGhoWrevLmee+45bdq0SWvWrCnw9u0+foAbsWfPHs2aNUvR0dHq3bu3goOD1bFjRw0ZMkT//ve/tX379gLvg7GHkoqyDwB54HA4rI5QaObNm6eUlJQbeuyyZcvk6enpsqx8+fKSpLS0tAJnu1yPHj3Ur18/paSkaNasWYW+/eJWkGMP9x+HBw4cUJUqVVx+jho1akiS9u3bV+j7s9v4gXXceeytXbtWOTk5uuWWW1yW33XXXZKkL7/8stD3ydhDSUHZB4DLGGMUFxen+vXry9fXV6GhoXr++edd1nnttdcUEBCg4OBgpaSkaNiwYapWrZp27twpY4wmT56shg0bytfXV2FhYerWrZt27NjhfPz06dPl5+enihUravDgwapSpYr8/PzUrl27K97hy8v2hgwZIh8fH1WuXNm57Omnn1ZgYKAcDoeOHz8u6fdLGYcNG6bk5GQ5HA7VqVOnwMfr0KFD8vf3V0REhHPZF198oZCQEI0fP77A2+/Xr58k6fPPP5fEsS8t8jIOJSk7O1tjxoxRzZo15e/vr2bNmmnRokWSpDfffFOBgYEKCAjQ0qVLdffddyskJETVq1fXggULXLazcuVK3XzzzQoICFBISIiaNm2q1NTU6+4jPyIjI6/4Y8+lz+tHRkY6lxXl+JHc65ih+Nlt7Hl4/F53/P39XZbXrVtXklze2WfswXYMANicJLNo0aI8rz969GjjcDjMG2+8YU6dOmXS0tLMzJkzjSSzceNGl/UkmZiYGDNjxgzz4IMPmu3bt5sxY8YYHx8f895775nTp0+bpKQk07JlS1O+fHlz5MgR5+MHDRpkAgMDzbZt28yFCxfM1q1bTevWrU1wcLDZv3+/c728bq9Pnz6mUqVKLj9LXFyckWSOHTvmXNa9e3dTu3btfB3Dqzl//rwJDg42Q4YMcVm+bNkyExwcbMaOHXvdbdSuXduEhoZe9fupqalGkqlRo4ZzWWk89osWLTLu/N92UY3D4cOHG19fX7N48WJz6tQpM2rUKOPh4WHWrl3r3I4k8/XXX5szZ86YlJQU06FDBxMYGGgyMjKMMcacO3fOhISEmEmTJpn09HRz5MgR8+CDDzqfu+vtI6+++eYb4+3tbaZPn25SU1PNli1bTMOGDc2dd97psl5Rjx93Ombuft5b7UaOn93GXlJSkpFk/v73v7ssz8rKMpLMAw884FzG2PufHj16mB49euTrMShxEnj1BGB7+SkZaWlpJiAgwNx+++0uyxcsWHDVsp+enu7y+KCgINO7d2+Xx//0009GkssvEIMGDbriF4W1a9caSebll1/O9/asKPujR4829erVM6mpqTe8jev9wmSMMQ6Hw5QpU8Zlv6Xt2Lt76SmKcZienm4CAgJcnqO0tDTj6+trnnrqKWNM7ufKpeLyyy+/GGOM2bJli5Fkli1bdkWWvOwjP1566SUjyflVvXp1c+DAgXxv55L8jh93O2buft5bLb/Hz65j76677jJly5Y1X3/9tUlPTze//fabSUhIMA6Hw9x333352tYldh97lH1bSOAyfgD4g19++UVpaWnq3LnzDT1+69atOnfunFq1auWyvHXr1vLx8bnuTbhatWqlgIAA52XiBd1eUfrwww+VkJCgL7/8UsHBwUW2n/Pnz8sYo5CQkGuuV5qOvd3ldRzu3LlTaWlpatKkiXOZv7+/Kleu7PJRi8v5+PhI+n06Lun3S+grVqyovn37KjY2Vnv37i3wPnIzevRozZ49W19//bXOnTunPXv2qF27dmrbtm2h3OAyN5ePH3c7Zihedh17CxcuVM+ePRUdHa2yZcvq1ltv1UcffSRjjMqVK5evbeUVYw8lAWUfAP7g4MGDkqQKFSrc0ONPnz4t6fe71l+uTJkyOnv27HW34evrq2PHjhXa9orCwoUL9eqrr+qbb75RrVq1inRfu3btkiQ1aNDgmuuVlmNfGuR1HJ4/f16S9NJLL7nMd71v37583TDS399fK1asUPv27TV+/HhFRkaqd+/eSk9PL7R9/Pbbb5o0aZKeeOIJ/fnPf1ZgYKAiIiI0Z84cHT58WHFxcXneVn5cPn7c6Zih+Nlx7ElSaGioZs2apYMHDyotLU3Jycl64403JElVq1bN17byirGHkoCyDwB/4OfnJ0m6ePHiDT2+TJkykpRrETx9+rSqV69+zcdnZma6rFfQ7RWFGTNm6P3339eKFSuK7JekP/riiy8kSXffffc11ysNx760yOs4vFRIpkyZImOMy9fq1avztc/GjRvrk08+0eHDhzVixAgtWrRIr7/+eqHtY/fu3crOzr5izISEhKhs2bLaunVrvvLm1eXjx52OGYqfHcfe1axdu1aS1KlTpwJvKzeMPZQElH0A+IMmTZrIw8NDK1euvOHHBwUFad26dS7L16xZo4yMDP3pT3+65uO/+eYbGWPUpk2bfG/Py8vLeZlfUTDGaMSIEdq8ebOWLFmS6zvehe3IkSOaMmWKqlevrscff/ya69r52Jc2eR2HNWrUkJ+fnzZt2lSg/R0+fFjbtm2T9Psv5BMnTlTLli21bdu2QtvHpT8O/fbbby7Lz549q5MnTzqn4CtMuY0fdzpmKH52HHtXM2fOHEVERCgqKqrQt83YQ0lB2QeAP6hQoYK6d++uxYsXa968eUpNTVVSUpJmz56dp8f7+flp2LBh+vDDD/X+++8rNTVVmzdv1pNPPqkqVapo0KBBLuvn5OTo1KlTysrKUlJSkoYOHaqaNWs6p+zJz/bq1KmjkydPasmSJcrMzNSxY8dynbu7bNmyOnz4sPbu3auzZ8/muaRu27ZNr732mubMmSNvb2+XSwQdDodef/1157qff/55vqYvMsbo3LlzysnJkTFGx44d06JFi3TrrbfK09NTS5Ysue5n9u187EubvI5DPz8/9e/fXwsWLNCbb76p1NRUZWdn6+DBg1eU6ms5fPiwBg8erB07digjI0MbN27Uvn371KZNm0LbR0REhDp16qQ5c+bo22+/VXp6ug4cOOA8jwYMGOBctyjHjzsdMxQ/O449Sbr55pu1b98+ZWVlae/evRo+fLi++uorzZs3z/mZeImxBxsqkvv+AUAJonxO+XX27FkzcOBAU65cORMUFGTat29vxowZ47xz9s8//2wmTZpk/P39ndPqvPfee87H5+TkmLi4OFO3bl3j7e1twsLCzAMPPGB27tzpsp9BgwYZb29vU61aNePl5WVCQkJMt27dTHJysst6ed3eiRMnTKdOnYyfn5+JiIgwf/vb38zzzz9vJJk6deo4p5TbsGGDCQ8PN/7+/qZ9+/YuU8hdy+bNm13uIn75V1xcnHPdzz77zAQHB5tXXnnlqtv7+OOPTbNmzUxAQIDx8fExHh4eRpLz7sU333yzGTt2rDlx4oTL40rjsTfG/e9KXhTj0BhjLl68aEaMGGFq1qxpvLy8TIUKFUz37t3N1q1bzcyZM01AQICRZOrWrWuSk5PN7NmzTUhIiJFkwsPDza5du8zevXtNu3btTFhYmPH09DRVq1Y1o0ePNllZWdfdR34cP37cDB061NSpU8f4+vqaoKAgc+utt5qPPvrIZb2iHD/udszc/by32o0cPzuOvdtvv92UKVPGeHl5mbCwMHPvvffmOhUdY+9/uBu/LSQ4jDGmOP6oAABWcTgcWrRokXr16mV1FBeDBw9WYmKiTpw4YXWUUscdj31CQoIeeughuet/2yV1HKJkc/fz3mocP9yonj17SpISExMtToICSOQyfgCwUHZ2ttURSi2OPQAAsDPKPgCUYjt27Ljis/e5ffXu3dvqqECJw/gBrMHYA/LGy+oAAFAajRo1SvPnz1dGRoYiIiIUFxenHj16FHuOBg0alLrLO0vKsYf7K43jBygJGHtA3lD2AcACEyZM0IQJE6yOUSpx7AEAQGnAZfwAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzXhZHQAAisOUKVOUmJhodQy4iZMnT8rT01OhoaFWR3E6ePCg1REKjHGI/LLDeV8S9OzZ84Yed+bMGWVmZqp8+fKFnAgl3Y8//qg2bdpYHQMFRNkHYHs9evSwOgLczC+//KL9+/erSpUqql+/fon4Rbd69epufS67c/aitm7dOklSq1atLE5S8rj7eW+1GjVq5Pv4GWN05MgR7d69WykpKapevXqJeA1E8WrTpo3atm1rdQwUkMMYY6wOAQBASWKM0bJlyzRp0iR9//33uvXWWzVixAjdd999cjgcVseDzfTq1UuSlJCQYHESlGbnzp3Tv//9b02dOlU7duxQ586dNWTIEF73APeVyGf2AQC4jMPhUJcuXfTdd99p1apVCgsL0/33369mzZopPj5emZmZVkcEgELx66+/auTIkQoPD1dMTIxatWqlzZs3a/ny5erSpQtFH3BjlH0AAK6hffv2+uSTT/Tzzz+rRYsWGjBggOrWratp06YpLS3N6ngAcEPWr1+v6Oho1atXT/Hx8frb3/6mgwcPKj4+Xo0bN7Y6HoBCQNkHACAPmjZtqvj4eO3evVv333+/Ro0apfDwcMXGxurEiRNWxwOA68rIyFBiYqLatm2rVq1aadu2bZo3b5727dun2NhYlStXzuqIAAoRZR8AgHyoVauWpk2bpr179+rpp5/WjBkzVKtWLcXExOjAgQNWxwOAKxw7dkyTJk1S7dq11adPH9WoUUPff/+91q1bp+joaHl7e1sdEUARoOwDAHADKlSooNjYWO3bt0+vvPKKPvzwQ9WuXVvR0dHatm2b1fEAQD///LMGDRqk8PBwTZw4UQ8++KCSk5OVkJCgdu3aWR0PQBGj7AMAUABBQUGKiYlRcnKy5s6dq3Xr1qlp06bq0qWLVq9ebXU8AKVMTk6OvvrqK3Xp0kUtWrTQN998o4kTJ+rw4cOaNm2aatSoYXVEAMWEsg8AQCHw8fFRdHS0tmzZoiVLluj48eNq166d8wZ/zHQLoCidPXtWs2fPVpMmTXTHHXfowoULWrp0qXbs2KGYmBgFBARYHRFAMaPsAwBQiDw8PJzv6v9x2r7mzZsrPj5eWVlZVkcEYCN/nDpv2LBh6tChg7Zs2cLUeQAo+wAAFJVL7+pv3LhRN910kwYMGKB69epp2rRpSk9PtzoeADd2+dR5Q4YM0b59+/T222+rUaNGVscDUAJQ9gEAKGI33XST4uPjtWvXLnXp0kUvvviiatWqpdjYWJ06dcrqeADcxNWmztu/f79iY2NVtmxZqyMCKEEo+wAAFJOIiAjntH1PPvmkpk+frvDwcMXExOjQoUNWxwNQQqWkpGjSpEmKjIxU3759VaNGDf3www/OqfO8vLysjgigBKLsAwBQzCpWrOictm/cuHH64IMPFBkZqejoaO3YscPqeABKiEtT59WqVUuvvvqqunfvrl9++UUJCQlq27at1fEAlHCUfQAALBIcHKyYmBjt2bNHc+bM0U8//aTGjRurS5cu+umnn6yOB8ACV5s679ChQ0ydByBfKPsAAFjs0rR927Zt05IlS5SSkqJbbrnFeYM/APZ3aeq8xo0b684772TqPAAFRtkHAKCEuDRt35o1a5zT9nXt2lUtWrRQfHy8srOzrY4IoJDt2bPHZeq82267janzABQKyj4AACXQpXf1N2zYoKZNm+rxxx9X/fr1NW3aNF245tDkvgAAIABJREFUcMHqeAAK6LvvvlOvXr1Uv359vffeey5T5zVs2NDqeABsgLIPAEAJduld/Z07d+ree+/VyJEjndP2nT592up4APLh0tR5bdq0UYcOHbRnzx7NmzdP+/btY+o8AIWOsg8AgBuoXbu2c9q+wYMHa+rUqc5p+3777Ter4wG4hsunzqtZs6ZWr17N1HkAihRlHwAAN1KpUiXFxsZq//79Gjt2rBITExUREaHo6Gjt3r3b6ngA/mDTpk3OqfPeeOMN9e3bV8nJyUpISFCbNm2sjgfA5ij7AAC4oZCQEMXExOjXX3/V7Nmz9eOPP6pBgwbq0qWL1q1bZ3U8oNTKycnRJ598ottvv10tWrTQypUrNXHiRO3du1evvvqqqlevbnVEAKUEZR8AADfm6+ur6Oho7dixQ0uWLNFvv/2m1q1bM20fUMwuTZ3XqFEjdevWTZL08ccfa/v27UydB8ASlH0AAGzg0rR969at06pVq+Tv76+uXbuqVatWTNsHFKFLU+fVrFlTw4YNU1RUFFPnASgRKPsAANhM+/bttXz5cq1atUpVqlRRv3791KBBA02bNk0XL160Oh5gC5emzqtXr54SEhI0cuRI7d+/n6nzAJQYlH0AAGzq0qX8SUlJatu2rZ5//nnVq1dP06ZN0/nz562OB7idixcvKj4+XjfddJNz6rx33nlHu3bt0ogRIxQWFmZ1RABwouwDAGBzTZo0UXx8vHbv3q1u3bpp1KhRqlWrlmJjY3XixAmr4wEl3tGjR51T5/31r39V/fr1mToPQIlH2QcAoJQIDw/XtGnTtG/fPj399NOaMWOGwsPDFRMTo/3791sdDyhxNm7c6Jw6b/LkyXr00UeZOg+A26DsAwBQypQvX16xsbHat2+fxo8fr48++kh16tRRdHS0tm3bZnU8wFJ/nDqvZcuWWrNmjaZNm8bUeQDcDmUfAIBSKigoSDExMUpOTtbcuXO1fv16NWnSRF26dNEPP/xgdTygWKWmpmratGmqXbu2y9R5Gzdu1BNPPCF/f3+LEwJA/lD2AQAo5by9vRUdHa3Nmzdr6dKlOnHihG699VbnDf6MMVZHBIpMcnKyRo4cqfDwcP3973/XHXfcoa1btzJ1HgC3R9kHAACSJA8PD+e7+qtWrVJYWJjuv/9+NW/eXPHx8crKyrI6IlBoLk2dV79+fSUmJmrkyJHat2+f3n77bTVo0MDqeABQYJR9AABwhUvv6m/atEk33XSTBgwYoLp162ratGlKS0uzOh5wQy6fOu/w4cNasGCBdu7cydR5AGyHsg8AAK6qWbNmio+P165du9S1a1e9+OKLzmn7Tp48aXU8IE+OHj2q2NhY1ahRwzl13o8//qjvvvtOPXv2ZOo8ALZE2QcAANcVERHhvCP5U089penTpzun7Tt48KDV8YBc/XHqvLfeekuPP/649uzZo4SEBN1yyy1WxwOAIuUw3HUHAADk09mzZ/XOO+8oLi5Ox44d00MPPaRRo0bxWefr+L//+z9NnTpV2dnZzmXHjh2TJFWoUMG5zNPTU0OHDlW/fv2KO6Lby8nJ0aeffqrp06frq6++UvPmzfXkk0/q0Ucf5Y76AEqTRMo+AAC4YRkZGVq4cKEmTpyoXbt26Z577tFLL73Eu6ZXsXPnzjz/QWT79u388SQfUlNTNX/+fE2dOlX79+/XPffco5iYGHXu3Jk76gMojRK5jB8AANwwHx8fRUdHa+vWrVqyZIlSUlLUpk0bpu27ivr166tp06bXLJ8Oh0NNmzal6OdRblPnbdu2TZ988on+8pe/UPQBlFqUfQAAUGCXpu1bs2aNc9q+rl27qmXLloqPj3e5bP1qUlNTNXPmzGJIa63o6Gh5enpe9fteXl567LHHijFRyfDTTz/la/1rTZ1Xv379IkoJAO6Dsg8AAArVpXf1N27cqKZNm+rxxx9XvXr1NG3aNKWnp1/1cW+//baeeeYZTZw4sRjTFr9HHnnkmn/8yMrK0kMPPVSMiaz3/vvv69Zbb71u4b80dV6zZs1cps7btWsXU+cBwGUo+wAAoEg0b95c8fHx2rlzp+677z6NHDlSERERio2N1enTp13WvXjxouLi4iRJo0aN0ssvv2xF5GJRtWpVtWvXTh4eV/4a5uHhoXbt2ql69eoWJLPGjBkzFB0drZycHE2ZMiXXdY4cOaLY2FhVr15df/3rX9W8eXNt2rTJOXXeta6UAIDSirIPAACKVO3atZ3T9g0ePFjTpk1TzZo1FRMTo8OHD0uS3n33XZ04ccL5mJdfflkjRoywKnKRe/TRR3P9LLnD4VB0dLQFiawxadIkxcTEyBijnJwcJSYmukzluGHDBg0aNEgRERGaNWuWBgwYoD179ig+Pl433XSThckBoOTjbvwAAKBYnTp1SjNnztT06dN17tw59evXT8uWLdOhQ4eUk5PjXM/Dw0NDhw7VG2+8YWHaonHy5ElVqlRJWVlZLss9PT119OhRlStXzqJkxcMYo+HDh2vKlCkuN3H09vbW8OHD1bZtW6bOA4CCYeo9AABgjbS0NL3zzjuKi4vTgQMHcr1zv4eHh/7617/qrbfest1d1e+55x4tX77cWfg9PT11++236/PPP7c4WdHKzs7WwIEDFR8f7/LHnUt8fX2VlZWl+++/XzExMbrtttssSAkAbo+p9wAAgDUCAgL0zDPPKCwsLNfPr0tSTk6O5syZoyeeeCLXYujO+vbt6/IzGWP06KOPWpio6F28eFHdu3fXe++9d9XnMzMzU+PGjdMHH3xA0QeAAuCdfQAAYJnly5frjjvuuO56Hh4eeuyxxzR37tyr/mHA3Zw/f17ly5fXhQsXJP3+jvbx48cVFBRkcbKice7cOXXt2lWrVq264uMLf+RwOFSnTh3t3LnTdldzAEAx4p19AABgnfHjx8vLy+u66+Xk5Cg+Pl59+vS5ZlF0J4GBgeratau8vb3l5eWlbt262bbonzx5Uh07dtR333133efPGKPdu3frq6++KqZ0AGBPlH0AAGCJ9evXa+XKlTLG5Okd3OzsbCUmJio6Ovqa89S7k0t/vMjOztYjjzxidZwiceDAAd18881KSkpSZmZmnh7jcDhseWNGAChO1/9TOgAARSAhIcHqCLBYcnKyHn74YZ0+fVonT57UiRMndOLECaWmprqUeQ8PD3l5eckYo8zMTC1YsEB79uxRTEyM28+vnp2dLT8/PxljdO7cOduNi8OHD2vs2LE6deqUpN+fSw8PDzkcDuXk5OT6RxsfHx8FBARo9+7deuedd2x7tQOurlevXlZHAGyBz+wDACzBZ3EBALmhngCFIpF39gEAllm0aBHv4OCGGWOUnZ2dp8/8l2T//e9/5XA41LFjR6ujAJZKSEjQQw89ZHUMwDbc+39HAABQajkcDrcv+pIUFRVldQQAgA25//+QAAAAbswuUwkCAEoW/ncBAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQCArU2cOFGhoaFyOBzatGmT1XHyrH///vLz85PD4dCFCxesjlNsPvvsM4WGhuqTTz4plPWKyuuvv66KFSvK4XBo1qxZlmTITevWreXp6anmzZsX+rYHDhyo4ODg646lq61n9XOWm507d+pvf/ubGjdurODgYHl5eSk0NFT16tXTvffeq9WrV1sdEQBuGGUfAGBrL774ot5++22rY+Tb/PnzNXz4cKtjFDtjTKGuV1SGDx+uH374wdIMuVm7dq06depUJNueO3eu5syZc8PrWf2cXW7evHlq2rSpkpKSNHnyZB04cEDnz5/Xxo0bNW7cOJ0+fVqbN2+2OiYA3DAvqwMAAJAX6enp6ty5c4ksWCg89957r86cOeOyLLfnPrf18D8Oh8PqCFcoSc/Zjz/+qEGDBikqKkpffvmlvLz+9ytxZGSkIiMjVaZMGe3evdvClNdm5Wsir8eAe6DsAwDcwrx585SSkmJ1DEuUxOJWnErzc3+jvL29i2S7eT0Xi+OcNcZo8eLFOnXqlJ544ol8PfaVV15Rdna2Jk6c6FL0/+jOO+/UnXfeWRhRi4SV44IxCbgHLuMHAJR4Q4cO1bBhw5ScnCyHw6E6depI+v2X/cmTJ6thw4by9fVVWFiYunXrph07dlxze0ePHlWtWrXk5eWlu+66y7k8OztbY8aMUc2aNeXv769mzZpp0aJFkqQ333xTgYGBCggI0NKlS3X33XcrJCRE1atX14IFC274Z3vvvffUqlUr+fn5KTAwULVq1dK4ceOc3/fw8NCnn36qu+++W6GhoapSpYreeecdl22sWrVKjRo1UmhoqPz8/NS0aVN9+eWXkqTXXntNAQEBCg4OVkpKioYNG6Zq1app586deco3ffp0+fn5qWLFiho8eLCqVKkiPz8/tWvXTmvWrHFZN6/Px8qVK3XzzTcrICBAISEhatq0qVJTU/Xdd9+pZs2acjgc+uc//ykp9+c+t/Xyuv/8PI/XOq4Fda1zberUqQoMDJSHh4f+9Kc/qVKlSvL29lZgYKBatmypDh06qEaNGvLz81OZMmX0wgsvXLH9X375RQ0aNFBgYKD8/f3VoUMHfffdd3nOcOl4xsXFqX79+vL19VVoaKief/75K/aVl/Vye87y81xkZ2drwoQJql+/vvz9/VW+fHlFRERowoQJ6tWrl3O9L774QiEhIRo/fvxVj31GRoa+/vprlStXTjfffPNV18vt5yzM80u69vi/1vl3tdfEwnoNK+x9A7CIAQDAApLMokWL8rx+9+7dTe3atV2WjRkzxvj4+Jj33nvPnD592iQlJZmWLVua8uXLmyNHjjjXW7BggZFkNm7caIwxJiMjw3Tv3t0sXbrUZXvDhw83vr6+ZvHixebUqVNm1KhRxsPDw6xdu9YYY8zo0aONJPP111+bM2fOmJSUFNOhQwcTGBhoMjIy8n0MpkyZYiSZiRMnmhMnTpiTJ0+at99+2/Tp0+eK/Z0+fdqcPHnS3HPPPcbX19ecP3/euZ3ExEQTGxtrTp48aU6cOGHatGljypUr5/z+pe3ExMSYGTNmmAcffNBs3749zzkHDRpkAgMDzbZt28yFCxfM1q1bTevWrU1wcLDZv3+/c728PB/nzp0zISEhZtKkSSY9Pd0cOXLEPPjgg+bYsWPGGGMOHDhgJJkZM2Y4t5vbc5/benk9H/L6PF7vuO7evdtIMm+99Vaej+Ul1zvX/vGPfxhJZs2aNeb8+fPm+PHj5q677jKSzKeffmqOHTtmzp8/b4YMGWIkmU2bNjm33blzZxMZGWl+/fVXk5mZabZs2WJuueUW4+fnZ3bt2pXnDKNHjzYOh8O88cYb5tSpUyYtLc3MnDnTZSzlZ73cnrO8Phfjx483np6eZunSpSYtLc2sX7/eVKpUyXTs2NHluC5btswEBwebsWPHXvXY79q1y0gybdq0yddzVtjn1/XG//XOv9zGRWG9hhXFvvNi0aJFhnoCFJoERhMAwBIFLftpaWkmKCjI9O7d22W9n376yUhy+WX/j2U/MzPTPPzww+bzzz93eVx6eroJCAhw2V5aWprx9fU1Tz31lDHmf78op6enO9e5VGp++eWXPP8sxvz+B4cyZcqYTp06uSzPysoyU6dOver+4uPjjSSzZcuWq257woQJRpJJSUm56nbyY9CgQSY0NNRl2dq1a40k8/LLLxtj8v58bNmyxUgyy5Yty3VfN1r283M+3OjzePlxvdGyn5dz7VLZP3v2rHOdd99910gymzdvvuLnW7hwoXNZ586dzU033eSyz6SkJCPJDB8+PE8Z0tLSTEBAgLn99ttdtnP5H87yup4x1y7713suWrdubW6++WaXfTzxxBPGw8PDXLx40eTHunXrjCTzl7/8Jc+PKezzKy/j/3KXn3+Xj4uifA0rjH3nBWUfKFQJXMYPAHBLW7du1blz59SqVSuX5a1bt5aPj88Vl5hLv19m+sgjj6hixYoul+9Lv0/BlZaWpiZNmjiX+fv7q3Llytf8WICPj48kKTMzM1/5k5KSdPr06Ss+E+zp6amYmJirPu7SZ7Gvtb9L62RnZ+crU360atVKAQEBzmOT1+cjMjJSFStWVN++fRUbG6u9e/cWSp4bOR/+KC/PY2Ed14Kea1lZWVdkut7517RpU4WGhiopKSlPGX755RelpaWpc+fO19xuXtfLj9yeiwsXLlxxN//s7Gx5e3vL09MzX9sPCgqSJKWlpeX5MYV9ft3I+L/e+VeUr2FFtW8ARYuyDwBwS6dPn5b0v1/c/6hMmTI6e/bsFcufeeYZ7d69W7NmzdK2bdtcvnf+/HlJ0ksvvSSHw+H82rdvX75KQV6lpqY6sxbUp59+qo4dO6pChQry9fXN9TPcRcHX11fHjh2TlPfnw9/fXytWrFD79u01fvx4RUZGqnfv3kpPTy9Qlhs5H66nqI5rcZ9rl3h7ezsL3fUyHDx4UJJUoUKFa24zr+sV1D333KP169dr6dKlSk9P17p167RkyRLdd999+S77tWrVkp+fn3bt2pXnxxT2+ZWX8Z/f868wzysr9w2g8FD2AQBu6dIvybn9kn369GlVr179iuW9evXS8uXLVaZMGUVHR7u8Q3qprEyZMkXGGJev1atXF3r+qlWrSpKOHz9eoO3s379fDzzwgCpXrqw1a9bozJkzmjRpUmFEvKbMzEyX45yf56Nx48b65JNPdPjwYY0YMUKLFi3S66+/XqA8N3I+XEtRHtfiPtek368GOHnypGrWrJmnDH5+fpKkixcvXnO7eV2voGJjY/XnP/9Z/fr1U0hIiB588EH16tVLc+bMyfe2fH19deedd+r48eP6/vvvr7reyZMnNXDgQEmFf35db/zfyPlXWOeVlfsGULgo+wAAt9SkSRMFBQVp3bp1LsvXrFmjjIwM/elPf7riMZ06dVL58uU1e/ZsrV+/Xq+88orze5fubr5p06Yizy79/u5i2bJl9Z///KdA29m8ebMyMzP11FNPKTIyUn5+fsUy7dk333wjY4zatGkjKe/Px+HDh51XVVSoUEETJ05Uy5Ytr7jSIr9u5Hy4lqI8rsV9rknSf//7X+Xk5Khly5Z5ytCkSRN5eHho5cqV19xuXtcrqK1btyo5OVnHjh1TZmam9u/frzfffFNhYWE3tL3Y2Fj5+vrqueeeu+pVJVu2bHFOy1fY59f1xv+NnH+FdV5ZuW8AhYuyDwBwC2XLltXhw4e1d+9enT17Vp6enho2bJg+/PBDvf/++0pNTdXmzZv15JNPqkqVKho0aNBVt9W1a1f169dP48eP1/r16yX9/g5l//79tWDBAr355ptKTU1Vdna2Dh48qN9++63Qfx5fX1+NGjVK3377rYYMGaJDhw4pJydHZ8+ezVfxvfRO7VdffaULFy5o9+7d1/388I3IycnRqVOnlJWVpaSkJA0dOlQ1a9ZUv379JP1+/PLyfBw+fFiDBw/Wjh07lJGRoY0bN2rfvn3OPxrk5vLnPrfPFud1/3lVlMe1OM61jIwMnTlzRllZWdqwYYOGDBmi8PBwl+frWhkqVKig7t27a/HixZo3b55SU1OVlJSk2bNnu+wnr+sV1DPPPKOaNWvq3Llz11zv888/v+7Ue5LUvHlz/etf/9KWLVvUoUMHffbZZzpz5owyMzP166+/as6cORowYIDzs+qFfX5db/zn5fzL7TWxMM4rK/cNoJAV5+0AAQC4RPm8G/+GDRtMeHi48ff3N+3btzdHjhwxOTk5Ji4uztStW9d4e3ubsLAw88ADD5idO3c6H/fBBx+YsLAwI8nUqlXLpKSkmNTUVFOjRg0jyQQFBZn4+HhjjDEXL140I0aMMDVr1jReXl6mQoUKpnv37mbr1q1m5syZJiAgwEgydevWNcnJyWb27NkmJCTESDLh4eEu05rl1T//+U/TtGlT4+fnZ/z8/EyLFi3MzJkzzaRJk4y/v7/L/t5//33nz1K9enXnHflHjBhhypYta8qUKWN69uxp/vnPfxpJpnbt2uaZZ55xbqdGjRrmvffey3fGQYMGGW9vb1OtWjXj5eVlQkJCTLdu3UxycrLLenl5Pvbu3WvatWtnwsLCjKenp6lataoZPXq0ycrKMjNmzDCVK1c2kkxAQIDp2rVrrs/9Sy+9lOt6edl/fp7Hax3XoUOHmkqVKhlJJjAw0Dz44IP5OqbXOtemTp3qzFirVi2zatUq8+qrr5rQ0FAjyVSqVMn861//MgsXLnRmCAsLMwsWLDDGGDN//nzTqVMnU7FiRePl5WXKlStnHn74YbNv3748ZzDGmLNnz5qBAweacuXKmaCgINO+fXszZswY5/n3888/53m93J7b/DwXK1asMOXKlTOSnF/e3t6mYcOG5oMPPnD+TJ999pkJDg42r7zySp6eh/3795vhw4ebpk2bmqCgIOPp6WnKlCljWrRoYQYMGGC+//5757qFfX4Zc/Xxb8y1z7/9+/fn+ppYWK9hhb3vvOJu/EChSnAYc9mtTQEAKAYOh0OLFi1Sr169rI6C6xg8eLASExN14sQJq6OglHrzzTe1e/duTZkyxbksIyNDI0eO1JtvvqlTp07J39/fwoQoDAkJCXrooYeumHkBwA1J9LI6AQAAKPmKcho/4FqOHDmiIUOGXPF5cB8fH9WsWVOZmZnKzMyk7APAZfjMPgAAhWTHjh0u005d7at3797ktBmOadHx9/eXt7e35s2bp6NHjyozM1OHDx/W3LlzNWbMGPXu3VshISFWxwSAEod39gEAKCQNGjRwi8tP85Nz1KhRmj9/vjIyMhQREaG4uDj16NGjiBO6H3d57t1RaGio/vOf/2js2LGqV6+ezp8/r6CgIDVu3FivvvqqnnjiCasjAkCJRNkHAABXNWHCBE2YMMHqGCjlOnTooOXLl1sdAwDcCpfxAwAAAABgM5R9AAAAAABshrIPAAAAAIDNUPYBAAAAALAZyj4AAAAAADZD2QcAAAAAwGYo+wAAAAAA2AxlHwAAAAAAm6HsAwAAAABgM5R9AAAAAABshrIPAAAAAIDNUPYBAAAAALAZyj4AAAAAADbjZXUAAEDptXr1aqsjAABKCP5PAAqXwxhjrA4BACh9HA6H1REAACUQ9QQoFIm8sw8AsAS/zAG/69WrlyQpISHB4iQAADvhM/sAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANgMZR8AAAAAAJuh7AMAAAAAYDOUfQAAAAAAbIayDwAAAACAzVD2AQAAAACwGco+AAAAAAA2Q9kHAAAAAMBmKPsAAAAAANiMl9UBAAAASouVK1fqxx9/dFm2Y8cOSdKkSZNclrdp00ZRUVHFlg0AYC8OY4yxOgQAAEBpsHz5ct1xxx3y9vaWh0fuF1jm5OQoMzNT//nPf3T77bcXc0IAgE0kUvYBAACKSXZ2tipVqqQTJ05cc72wsDClpKTIy4uLMAEANySRz+wDAAAUE09PT/Xp00c+Pj5XXcfHx0ePPvooRR8AUCCUfQAAgGL08MMPKyMj46rfz8jI0MMPP1yMiQAAdsRl/AAAAMUsPDxc+/fvz/V71atX1/79++VwOIo5FQDARriMHwAAoLj17dtX3t7eVyz38fHRY489RtEHABQYZR8AAKCY9e3bV5mZmVcsz8jIUO/evS1IBACwG8o+AABAMWvYsKEaNmx4xfIGDRqoSZMmFiQCANgNZR8AAMAC0dHRLpfye3t767HHHrMwEQDATrhBHwAAgAX279+vWrVq6dKvYg6HQ3v27FGtWrWsDQYAsANu0AcAAGCFmjVrqlWrVvLw8JDD4VDr1q0p+gCAQkPZBwAAsEh0dLQ8PDzk6empRx991Oo4AAAb4TJ+AAAAixw7dkxVqlSRJB06dEiVKlWyOBEAwCYSvaxOAAAASraePXtq8eLFVsewvcqVK1sdwbZ69OihxMREq2MAQLGi7AMAgOtq06aNnn32Watj2NLKlSvlcDh02223WR3FlqZMmWJ1BACwBGUfAABcV/Xq1dWrVy+rY9jSXXfdJUkKCQmxOIk98Y4+gNKKsg8AAGAhSj4AoChwN34AAAAAAGyGsg8AAAAAgM1Q9gEAAAAAsBnKPgAAAAAANkPZBwAAAADAZij7AAAAAADYDGUfAAAAAACboewDAAAAAGAzlH0AAAAAAGyGsg8AAAAAgM1Q9gEAAAAAsBnKPgAAAAAANkPZBwAAAADAZij7AAAABfD666+rYsWKcjgcmjVrltVx8uWDDz5QZGSkHA6Hy5ePj48qVqyojh07Ki4uTqdOnbI6KgAgnyj7AAAABTB8+HD98MMPVse4Id27d9eePXtUu3ZthYaGyhijnJwcpaSkKCEhQRERERoxYoQaN26sdevWWR0XAJAPlH0AAFDipaenq127dm6/D3fgcDhUpkwZdezYUfPnz1dCQoKOHj2qe++9V2fOnLE6HgAgjyj7AACgxJs3b55SUlLcfh/uqEePHurXr59SUlLc7mMKAFCaUfYBAEChM8Zo8uTJatiwoXx9fRUWFqZu3bppx44dznWGDBkiHx8fVa5c2bns6aefVmBgoBwOh44fPy5JGjp0qIYNG6bk5GQ5HA7VqVNH06dPl59CllguAAAJoElEQVSfnypWrKjBgwerSpUq8vPzU7t27bRmzZpC2UdBrVq1So0aNVJoaKj8/PzUtGlTffnll5KkgQMHOj8fX7t2bW3cuFGS1L9/fwUEBCg0NFQff/yxJCk7O1tjxoxRzZo15e/vr2bNmmnRokWSpNdee00BAQEKDg5WSkqKhg0bpmrVqv2/9u4vNqpqi+P4b+i0nU6Zw7QyFqV/0hZMU8BEBVIRE/UNica2Q2yUB9CYtkajiX8aAZWgRJuqPAjVECOJmAAOGEQUYsToE5dg1JQUaRtMW7U21oEybWlpoes+eO/Ekf9l+m/4fpJ56N777LV2el5Wztlnq6mpSfv375fjOFq/fv01r2XFihWSpH379kXbLpVXfX290tPT5fV69dlnn2nJkiVyHEfZ2dnatm1bzNzfffedFi5cKK/XK8dxNG/ePEUikcvGAABchgEAAFxCMBi0YDB4Vde88sorlpKSYlu3brXu7m5raGiw22+/3aZPn26dnZ3RcY8++qhlZWXFXFtXV2eSrKurK9pWXl5uhYWFMeMqKystPT3djh49agMDA9bY2GgLFiwwn89n7e3tcYlxpVpaWkySvffee9G2UChka9eutRMnTlg4HLaSkhK74YYbYuIlJSXZ77//HjPXI488Ynv27In+/fzzz1tqaqrt3LnTTp48aatWrbIpU6bY4cOHzcxs9erVJsmeeeYZe/fdd62srMx+/vln27t3r/l8Plu3bt1l8y8sLLRp06ZdtD8SiZgky8nJueq8Dhw4YKdOnbI///zT7r77bktPT7fBwUEzM+vt7TXHcay2ttb6+/uts7PTysrKov+Xy8W4EiO5fwEgAXzCk30AABBX/f39euedd1RWVqbly5dr2rRpmjdvnt5//3399ddf2rx5c9xiud3u6NsDxcXFqq+vV09Pj7Zs2RK3GCMVDAb16quvKiMjQ5mZmXrwwQcVDofV1dUlSaqurta5c+dico1EIjp8+LDuv/9+SdLAwIDq6+tVWlqq8vJy+f1+rVmzRsnJyeet8c0339RTTz2lXbt2qaioSEuXLlUkEtHLL798zWvx+XxyuVzq6em56rwWLVokx3EUCARUUVGhvr4+tbe3S5JaW1sViUQ0Z84ceTweZWVladeuXZo+ffpVxQAAnI9iHwAAxFVjY6N6e3s1f/78mPYFCxYoJSUl5jX7eJs/f768Xm/MdoGJIjk5WdLfr6ZL0n333adbbrlFH374ocxMkrR9+3ZVVFQoKSlJktTU1KTTp09r7ty50XnS0tI0Y8aMMV1jX1+fzEyO41xTXikpKZKkoaEhSVJBQYFuvPFGLV++XGvXrlVra2t07ERZOwBMVhT7AAAgrrq7uyVJU6dOPa/P7/dHnw6PltTU1OjT8/H0xRdf6J577lEgEFBqaqpefPHFmH6Xy6Wqqir98ssvOnDggCTpo48+0uOPPx4d09fXJ0las2ZNdI+/y+VSW1ubTp8+PWZraW5uliQVFRXFNa+0tDR98803Wrx4sdavX6+CggJVVFSov79/wqwdACYrin0AABBXfr9fki5Y1Hd3dys7O3vUYg8NDY16jCvR3t6u0tJSzZgxQ4cOHdKpU6dUW1t73rgVK1bI4/Hogw8+UFNTkxzHUV5eXrQ/EAhIkjZs2CAzi/kdPHhwzNazf/9+SdKSJUvintecOXP0+eefq6OjQzU1NdqxY4feeuutCbN2AJis3OOdAAAASCxz587V1KlT9f3338e0Hzp0SIODg7rjjjuibW63O/pKdzx8++23MjOVlJSMWowrceTIEQ0NDenJJ59UQUGBpL+f5P9bRkaGHn74YW3fvl0+n09PPPFETH9OTo48Ho9++umnMcn7Qjo7O7VhwwZlZ2frsccei2teHR0d6u7uVnFxsQKBgN544w199dVXOnr06IRYOwBMZjzZBwAAceXxePTcc8/p008/1ccff6xIJKIjR46ourpaN910kyorK6NjZ82apRMnTmj37t0aGhpSV1eX2trazpszMzNTHR0dam1tVU9PT7R4Hx4e1smTJ3X27Fk1NDTo2WefVW5ubvSouHjEGInc3FxJ0tdff62BgQG1tLRc9FsF1dXVOnPmjPbu3asHHnggps/j8WjlypXatm2b6uvrFYlEdO7cOf3222/6448/LpnDvn37ruroPTNTb2+vhoeHZWbq6urSjh07dNdddykpKUm7d++O7tm/lrz+qaOjQ1VVVTp27JgGBwf1448/qq2tTSUlJXGLAQDXrTE/AAAAAEwqIzm6bHh42Orq6mz27NmWnJxsGRkZVlpaak1NTTHjwuGw3XvvvebxeCw/P9+efvppe+GFF0ySzZo1K3qE3g8//GB5eXmWlpZmixcvts7OTqusrLTk5GSbOXOmud1ucxzHHnroITt+/HjcYlyJt99+27KyskySpaenW1lZmZmZ1dTUWGZmpvn9flu2bJlt3LjRJFlhYWHM0YBmZrfddpu99NJLF5z/zJkzVlNTY7m5ueZ2uy0QCFh5ebk1NjZabW2tpaWlRY/F27p1a/S6L7/80nw+n73++usXzX3Pnj126623mtfrtZSUFJsyZYpJMpfLZX6/3xYuXGjr1q2zcDh8VXlt2rTJvF6vSbLZs2fb8ePHbfPmzeY4jkmyvLw8a25uttbWVlu0aJFlZGRYUlKS3XzzzbZ69Wo7e/bsZWNcKY7eA3Cd+sRl9r/PvwIAAFzAsmXLJEmhUGicM4lVVVWlUCikcDg83qlcs6VLl2rjxo3Kz88f71QSzkS9fwFglIV4jR8AAExa/z/GbrL55xaBhoYGeTweCn0AQFxR7AMAAPzLsWPHYo57u9ivoqJiRPPX1NSopaVFzc3NWrlypV577bU4rwAAcL3ja/wAAGDSWbVqlbZs2aLBwUHl5+errq5OwWAwbvMXFRVpNHc6er1eFRUVaebMmdq0aZOKi4tHLRYA4PrEnn0AAHBJ7HnGZMb9C+A6xZ59AAAAAAASDcU+AAAAAAAJhmIfAAAAAIAEQ7EPAAAAAECCodgHAAAAACDBUOwDAAAAAJBgKPYBAAAAAEgwFPsAAAAAACQYin0AAAAAABIMxT4AAAAAAAmGYh8AAAAAgARDsQ8AAAAAQIKh2AcAAAAAIMG4xzsBAAAw8e3cuVMul2u80wBGJBgMjncKADDmXGZm450EAACYuA4ePKhff/11vNMARiwnJ0d33nnneKcBAGMpRLEPAAAAAEBiCbFnHwAAAACABEOxDwAAAABAgqHYBwAAAAAgwbglhcY7CQAAAAAAEDf/+S8DAvvh7Wo8qgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<IPython.core.display.Image object>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 90
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jljyZqfnZXQd",
        "outputId": "78bc032d-a548-4c78-f016-135793073f9e"
      },
      "source": [
        "# Check which layers of our model are trainable or not\n",
        "for layer in model_5.layers:\n",
        "  print(layer, layer.trainable)"
      ],
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7f1bda21d190> True\n",
            "<tensorflow.python.keras.layers.preprocessing.text_vectorization.TextVectorization object at 0x7f1af50a2e90> True\n",
            "<tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7f1bda1c9710> True\n",
            "<tensorflow.python.keras.layers.embeddings.Embedding object at 0x7f1af509bb50> True\n",
            "<tensorflow_hub.keras_layer.KerasLayer object at 0x7f1be2685c10> False\n",
            "<tensorflow.python.keras.layers.wrappers.Bidirectional object at 0x7f1bda15bb50> True\n",
            "<tensorflow.python.keras.layers.merge.Concatenate object at 0x7f1a9dd21d90> True\n",
            "<tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7f1bda1adb10> True\n",
            "<tensorflow.python.keras.engine.input_layer.InputLayer object at 0x7f1a9deca110> True\n",
            "<tensorflow.python.keras.layers.core.Dense object at 0x7f1a9dd2af50> True\n",
            "<tensorflow.python.keras.layers.core.Dense object at 0x7f1af43f9110> True\n",
            "<tensorflow.python.keras.layers.core.Dense object at 0x7f1a9dd22c10> True\n",
            "<tensorflow.python.keras.layers.core.Dropout object at 0x7f1a9dd35910> True\n",
            "<tensorflow.python.keras.layers.merge.Concatenate object at 0x7f1a9dd355d0> True\n",
            "<tensorflow.python.keras.layers.core.Dense object at 0x7f1a9dd35c90> True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0C4fPi0mZXNA"
      },
      "source": [
        "# Compile token, char, positional embedding model\n",
        "model_5.compile(loss=tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.2), # add label smoothing (examples which are really confident get smoothed a little)\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])"
      ],
      "execution_count": 92,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0wzuMn0yaO1N"
      },
      "source": [
        "### Create tribid embedding datasets and fit tribid model\n",
        "\n",
        "Again, to keep our experiments swift, let's fit on 20,000 examples for 3 epochs.\n",
        "\n",
        "This time our model requires four feature inputs:\n",
        "\n",
        "1. Train line numbers one-hot tensor (`train_line_numbers_one_hot`)\n",
        "2. Train total lines one-hot tensor (`train_total_lines_one_hot`)\n",
        "3. Token-level sequences tensor (`train_sentences`)\n",
        "4. Char-level sequences tensor (`train_chars`)\n",
        "\n",
        "We can pass these as tuples to our `tf.data.Dataset.from_tensor_slices()` method to create appropriately shaped and batched `PrefetchedDataset`'s."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lDXhCx8faoMq",
        "outputId": "12867070-442c-4d03-cd9e-b33405dfeade"
      },
      "source": [
        "# Create training and validation datasets (all four kinds of inputs)\n",
        "train_pos_char_token_data = tf.data.Dataset.from_tensor_slices((train_line_numbers_one_hot, # line numbers\n",
        "                                                                train_total_lines_one_hot, # total lines\n",
        "                                                                train_sentences, # train tokens\n",
        "                                                                train_chars)) # train chars\n",
        "train_pos_char_token_labels = tf.data.Dataset.from_tensor_slices(train_labels_one_hot) # train labels\n",
        "train_pos_char_token_dataset = tf.data.Dataset.zip((train_pos_char_token_data, train_pos_char_token_labels)) # combine data and labels\n",
        "train_pos_char_token_dataset = train_pos_char_token_dataset.batch(32).prefetch(tf.data.AUTOTUNE) # turn into batches and prefetch appropriately\n",
        "\n",
        "# Validation dataset\n",
        "val_pos_char_token_data = tf.data.Dataset.from_tensor_slices((val_line_numbers_one_hot,\n",
        "                                                              val_total_lines_one_hot,\n",
        "                                                              val_sentences,\n",
        "                                                              val_chars))\n",
        "val_pos_char_token_labels = tf.data.Dataset.from_tensor_slices(val_labels_one_hot)\n",
        "val_pos_char_token_dataset = tf.data.Dataset.zip((val_pos_char_token_data, val_pos_char_token_labels))\n",
        "val_pos_char_token_dataset = val_pos_char_token_dataset.batch(32).prefetch(tf.data.AUTOTUNE) # turn into batches and prefetch appropriately\n",
        "\n",
        "# Check input shapes\n",
        "train_pos_char_token_dataset, val_pos_char_token_dataset"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<PrefetchDataset shapes: (((None, 15), (None, 20), (None,), (None,)), (None, 5)), types: ((tf.float32, tf.float32, tf.string, tf.string), tf.float64)>,\n",
              " <PrefetchDataset shapes: (((None, 15), (None, 20), (None,), (None,)), (None, 5)), types: ((tf.float32, tf.float32, tf.string, tf.string), tf.float64)>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 93
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lZtnZxolaqoy",
        "outputId": "b88dd746-9cbb-46fd-dbb2-bd5bf0cc27b4"
      },
      "source": [
        "# Fit the token, char and positional embedding model\n",
        "history_model_5 = model_5.fit(train_pos_char_token_dataset,\n",
        "                              steps_per_epoch=int(0.1 * len(train_pos_char_token_dataset)),\n",
        "                              epochs=3,\n",
        "                              validation_data=val_pos_char_token_dataset,\n",
        "                              validation_steps=int(0.1 * len(val_pos_char_token_dataset)))"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/3\n",
            "562/562 [==============================] - 37s 54ms/step - loss: 1.1022 - accuracy: 0.7267 - val_loss: 0.9890 - val_accuracy: 0.7995\n",
            "Epoch 2/3\n",
            "562/562 [==============================] - 28s 49ms/step - loss: 0.9773 - accuracy: 0.8127 - val_loss: 0.9584 - val_accuracy: 0.8201\n",
            "Epoch 3/3\n",
            "562/562 [==============================] - 26s 45ms/step - loss: 0.9620 - accuracy: 0.8177 - val_loss: 0.9495 - val_accuracy: 0.8285\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vnNdrZNOatD5",
        "outputId": "439c6743-3d18-4a5b-88b5-a227cc8182c9"
      },
      "source": [
        "# Make predictions with token-char-positional hybrid model\n",
        "model_5_pred_probs = model_5.predict(val_pos_char_token_dataset, verbose=1)\n",
        "model_5_pred_probs"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 23s 22ms/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.51367944, 0.11580934, 0.01004686, 0.34096786, 0.01949645],\n",
              "       [0.5387638 , 0.08762626, 0.0396519 , 0.32496384, 0.00899414],\n",
              "       [0.28190422, 0.11497217, 0.12618461, 0.4056243 , 0.0713147 ],\n",
              "       ...,\n",
              "       [0.03845271, 0.09928644, 0.05113291, 0.0329084 , 0.7782196 ],\n",
              "       [0.02728448, 0.34787413, 0.08743301, 0.02114793, 0.51626045],\n",
              "       [0.15178181, 0.5739851 , 0.15927956, 0.03313894, 0.08181465]],\n",
              "      dtype=float32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hIH-bJ5kavQV",
        "outputId": "78c2cf7b-c0bc-4f4b-f6b9-2c964f608b31"
      },
      "source": [
        "# Turn prediction probabilities into prediction classes\n",
        "model_5_preds = tf.argmax(model_5_pred_probs, axis=1)\n",
        "model_5_preds"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(30212,), dtype=int64, numpy=array([0, 0, 3, ..., 4, 4, 1])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6xUvADnXaxFk",
        "outputId": "6de8fec5-4f71-409a-c73c-8667bfd19852"
      },
      "source": [
        "# Calculate results of token-char-positional hybrid model\n",
        "model_5_results = calculate_results(y_true=val_labels_encoded,\n",
        "                                    y_pred=model_5_preds)\n",
        "model_5_results"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 82.91738382099828,\n",
              " 'f1': 0.828093215496095,\n",
              " 'precision': 0.8274794671143357,\n",
              " 'recall': 0.8291738382099828}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 97
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Buf0fTkfazeX"
      },
      "source": [
        "## Compare model results"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "id": "lv8zu_1DdLtp",
        "outputId": "f47ae241-7f30-484a-e241-0de18a6939e0"
      },
      "source": [
        "# Combine model results into a DataFrame\n",
        "all_model_results = pd.DataFrame({\"baseline\": baseline_results,\n",
        "                                  \"custom_token_embed_conv1d\": model_1_results,\n",
        "                                  \"pretrained_token_embed\": model_2_results,\n",
        "                                  \"custom_char_embed_conv1d\": model_3_results,\n",
        "                                  \"hybrid_char_token_embed\": model_4_results,\n",
        "                                  \"tribrid_pos_char_token_embed\": model_5_results})\n",
        "all_model_results = all_model_results.transpose()\n",
        "all_model_results"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>accuracy</th>\n",
              "      <th>precision</th>\n",
              "      <th>recall</th>\n",
              "      <th>f1</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>baseline</th>\n",
              "      <td>72.183238</td>\n",
              "      <td>0.718647</td>\n",
              "      <td>0.721832</td>\n",
              "      <td>0.698925</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>custom_token_embed_conv1d</th>\n",
              "      <td>78.548259</td>\n",
              "      <td>0.782573</td>\n",
              "      <td>0.785483</td>\n",
              "      <td>0.782641</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>pretrained_token_embed</th>\n",
              "      <td>71.666887</td>\n",
              "      <td>0.717348</td>\n",
              "      <td>0.716669</td>\n",
              "      <td>0.713684</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>custom_char_embed_conv1d</th>\n",
              "      <td>62.673772</td>\n",
              "      <td>0.622177</td>\n",
              "      <td>0.626738</td>\n",
              "      <td>0.601498</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>hybrid_char_token_embed</th>\n",
              "      <td>73.924269</td>\n",
              "      <td>0.739345</td>\n",
              "      <td>0.739243</td>\n",
              "      <td>0.736495</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>tribrid_pos_char_token_embed</th>\n",
              "      <td>82.917384</td>\n",
              "      <td>0.827479</td>\n",
              "      <td>0.829174</td>\n",
              "      <td>0.828093</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                               accuracy  precision    recall        f1\n",
              "baseline                      72.183238   0.718647  0.721832  0.698925\n",
              "custom_token_embed_conv1d     78.548259   0.782573  0.785483  0.782641\n",
              "pretrained_token_embed        71.666887   0.717348  0.716669  0.713684\n",
              "custom_char_embed_conv1d      62.673772   0.622177  0.626738  0.601498\n",
              "hybrid_char_token_embed       73.924269   0.739345  0.739243  0.736495\n",
              "tribrid_pos_char_token_embed  82.917384   0.827479  0.829174  0.828093"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 103
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        },
        "id": "3Dpy8mGodNmf",
        "outputId": "0c668440-f445-4ac5-fc57-634878cc5967"
      },
      "source": [
        "# Reduce the accuracy to same scale as other metrics\n",
        "all_model_results[\"accuracy\"] = all_model_results[\"accuracy\"]/100\n",
        "\n",
        "# Plot and compare all of the model results\n",
        "all_model_results.plot(kind=\"bar\", figsize=(10, 7)).legend(bbox_to_anchor=(1.0, 1.0));"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAIqCAYAAAAHAtOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZxU1Z3///e7WUQUiUuDKCKgNE0jIIiMazQuiY5xNxF11GQyYWI0yZhFzWTGGLNqYjJfE/ML7k6iMca4YCQxMVGcGU2kQQHZFJWgxgUVASUILZ/fH3XbFNgN1U1V30PV6/l49KP73jpd/en7qMetd517zzmOCAEAAAApqcu7AAAAAGBDhFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByuuf1h3faaacYPHhwXn8eAACgZDNmzHg1IurzrqOW5BZSBw8erObm5rz+PAAAQMls/yXvGmoNl/sBAACQHEIqAAAAkkNIBQAAQHJyuycVAABgSzZjxox+3bt3v1bSXqLjr6PWSXqipaXlX/bZZ59X2mpASAUAAOiE7t27X7vzzjuPqK+vX1ZXVxd517MlWbdunZcuXdr00ksvXSvpuLbakPoBAAA6Z6/6+voVBNSOq6uri/r6+uUq9EK33aYL6wEAAKgmdQTUzsuOXbtZlJAKAACA5HBPKgAAQBkMvujefcr5fIu/c8yMcj7f5li7dq169OjRpX+TnlQAAIAt2BFHHLHHyJEjR+y5554jv/e97+0kSbfffvt2TU1NI4YPH960//77N0jS8uXL60455ZTBDQ0NTQ0NDU033njj+ySpd+/eY1uf64Ybbtj+5JNPHixJJ5988uDTTz990OjRoxvPOeecgQ888EDvvffeu3HEiBFNY8eObZw1a9ZWktTS0qJJkyYNHDZs2MiGhoamb37zm/2mTJnS54gjjtij9XnvvPPO7Y488sg91AH0pAIAAGzBbr755sX9+/d/58033/TYsWObTj311DfOO++8wQ8++OCCxsbGNS+//HI3SbrooosGbLfddu88+eST8yRp6dKl3Tb13C+++GLPmTNnLujevbtef/31uunTpy/o0aOH7rrrrj4XXHDBwPvuu+/pK664on7JkiU9582bN7dHjx56+eWXu9XX17/zuc99btBf//rX7rvsskvL9ddfv+PHP/7xVzvyfxFSAQAAtmCXXXZZ/3vvvfd9kvTSSy/1uPLKK+snTJiwsrGxcY0k9e/f/x1Jeuihh7a79dZbn2n9vfr6+nc29dwnnXTSsu7dC3Hx9ddf73bqqacOWbx4cS/bsXbtWkvSH//4x+0+9alPLW29HaD17330ox997Zprrtnh3HPPfW3mzJnb3nHHHc925P8ipAIAAGyhfv3rX/eZNm1an+bm5gV9+vRZN2HChOFjx45dtXDhwl6lPoftd3/+29/+5uLHtt1223WtP1944YW7HnLIISt///vfP71w4cKehx122PCNPe8555zz2jHHHLNnr1694thjj13W0XtauScVAABgC/XGG29069u37zt9+vRZ99hjj/WaNWvWNqtXr6579NFH+yxYsKCnJLVe7j/kkENW/OAHP+jX+rutl/t33HHHtTNnzuz1zjvv6O67796+vb+1YsWKbgMHDlwjSZMnT96pdf/hhx++YvLkyTutXbtWxX9v8ODBa/v377/2iiuuGDBp0qQOXeqXCKkAAABbrJNPPnl5S0uLhw4dOvJLX/rSrmPGjHmrX79+LVdeeeXiE088cc/hw4c3nXjiiUMl6dvf/vaLb7zxRrdhw4aNHD58eNPUqVP7SNLXvva1F44//vg9x40b19i/f/+17f2tCy+88KVLLrlk4IgRI5paWlre3X/++ecvHThw4JrGxsaRw4cPb7ruuut2aH1s4sSJrw0YMGDNuHHjVnf0f3NEPnPQjh8/Ppqbm3P52wAAAB1he0ZEjC/eN2vWrMVjxozpcA9hLTnrrLMGjR07dtX555/f5nGaNWvWTmPGjBnc1mPckwoAAN41+KJ7O9R+8XeO6VD7UTeNKrntbd9u2XSjIiMWzO9Qe1TWyJEjR2y99dbrJk+e/Fxnfp+QCgAAOu+Svh1rP2RQZepAcubOnbtZnxq4JxUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAPCuhx56qPfHPvax3dp7fPHixT2OOuqooZWug9H9AAAA5XBJ333K+3zLZ5TjaVpaWtS9e+mR7/3vf/+q97///avae3zw4MFrf/vb3z5Tjto2hp5UAACALdTChQt7DhkyZORxxx03ZOjQoSOPOuqooStXrqzbddddR51zzjm7NjU1jbj++uu3v+OOO7bbe++9G5uamkYcffTRQ5cvX14nSdOmTes9duzYxuHDhzeNGjVqxLJly+p+/etf9/nABz6wpyTde++92zY2NjY1NjY2jRgxomnZsmV1Cxcu7Dls2LCRkrRq1SqfcsopgxsaGppGjBjRdM899/SRpCuvvHLHD37wg3scfPDBw3bfffe9PvWpTw3s6P9WUki1fZTthbYX2b6ojccH2X7A9mO2Z9v+x44WAgAAgI5bvHhxr/POO++VZ555Zm6fPn3Wffe7362XpB133LFl3rx584899tiV3/rWtwY89NBDT86bN2/+uHHjVn3961/vv3r1ap9xxhl7/Nd//deShQsXzps2bdrCbbfddl3xc19xxRU7X3nllX9ZsGDBvD/96U8LNnz8sssu62dbTz755LxbbrnlmUmTJg1etWqVJWnevHm977rrrmfmz58/d8qUKdsvWrSoR0f+r02GVNvdJF0l6WhJTZJOs920QbP/kHRbRIyVNFHSjztSBAAAADpn5513XvPBD37wLUk688wzX3v44Ye3laSzzjprmSQ9+OCD2zz99NO9JkyY0NjY2Nh066237rhkyZKes2fP7tWvX7+1hxxyyCpJ2mGHHdb16LF+jtxvv/3e/OIXv7jbN77xjX6vvvpqtw0ff/jhh7c988wzX5OksWPHrt5ll13WzJkzp5ckHXTQQSt23HHHd3r37h177rnn6qeffnqrjvxfpdygMEHSooh4RpJs3yrpeEnzitqEpO2yn/tK+mtHigAAAEDn2G5zu0+fPuskKSJ00EEHrbjnnnueLW736KOPbr2p5/7Wt7710gknnLD87rvv7nvwwQc33nvvvU/17t173aZ+T5J69uwZrT9369Yt1q5d642131Apl/t3lVS85urz2b5il0j6J9vPS5oq6TMdKQIAAACd8+KLL/a8//77t5Gkm2++eYcDDjjgzeLHDz300Leam5u3feKJJ7aSpBUrVtTNnj17q9GjR69+5ZVXekybNq23JC1btqxu7dq16z333Llzt5owYcLfvvnNb740evTot5544olexY8feOCBb/7sZz/bQZJmz5691Ysvvthz9OjRq8vxf5Vr4NRpkm6MiIGS/lHST22/57ltT7LdbLt56dKlZfrTAAAAtWvw4MGrf/jDH/YbOnToyDfeeKP7F7/4xfVC1i677NIyefLkxRMnThza0NDQNH78+MY5c+b06tWrV9x8881Pf/aznx00fPjwpkMPPbRh1apV6+W3yy+/vN+wYcNGNjQ0NPXo0SNOOeWU5cWPX3DBBa+sW7fODQ0NTaeeeuoekydPXrz11luHysARG38e2/tLuiQiPpRtf1mSIuLbRW3mSjoqIp7Ltp+RtF9EvNLe844fPz6am5s3/z8AAABlM/iiezvUfnGv0zvUftSQQSW3ve3bLR167hEL5neofUfYnhER44v3zZo1a/GYMWNerdgfLcHChQt7fvjDHx721FNPzc2zjs6aNWvWTmPGjBnc1mOl9KROlzTM9hDbPVUYGDVlgzZLJB0uSbZHSOolia5SAAAAdMomQ2pEtEg6T9J9kuarMIp/ru1LbR+XNfuCpE/aniXp55I+FpvqogUAAMBmGT58+JottRd1U0pafiAipqowIKp438VFP8+TdGB5SwMAAECtYsUpAAAAJKf0hVyBjejwjfbfOaZD7UfdNKrktnPOntOh5wYAAOkhpCIfl/TtWPsOjAad3ziiQ09dydGgAACgc7jcDwAAgHddeeWVO5511lmDJOnzn//8LhdffHH/POqgJxUAAKAMRt00ap9yPt+cs+fM6Ej7devWKSLUrVu3cpaRG3pSAQAAtlALFy7sOXjw4L1OPPHEwQ0NDSMvuOCCAXvttdeIhoaGpvPPP3+X1nY/+tGPdmxoaGgaPnx40wknnDBEkm655Za+o0ePbhwxYkTTAQcc0PDcc88l1XmZVDEAAADomCVLlmx13XXXPbt8+fLXf/nLX24/e/bs+RGhI444Ys/f/OY329bX17d873vfG/DII48sGDBgQMvLL7/cTZKOPPLINydOnLigrq5O3//+93e69NJLd77mmmuez/v/aUVIBQAA2IINGDBgzeGHH/7WpEmTBj700EPbNTU1NUnSqlWr6hYsWNBr5syZdccee+yyAQMGtEhS//7935GkZ599tucJJ5wwcOnSpT3WrFlTt9tuu72d5/+xIS73AwAAbMF69+69TpIiQv/2b//24oIFC+YtWLBg3pIlS544//zzX23v984777xBn/70p1958skn5/3oRz/6y9tvv51ULkyqGAAAAHTO0UcfveKnP/3pTsuXL6+TpGeffbbHCy+80P1DH/rQinvuuWf7l156qZsktV7uX7lyZbdBgwatlaQbb7xxx/wqbxuX+wEAAKrASSedtGLu3Lm99t1330ap0MN68803Pzt+/PjVX/jCF148+OCDG+vq6mKvvfZa9atf/WrxV77ylb+edtppe/Tt27floIMOWrlkyZKt8v4fijkicvnD48ePj+bm5lz+NsqvwytO9Tq9Q+1HdWAy/9u+3dKh52YyfwD4O87nbbM9IyLGF++bNWvW4jFjxrR7OR2bNmvWrJ3GjBkzuK3HuNwPAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAbKG+8Y1v9Bs6dOjID33oQ3vsvffejT179hx38cUX98+7rnJgMn8AAIAymN84Yp9yPt+IBfNnbKrNddddV3///fc/2atXr1i0aFHP22+/ffty1pAnelIBAAC2QKeffvqg559/fqujjz562LXXXrvDIYccsqpHjx75rNJUAfSkAgCSldLqR3POntOh5wYq7ZZbblkybdq0vtOmTXtywIABHVueawtATyoAAACSQ08qAAAlmN84okPtK7mOPFAL6EkFAABAcuhJBQAA2MItWbKk+7777tv01ltvdbMdkydP7j9//vwndthhh3V519ZZhFQAAIAyKGXKqHJ74YUX3h3R9/LLL8/u6r9fSVzuBwAAQHIIqQAAAEhOVV7u7/C8et85pkPtR900quS2zKsHAADQcVUZUjvskr4da9+ByZ8BAEDVWrdu3TrX1dVVzSpPXWndunWW1O7ALkIqsIViJR4AyN0TS5cubaqvr19OUO2YdevWeenSpX0lPdFeG0IqAABAJ7S0tPzLSy+9dO1LL720lxjn01HrJD3R0tLyL+01IKQC2GysxAOgFu2zzz6vSDou7zqqFakfAAAAySkppNo+yvZC24tsX9TG4z+w/Xj29aTtN8pfKgAAAGrFJi/32+4m6SpJR0p6XtJ021MiYl5rm4g4v6j9ZySNrUCtAAAAqBGl9KROkLQoIp6JiDWSbpV0/Ebanybp5+UoDgAAALWplJC6q6Tnirafz/a9h+3dJQ2R9Md2Hp9ku9l289KlSztaKwAAAGpEuUf3T5R0e0S809aDEXG1pKslafz48TUxnxijngEAADqulJ7UFyTtVrQ9MNvXloniUj8AAAA2UykhdbqkYbaH2O6pQhCdsmEj242Stpf0SHlLBAAAQK3ZZEiNiBZJ50m6T9J8SbdFxFzbl9ounsB2oqRbI6ImLuMDAACgckq6JzUipkqausG+izfYvqR8ZQEAAKCWseIUAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEhO97wLAIAtxeCL7u1Q+8XfOaZD7UfdNKrktnPOntOh5waALQ0hFQAq5ZK+HWs/ZFBl6gCALRCX+wEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkpKaTaPsr2QtuLbF/UTpuP2p5ne67tW8pbJgAAAGrJJqegst1N0lWSjpT0vKTptqdExLyiNsMkfVnSgRGxzHa/ShUMAACA6ldKT+oESYsi4pmIWCPpVknHb9Dmk5KuiohlkhQRr5S3TAAAANSSUkLqrpKeK9p+PttXrEFSg+3/s/0n20eVq0AAAADUnnKtONVd0jBJh0oaKOkh26Mi4o3iRrYnSZokSYMGsbIKAHTW/MYRHWo/YsH8ClUCAJVRSk/qC5J2K9oemO0r9rykKRGxNiKelfSkCqF1PRFxdUSMj4jx9fX1na0ZAAAAVa6UkDpd0jDbQ2z3lDRR0pQN2tylQi+qbO+kwuX/Z8pYJwAAAGrIJkNqRLRIOk/SfZLmS7otIubavtT2cVmz+yS9ZnuepAckfSkiXqtU0QAAAKhuJd2TGhFTJU3dYN/FRT+HpM9nXwAAAMBmYcUpAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkJySQqrto2wvtL3I9kVtPP4x20ttP559/Uv5SwUAAECt6L6pBra7SbpK0pGSnpc03faUiJi3QdNfRMR5FagRAAAANaaUntQJkhZFxDMRsUbSrZKOr2xZAAAAqGWlhNRdJT1XtP18tm9DJ9uebft227uVpToAAADUpHINnLpH0uCIGC3p95JuaquR7Um2m203L126tEx/GgAAANWmlJD6gqTintGB2b53RcRrEfF2tnmtpH3aeqKIuDoixkfE+Pr6+s7UCwAAgBpQSkidLmmY7SG2e0qaKGlKcQPbA4o2j5M0v3wlAgAAoNZscnR/RLTYPk/SfZK6Sbo+IubavlRSc0RMkfRZ28dJapH0uqSPVbBmAAAAVLlNhlRJioipkqZusO/iop+/LOnL5S0NAAAAtYoVpwAAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEBySgqpto+yvdD2ItsXbaTdybbD9vjylQgAAIBas8mQarubpKskHS2pSdJptpvaaNdH0uck/bncRQIAAKC2lNKTOkHSooh4JiLWSLpV0vFttPu6pMskrS5jfQAAAKhBpYTUXSU9V7T9fLbvXbbHSdotIu4tY20AAACoUZs9cMp2naTvS/pCCW0n2W623bx06dLN/dMAAACoUqWE1Bck7Va0PTDb16qPpL0kPWh7saT9JE1pa/BURFwdEeMjYnx9fX3nqwYAAEBVKyWkTpc0zPYQ2z0lTZQ0pfXBiFgeETtFxOCIGCzpT5KOi4jmilQMAACAqrfJkBoRLZLOk3SfpPmSbouIubYvtX1cpQsEAABA7eleSqOImCpp6gb7Lm6n7aGbXxYAAABqGStOAQAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOSUFFJtH2V7oe1Fti9q4/FP2Z5j+3Hb/2u7qfylAgAAoFZsMqTa7ibpKklHS2qSdFobIfSWiBgVEXtLulzS98teKQAAAGpGKT2pEyQtiohnImKNpFslHV/cICJWFG1uIynKVyIAAABqTfcS2uwq6bmi7ecl/cOGjWyfK+nzknpKOqytJ7I9SdIkSRo0aFBHawUAAECNKNvAqYi4KiL2kHShpP9op83VETE+IsbX19eX608DAACgypQSUl+QtFvR9sBsX3tulXTC5hQFAACA2lZKSJ0uaZjtIbZ7SpooaUpxA9vDijaPkfRU+UoEAABArdnkPakR0WL7PEn3Seom6fqImGv7UknNETFF0nm2j5C0VtIySWdXsmgAAABUt1IGTikipkqausG+i4t+/lyZ6wIAAEANY8UpAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5JQUUm0fZXuh7UW2L2rj8c/bnmd7tu0/2N69/KUCAACgVmwypNruJukqSUdLapJ0mu2mDZo9Jml8RIyWdLuky8tdKAAAAGpHKT2pEyQtiohnImKNpFslHV/cICIeiIhV2eafJA0sb5kAAACoJaWE1F0lPVe0/Xy2rz2fkPSbzSkKAAAAta17OZ/M9j9JGi/pkHYenyRpkiQNGjSonH8aAAAAVaSUntQXJO1WtD0w27ce20dI+oqk4yLi7baeKCKujojxETG+vr6+M/UCAACgBpQSUqdLGmZ7iO2ekiZKmlLcwPZYSZNVCKivlL9MAAAA1JJNhtSIaJF0nqT7JM2XdFtEzLV9qe3jsmbflbStpF/aftz2lHaeDgAAANikku5JjYipkqZusO/iop+PKHNdAAAAqGGsOAUAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSU1JItX2U7YW2F9m+qI3H3297pu0W26eUv0wAAADUkk2GVNvdJF0l6WhJTZJOs920QbMlkj4m6ZZyFwgAAIDa072ENhMkLYqIZyTJ9q2Sjpc0r7VBRCzOHltXgRoBAABQY0q53L+rpOeKtp/P9gEAAAAV0aUDp2xPst1su3np0qVd+acBAACwBSklpL4gabei7YHZvg6LiKsjYnxEjK+vr+/MUwAAAKAGlBJSp0saZnuI7Z6SJkqaUtmyAAAAUMs2GVIjokXSeZLukzRf0m0RMdf2pbaPkyTb+9p+XtJHJE22PbeSRQMAAKC6lTK6XxExVdLUDfZdXPTzdBVuAwAAAAA2GytOAQAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOQQUgEAAJAcQioAAACSQ0gFAABAcgipAAAASA4hFQAAAMkhpAIAACA5hFQAAAAkh5AKAACA5BBSAQAAkBxCKgAAAJJDSAUAAEByCKkAAABIDiEVAAAAySGkAgAAIDmEVAAAACSHkAoAAIDkEFIBAACQHEIqAAAAkkNIBQAAQHIIqQAAAEgOIRUAAADJIaQCAAAgOYRUAAAAJIeQCgAAgOSUFFJtH2V7oe1Fti9q4/GtbP8ie/zPtgeXu1AAAADUjk2GVNvdJF0l6WhJTZJOs920QbNPSFoWEXtK+oGky8pdKAAAAGpHKT2pEyQtiohnImKNpFslHb9Bm+Ml3ZT9fLukw227fGUCAACglnQvoc2ukp4r2n5e0j+01yYiWmwvl7SjpFeLG9meJGlStvmm7YWdKbrcOp6mn9hJG/xv7dmwy3nTxdRGtueYdz2OedfjmHc9jnnXq6FjvnslnxzvVUpILZuIuFrS1V35NyvBdnNEjM+7jlrCMe96HPOuxzHvehzzrscxR6lKudz/gqTdirYHZvvabGO7u6S+kl4rR4EAAACoPaWE1OmShtkeYrunpImSpmzQZoqks7OfT5H0x4iI8pUJAACAWrLJy/3ZPabnSbpPUjdJ10fEXNuXSmqOiCmSrpP0U9uLJL2uQpCtZlv8LQtbII551+OYdz2OedfjmHc9jjlKYkAeiF0AACAASURBVDo8AQAAkBpWnAIAAEByCKkAAABIDiEVAAAAySGkAgAAIDldOpn/lsz2QZKGRcQNtuslbRsRz+ZdVzWy/fmNPR4R3++qWmoFx7zr2Z4jqd2RqxExugvLqQm2T9rY4xFxR1fVUgs4r2BzEVJLYPurksZLGi7pBkk9JP1M0oF51lXF+mTfh0vaV3+fl/dYSY/mUlH145h3vQ9n38/Nvv80+35GDrXUimOz7/0kHSDpj9n2ByQ9LImQWl6cV7BZmIKqBLYflzRW0syIGJvtm01PR2XZfkjSMRGxMtvuI+neiHh/vpVVL45517P9WOt5pWjfzIgYl1dN1c727ySdHREvZtsDJN0YER/Kt7LqxHkFncU9qaVZk62gFZJke5uc66kV/SWtKdpek+1D5XDMu55tH1i0cYA4N1fabq0BNfOypEF5FVMDOK+gU7jcX5rbbE+W9D7bn5T0z5KuybmmWvDfkh61fWe2fYKkG/Mrpya0dcxvyrGeWvAJSdfb7pttv6HCOQaV8wfb90n6ebZ9qqT7c6yn2nFeQadwub9Eto+U9EFJlnRfRPw+55Jqgu1xkg7ONh+KiMfyrKcWcMzz0RpSI2J53rXUAtsnSmq93PxQRNy5sfbYPJxX0BmEVGxRbG8bEW/mXUc1YyaL/Nn+eETckHcd1cz27iq8zu+33VtSt9Z7JlF+nFfQGdz3VALbJ9l+yvZy2ytsr7S9Iu+6atS8vAuoZtlMFhdK+nK2q3UmC3Str+VdQDXLbtu6XdLkbNeuku7Kr6LqxnkFncU9qaW5XNKxETE/70JqwUbm1rOkbbuylhp0orKZLCQpIv6ajcRFmdme3d5DYlBJpZ0raYKkP0tSRDxlu1++JVU1zivoFEJqaV4moHapb0n6rqSWNh6j97+y1kRE2GYmi8rrL+lDkpZtsN8qzNmJynk7ItbYliTZ7q6NLKyAzcZ5BZ1CSC1Ns+1fqHA56O3WnaxOUjEzJd0VETM2fMD2v+RQTy1hJouu82sV7st7fMMHbD/Y9eXUlGm2/13S1tmg2E9LuifnmqoZ5xV0CgOnSmC7rQEMERFME1MBtodLei0iXm3jsf4R8XIOZdWMopksJOl3zGSBamO7ToWpv96dsUXStcEbYsVwXkFnEFKRLNvjImJm3nXUGts7q3C/XkiaHhEv5VxSVbN9paRbI4JL/F3Idk9JjSq8zhdGxJpN/Ao2A+cVdAYhdSNsXxARl9v+odq4XykiPptDWTXD9gOSdlZhFO4vIuKJnEuqetntFBersKa5JR0i6dKIuD7XwqqY7bNVmEx+uKQ7VQiszflWVd1sHyPpJ5KeVuF1PkTSv0bEb3ItrEpxXkFnEVI3wvaxEXFP9ibyHhHBihkVln36/qgKb+LbqRBWv5FvVdXL9kJJB0TEa9n2jpIejojh+VZW/WzvIOlkSRMlDYqIYTmXVLVsL5D04YhYlG3vocJa8o35VladOK+gsxg4tRERcU/2nTCak+yS0JVZr+oFKnwaJ6RWzmuSiic0X5ntQ+XtqcLl590lMZtIZa1sDaiZZ7T+6x7lxXkFnUJI3Qjb92gj05JExHFdWE7NsT1ChR7Uk1U4of1C0hdyLapKFc1Nu0jSn23frcJr/3hJ7c3niTKwfbkK80g+rcJr/OsR8Ua+VVUn2ydlPzbbnirpNhVe5x+RND23wqoU5xVsLkLqxn0v7wJq3PWSbpX0oYj4a97FVLnWibWfzr5a3Z1DLbXmaUn7tzWbBcru2KKfX1bh3khJWipp664vp+pxXsFm4Z7UEtneWoX7xBbmXQuA6mJ7VxUu87/bcRARD+VXEQDkj57UEtg+VoVe1Z6ShtjeW4WRiVzuryDbB0q6RH9/87YK89MOzbOuamZ7vKSv6L2BaXRuRVU5299RYbDUPEnvZLtDEiG1QmwPkfQZSYO1/uucc3oFcF5BZ9GTWgLbMyQdJunBiBib7ZsTEaPyray6ZSNwz5c0Q39/81brCFGUXzYK90uS5kha17o/Iv6SW1FVLjvmoyPi7U02RlnYniXpOr33dT4tt6KqGOcVdBY9qaVZGxHLW9d5zpDuK2858xZ2uaURMSXvImrMM5J6qGjJZVTc6oi4Mu8iagjnFXQKIbU0c22fLqmb7WGSPiuJ1WEq7wHb35V0h4rewFmFqqK+avtaSX/Q+sf8jvxKqnqrJD1ue8NjzmIhlfP/bH9V0u/EuaUrcF5BpxBSS/MZFe6neVvSz1VY5/nruVZUG/4h+z6+aF+ocOsFKuPjKszV2UN/vywXKnxQQGVMyb7QdUZJOlOFc0nx65xzS2VwXkGncE9qB9nuJmmbiFiRdy1AudleyCowXS9bR74h21wYEWvzrKfa2V4kqSki1uRdSy3gvILOqsu7gC2B7Vtsb2d7GxVu/J5n+0t511XtbPe1/X3bzdnXFbb75l1XlXvYdlPeRdQS24dKekrSVZJ+LOlJ2+/Ptajq94Sk9+VdRA3hvIJOoSe1BLYfj4i9bZ8haZykiyTNYPqMyrL9KxXeTFqXpT1T0piIOKn938LmsD1f0h6SnlXh9pbWab94rVdINnvI6a1zMNtukPTziNgn38qql+0HJY1WYZWp4nskmYKqAjivoLO4J7U0PWz3kHSCpB9FxFrbpPvK2yMiTi7a/prtx3OrpjYclXcBNahH8SIhEfFkdr5B5Xw17wJqDOcVdAqX+0szWdJiSdtIesj27pK4J7Xy/mb7oNaNbHL/v+VYT9XL5i3cTdJh2c+rxHmi0pptX2v70OzrGknNeRdVzbL5UBer8AFhmgo9qozsrxDOK+gsLvd3ku3uEdGSdx3VLFvZ6yZJrfehLpP0sYiYlV9V1S2blme8pOER0WB7F0m/jIgDcy6tatneStK5klo/kP2PpB8zuX/l2P6kpEmSdoiIPbKpBX8SEYfnXFpV4ryCziKklsj2MZJGSurVui8iLs2votpheztJYkaFystupxgraWbR6mqzuXescrIBmasj4p1su5ukrSJiVb6VVa/sdT5B0p9ZRbDyOK+gs+huL4Htn0g6VYX5Ui3pIyqsQYwKsv0t2++LiBURscL29ra/kXddVW5NFD65hvRugEJl/UHS1kXbW0u6P6daasXbxdNP2e4uVhGsJM4r6BRCamkOiIizJC2LiK9J2l9/n9MQlXN0RLzRuhERyyT9Y4711ILbbE+W9L7skuj9kq7JuaZq1ysi3mzdyH7unWM9tWCa7X+XtLXtIyX9UtI9OddUzTivoFMY3V+a1sE6q7J7aV6TNCDHempFN9tbtd6bZ3trSVvlXFNVi4jvZW/aKyQNl3RxRPw+57Kq3Vu2x7UuyWl7HzFAsNIukvQJFea9/ldJUyVdm2tFVYzzCjqLe1JLYPs/Jf1QhSXzrsp2XxsR/5lfVdXP9oWSjpV0Q7br45KmRMTl+VVV22w/EhH7511HNbG9r6RbJf1VhduJdpZ0akTMyLWwGmb7VxtMf4cK4ryC9hBSS5D14J0j6WAV7qn5H0n/X0SszrWwGmD7KElHZJu/j4j78qyn1tl+rHXgA8onmxe1ddnI9ZZFtX0kvU5di9d51+J4oz2E1BLYvk3SSkk/y3adLqlvRHw0v6rAp++uZ3tmRIzLu45awjHvehzzrsXxRnu4J7U0e0VE8brDD9iel1s1aNVr002ALZ7zLgAA8sDo/tLMtL1f64btfxArwqSAywBdj8DU9Xiddz1e512L44020ZO6EbbnqPAG0UPSw7aXZNu7S1qQZ21ATs7MuwCgC1yYdwE1hvMK2kRI3bgP510ANopP32Vm+yRJl0nqp8LxtaSIiNZVv57IsbxatTjvAqqN7QMlXaJCh0N3/f11PlSFH36XX3XVh/MKOouBU9hi2d6Lk1t52V4k6diImJ93LdUue+NuV0Tc0VW11BrbCySdL2mGpHda90fEa7kVVcU4r6Cz6ElFcmyv1Ebuw+PTd0W9zBtJlzk2+95P0gGS/phtf0DSw5IIqZWzPCJ+k3cRNYTzCjqFkIrkREQfSbL9dUkvSvqpCpeHzhArfVVas+1fSLpL0tutO+nVK7+I+Lgk2f6dpKaIeDHbHiDpxhxLqwUP2P6uCh8Eil/nM/MrqapxXkGncLkfybI9KyLGbGofysf2DW3sjoj45y4vpkbYnh8RI4q26yTNLd6H8rL9QBu7IyIO6/JiagDnFXQWPalI2Vu2z1BhyciQdJqkt/Itqbq19u6hS/3B9n2Sfp5tnyrp/hzrqXoR8YG8a6glnFfQWcyTipSdLumjkl7Ovj6S7UOF2G6w/QfbT2Tbo23/R951VbOIOE/STySNyb6ujojP5FtVdbPd3/Z1tn+TbTfZ/kTedVUrzivoLC73A3iX7WmSviRpcuta2rafiIi98q2sutneXdKwiLjfdm9J3SJiZd51VassnN4g6SsRMcZ2d0mPRcSonEurSpxX0Fn0pCJZfPrORe+IeHSDfS25VFIjbH9S0u2SJme7dlVhgAkqZ6eIuE3SOkmKiBYVTUWFsuO8gk4hpCJl10j6sqS1khQRsyVNzLWi6veq7T2UTQFm+xQVZlhA5Zwr6UBJKyQpIp5SYVoqVM5btnfU31/n+0lanm9JVY3zCjqFgVNIWe+IeNReb2EpPn1X1rmSrpbUaPsFSc+qMPUXKuftiFjT+jrPLj1zH1ZlfV7SFEl72P4/SfWSTsm3pKrGeQWdQkhFyvj03fW2j4gjbG8jqS4iVtr+sKS/5F1YFZtm+98lbW37SEmflnRPzjVVu2WSDpE0XIU5mBdK2jvXiqob5xV0CgOnkCzbQ1X49H2ACm8qz0o6IyI4sVWI7ZmSzmpdzcv2REnnR8Q/5FtZ9crmRf2EpA+qEJjuk3RtcHKuGNszJB0XES9k2++XdBUDpyqD8wo6i5CK5BV/+s67lmqXfTC4XYWpvg6WdJakD0cE9+tVkO2ekhpVuGqwMCLW5FxSVbO9r6Qfq7A07ThJ31bhdf5croVVKc4r6CxCKpKVDWz4qqSDVHjz/l9Jl0bEa7kWVuVsN6gwunyJpBMj4m85l1TVbB+jwjypT6vQkzpE0r+ytnxl2d5fhRkVVks6JiKW5lxSVeO8gs4gpCJZtn8v6SFJP8t2nSHp0Ig4Ir+qqpPtOVp/sE4/FUY7vy1JETE6j7pqge0FKvQqLcq295B0b0Q05ltZ9bF9j9Z/nTepcJ/7MkmKiOPyqKtacV7B5iKkIlltTfZsew73jZVfNpl8u7gPuHJsT4+IfYu2LenR4n0oD9uHbOzxiJjWVbXUAs4r2FyM7kfKfpfdYH9btn2KCoNKUGbFbxa2x6hw35gk/U9EzMqnqupm+6Tsx2bbU1V4nYcKy/9Oz62wKlYcQm33l9T6QeDRiHgln6qqF+cVbC56UpEc2ytVeLO2pG2UrQqjwuITb0bEdnnVVu1sf07SJyXdke06UYW15H+YX1XVyfYNG3s8Ij7eVbXUGtsflfRdSQ+qcJ45WNKXIuL2POuqVpxX0FmEVADvsj1b0v4R8Va2vY2kR7h3DNXE9ixJR7b2ntqul3R/RIzJt7LqxHkFncXlfiTN9mhJg1X0Wo2IO9r9BWwua/01zN/J9qFCbA+R9Bm993XOIJ7Kqdvg8v5rYpnwSuK8gk4hpCJZtq+XNFrSXP39kn/o75eMUH43SPqz7Tuz7RMkXZ9jPbXgLknXqbDK1LpNtEV5/Nb2fZJ+nm2fKokpvyqH8wo6hcv9SJbteRHRlHcdtcb2OBXmppUKAxwey7Oeamf7z6y80/WygWvFr/M7N9Yem4fzCjqDkIpk2b5O0hURMS/vWmqF7Z9GxJmb2ofysX26pGGSfqds/khJioiZuRVV5WxfFhEXbmofyoPzCjqLy/1I2X9LesT2Syq8eVtScLN9RY0s3rDdTdI+OdVSK0ZJOlPSYVr/tpbDcquo+h0pacNAenQb+1AenFfQKYRUpOw6Fd6854h79SrK9pcl/bukrW2vaN0taY2kq3MrrDZ8RNLQiFiTdyHVzvY5kj4taWg24rxVH0n/l09V1YvzCjYXl/uRLNuPRMT+eddRS2x/OyK+vJHHR0bE3K6sqdrZvkvSJCaTrzzbfSVtL+nbki4qemhlRLxe1G77iFjW1fVVK84r6CxCKpJl+8eS3qfCqOfie/UY3Z8T2zMjYlzedVQT2w+qMIvFdK3/OmcKqpzwOu9aHG+0h8v9SNnWKrxpf7BoH1NQ5Yu5Dcvvq3kXgPfgdd61ON5oEyEVyWJZyCRx6aXMImKa7d0lDYuI+233ltQt77pqHK/zrsXxRptYYQPJst1g+w+2n8i2R9v+j7zrAsrJ9icl3S5pcrZrVxUm+AeAmkZIRcqukfRlSWslKSJmS5qYa0VgBHr5nSvpQEkrJCkinpLUL9eKwOXnMnHBbptoxnkFbeJyP1LWOyIetdd7v2jJq5hqlq0G067WieUjYr+uqaimvB0Ra1pf57a7i8ufFZPN0Tk3Iho30uzwrqqn2kVE2J6qwnzA7bXhvII2EVKRsldt76HsDdv2KZJezLekqnVF9r2XpPGSZqnQmzRaUrMkpgKrnGm2W+eSPFKFeTzvybmmqhUR79heaHtQRCxpp83rbe1Hp820vW9ETM+7EGxZmIIKybI9VIUJnw+QtEzSs5LOiIi/5FpYFbN9h6SvRsScbHsvSZdExCn5Vla9bNdJ+oQKs1hY0n2Srg1OzhVj+yFJYyU9Kumt1v1M+1UZthdI2lPSX1Q43qweiJIQUpE829tIqouIlRvsPzsibsqprKpke25EbLiE4Xv2oevY/lVEnJx3HdXE9iFt7Y+IaV1dSy3IZq94DzocsCmEVGyxmAC6/Gz/XIWejp9lu86QtG1EnJZfVbXN9mMRMTbvOoDNZbufCrcUSZLau90CaMXofmzJGIFbfh+XNFfS57Kvedk+5IeehDKzvZ/t6bbftL3G9jtFa8ujzGwfZ/spFW7ZmiZpsaTf5FoUtggMnMKWjDfvMouI1bZ/ImlqRCzMux6gQn6kwnR2v1RhoOBZkhpyrai6fV3SfpLuj4ixtj8g6Z9yrglbAHpSsSWjJ7XMbB8n6XFJv82297Y9Jd+qah6v8wqIiEWSukXEOxFxg6Sj8q6piq2NiNck1dmui4gHVPhwAGwUPanYkv1f3gVUoa9KmiDpQUmKiMdtD8m1oiqWzdn53xFxxkaaXdhV9dSQVbZ7Snrc9uUqTG1Hp03lvGF7W0kPSbrZ9isqmlUBaA8Dp5As21tJOlnSYBV9oIqIS/OqqdrZ/lNE7Fc8WMf2bKaKqRzb/yvpsIhg1Z0uko02f1lST0nnS+or6cdZ7yrKLJuhZbUKVwXOUOF435z1rgLtoicVKbtb0nJJMyS9nXMttWKu7dMldbM9TNJnJT2cc03V7hlJ/5fdVlE8Z+f38yupuhVNfbRa0tfyrKUWRERxrynTBqJkhFSkbGBEcJ9Y1/qMpK+o8KHg5ypMLP/1XCuqfk9nX3WS+uRcS02wfaCkSyTtrvWv0gzNq6ZqZvskSZdJ6qdCb2rrZP7b5VoYksflfiTL9tWSfti6+hEAlEO2AtL5Klylead1P5efK8P2IknHRsT8vGvBloWeVKTsIEkfs/2sCj17LKVXYbYbJH1R770P+LC8aqp2tuslXSBppNaf6JxjXjnLI4J5OrvOywRUdAY9qUgWS+l1PduzJP1E7+1hmpFbUVXO9u8k/UKFDwefknS2pKURwaj+MrPdukLdRyV1k3SHiu53j4iZedRVrbLL/JJ0iKSdJd2l9Y/3HXnUhS0HIRVJs32QpGERcUPW47RtRDybd13VyvaMiNgn7zpqSesxL55Fwfb0iNg379qqje0HNvJw0HtdXrZv2MjDERH/3GXFYIvE5X4ky/ZXVZjwebikGyT1UGFN+QPzrKvK3WP705Lu1Po9Hq/nV1LVW5t9f9H2MZL+KmmHHOupWhHxgbxrqCURwZLK2CxMXoyUnSjpOGXT8kTEX8Xo50o7W9KXVJh2akb21ZxrRdXvG7b7SvqCCpf8r1VhUA8qxPa3bL+vaHt729/Is6ZqZvumNo739XnWhC0Dl/uRLNuPRsQE2zMjYlw2IfQjDJwCsDmKF6so2jczIsa19zvovHaO93v2ARvicj9SdpvtyZLeZ/uTkv5ZhV4mlJntwyLij0UDHdbDAIfKye61/qTeO6MC9+tVTjfbW0XE25Jke2tJW+VcUzWrs719RCyTJNs7iPyBEvAiQcqukHSEpBUq3Jd6sQprP6P8DpH0R0nHtvFYqDAKGpVxt6T/kXS/imZUQEXdLOkPRQN7Pi5WQqqkKyQ9YvuX2fZHJH0zx3qwheByP5Jl+/ri3iTb20q6OyIOz7EsoKxsPx4Re+ddR62xfZQKH4Il6fcRcV+e9VQ7202SWmdP+GNEzCt67N1eVqAYIRXJsv11STtGxKdtby/pXknXRMTGpjXBZspGmG84sfyl+VVU3bIBOw9HxNS8a0GB7UciYv+866gV3A+M9hBSkTTbl0vaTtI+kr4TEb/KuaSqZvsnknpL+oAK9/+eIunRiPhEroVVIdsrVbiVwpK2UWHKr7ViXfPcMaina3G80R5CKpKzweAdS/pPSY9K+q3EIJ5Kap1Qvuj7tpJ+ExEH510b0FXo2etaHG+0h4FTSNGGg3ceU2Ei/2PFIJ5KW519X2V7F0mvSRqQYz1Vz/aJKtyjtzzbfp+kQyPirnwrA4B8EVKRHFYpydU9WUj6rqSZKnwouCbfkqreVyPiztaNiHgjW22NkJof511AjeF4o02sOIVk2R5o+07br2Rfv7I9MO+6qpXtOkl/iIg3snt/d5fUGBEX51xatWvrPEwHQr7OzLuAamJ7D9tbZT8favuzxStQSWLGFrSJkIqU3SBpiqRdsq97sn2ogIhYJ+mqou23Wy9Bo6KabX8/eyPfw/b3VViOFmVme6XtFe19tbaLiCfyrLMK/UrSO7b3lHS1pN0k3dL6YES8nldhSBshFSmrj4gbIqIl+7pRUn3eRVW5P9g+2TaX37rOZyStkfQLSbeqcF/wublWVKUiok82a8L/k3SRpF0lDZR0oaT/yrO2KrcuIloknSjphxHxJXGvO0rA6H4ky/b/396dx8pV1mEc/z63llBLa0o0MRGKgKxiK4tpa1ER4xZRFAPIJlAibgEUFwSNGBMEZDFYBEKAyiYgCSpi0EQ0yiZYC1gFaoCCiZG4AKFWBYHHP84ZOpTbFsLMvOeceT4Joee9JXkyTOb+zjvv73duoNo5vaJeOgA4PMP8h6ceizQdeIqqWMo4pMIkLbZ9VOkcXSLpLttzN7QWgyHpNqqbgC8D77e9UtIfbO9UOFo0XHZSo8kWAfsBDwN/pZrZeVjJQF1X7zRN2N7I9sy+nacoZ2HpAB20WtJBkqZImpB0ELC6dKgOOxxYAJxUF6hbApcWzhQtkJ3UaCxJC23fvKG1GBxJN6y9Uz3ZWoxOZkgOnqTXUn3lv5BqgsXNwGdsP1guVbdJ2gjYtr5cYft/JfNEO6SDNJpsMbD2L+fJ1uIlkrQx1ZOmXlk/grZ3JnUm1bm9iM6oi9G9S+cYF5L2AC4GHqT6bNlc0qG2f10yVzRfitRoHEkLgDcDr5J0bN+PZgJTyqTqvI8Dn6GaorCsb/1x4OwiiaInTWwDIumLtr8paTHVDupz2D66QKxxcAbwLtsrACRtS9VrsGvRVNF4KVKjiTYCNqF6f87oW3+c6lxqDJjts4CzJB1le3HpPPEcZ5UO0CH31P9eWjTF+JnaK1ABbP9J0tSSgaIdciY1GkvSFrYfWs/P0/U8YJKmA58FZts+UtI2wHa2ryscrbMk7UbV9bwF1Y1Zb6LCnKLBOkrSFOBU258vnWVcSLoIeAa4rF46CJhie1G5VNEGKVKjtdJQMniSrqIaJP9R2ztJejlwi+03Fo7WWZJWAF8AllP9IgdgfTdo8dJIutX2gtI5xkX9tKlPA7vXSzcC59h+olyqaIN83R8R/ba2vb+kAwBs/zuD/Yfu77avLR1izNwp6VrgavpGT9m+plyk7rL9hKSzgRuobsRW2H6ycKxogRSpEdHvSUnTqJtKJG0NZLdjuE6UdAHVL/BnX+sUTEO1MfBPYM++NQN5zYdA0vuA84D7qY6zbCnp47avL5ssmi5FarRZdvgG70Tgp1QjYi6nmiN5WNFE3Xc4sD0wlTVf96dgGiLbh5fOMGbOAN5u+z549ub3J0CK1FivFKnRZul6HiBJE8AsYB9gPtVNwDG2/1E0WPe9yfZ2pUOME0lbUX1+zKe6IbiVapj/yqLBumtVr0CtPQCsKhUm2iONU9FY6XoePUlLbe9WOsc4kbQEOM323aWzjAtJvwG+QzWrE+AjwFG255VL1V2SzqX6HP8+1U3BvsCfgZ9DjrbEuqVIjcZK1/PoSToF+AdwFc9tKHmkWKiOk3QPsDWwkupMam7GhkzS79d+fSXdZXtuqUxdVt+IrYsziirWJUVqNJakm2zvvuG/GYMiaSWTP4lnqwJxxoKkLSZbz83Y4EnatP7jccCjwJVU7/f9gVm2jy+VbZxJOt72yaVzRPOkSI3GkvQO4ADS9TwydWf/p6jmGZpqnuF5tv9TNFjHSZoLvKW+vNH2XSXzdFXfTdhkTZfOzVgZmXkd65LGqWiydD2P3sVUj5/9dn19YL22X7FEHSfpGOBjrHlfXybp/DyedvBsb1k6Q0wqk1piUtlJjcaStCJdz6Ml6W7bO25oLQZH0u+BBbZX19fTgVtzJnV4JP0OuBD4nu3HSucZd9lJjXWZKB0gYj1ukZTiaLSWSZrfu5A0D1haMM84EPB03/XTZGdp2PYHXgMslXSlpHfnyWpF5bWPSWUnNRorXc+jV7/m21GNhwGYDawAniKv/VBIOhY4FPhBvfRB4GLb3yqXajzUs4H3As6lujlYApyVaRajJekE298onSOaJ0VqNFa6nkdv4bt9/QAABORJREFUXa95T1774ZC0C1WzGlSNU3eUzDMOJM0BFgHvBX4GXE71/+AQ228sma0rJC1mkmkhPbaPHmGcaKE0TkVj2X4oXc+jlSJ09CRdavsQYNkkazEE9ZnUx4ALgONs96aH3CZpYblkndM7KrQQ2JFq/jJUw/zz8IrYoOykRmNN0vX8ISBdz9EpazeNSJoCLE+z2vDUZ913Zs3T7ACw/fVioTqsfsLX7rafqq+nUm06zF//fxnjLjup0WRHAPP6up5PpXrGdorUaD1JxwMnANMkPd5bBp4Ezi8WbDycSbWTuoy+GcwxNLOAmUDvrO8m9VrEeqVIjSZL13N0Vv2EnZMlnZwnHY3cZrbfUzrEGDkFuEPSL6k+w98KfK1oomiFFKnRZEuozoj1dz1fVDBPxDBcJ2m67dWSDgZ2oeowz/ng4blF0htsLy8dZBzYXiLpemBevXSc7YdLZop2yJnUaLR0PUfX1cP85wJzgO9SNfPsZ/ttJXN1kaTlVN3mLwO2AR4g4+2GRtL2tu+tP8efx/ayydYjelKkRmNN1uGcrufoml7jlKSvAn+xfWGewDMcGbE2WvXjfY+sv+Zfm23vOfJQ0Sr5uj+a7PX9F3XX866FskQMy6q6iepg4K31gPmphTN1UorQ0aoL1AngK7ZvLp0n2iePRY3GkXS8pFXAHEmP1/+sAv4G/KhwvIhB25/qK+cj6nN6mwGnlY0UMRi2nwHOLp0j2ilf90djpes5IqL9JJ1ONT7wGqfoiBchRWo0Vv3klzvT9RxdVn9L0Psg3ojqq/5/2X5FuVQRg1O/x6dTjRH8D2sa1WYWDRaNl6/7o8nOBf5dPxr1c8D9wCVlI0UMlu0ZtmfWv7CnAR+meu9HdEL9Hp+wPbV+r89IgRovRHZSo7HS9RzjStIdtncunSNiUCTtQzVO0FTjBH9YOFK0QLr7o8nS9RydV//y7pkAdgP+WyhOxMBJOgd4HXBFvfQJSe+0/emCsaIFspMajSXp1cCBwG9t3yhpNrCH7XzlH50haUnf5VPAg8D5tv9eJlHEYEm6F9ih1zRVbzj80fYOZZNF02UnNRqrHsdzZt/1n8mZ1OieCeAY248BSJoFnAEsKpoqYnDuA2YDvabXzeu1iPVKkRqNla7nGBNzegUqgO1HJeU8arSepB9TfYbPAO6RdHt9PQ+4vWS2aIcUqdFYtmf0/ixJwN7A/HKJIoZiQtIs248CSNqUfDZHN5xeOkC0W86kRquk6zm6RtJHgROAq+ulfYGTbF9aLlVERHkpUqOx1tH1/DbbCwpFihgKSTsCe9aXv7B9d8k8EYMg6Sbbu691dAsyzD9eoBSp0Vjpeo6IiBhfOfcUTZau54iIFpM0hWrc1Pals0T75LGo0WTP63oGch41IqIlbD8NrKjnXEe8KNlJjSZL13NERPvNAv5Yj6Ba3Vu0/YFykaIN8gs/muwM4FZJz+l6LpgnIiJevI2BvfquBZxaKEu0SIrUaCzbl0haypqu533S9RwR0Tovs/2r/gVJ00qFifZId39EREQMnKRPAp8CtgLu7/vRDOBm2wcXCRatkSI1IiIiBk7SK6jOo54MfKnvR6tsP1ImVbRJitSIiIiIaJyMoIqIiIiIxkmRGhERERGNkyI1IiIiIhonRWpERERENE6K1IiIiIhonP8D7S5LjdskSFcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        },
        "id": "xgVN3aaVdrkR",
        "outputId": "b2e6fca8-8570-4deb-d4f0-9e3866a7ef62"
      },
      "source": [
        "# Sort model results by f1-score\n",
        "all_model_results.sort_values(\"f1\", ascending=False)[\"f1\"].plot(kind=\"bar\", figsize=(10, 7));"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlMAAAIqCAYAAAAAbM/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3debgkZX3+//fNAIoIinHMwi5BDFEUHHFBo6ImGATiDuJuJImifjVfI5h8wWASt2h+xmAUFTQaRdyHiEFFRaNEGBBFwIkjoKJGR0QhGhnBz++PqiPNMEtLdXedU/1+Xddcc6q6mLlte+rcp+p5nkpVIUmSpFtmi74DSJIkLWWWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSepgy77+4jve8Y6122679fXXS5Ikje3888//QVUt39BrvZWp3XbbjVWrVvX110uSJI0tyTc29pq3+SRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHWzZd4Bp2O2Yj/Qd4Ra74hUH9x1BkiT9CrwyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSepgrDKV5KAkq5OsSXLMBl7fJcmnknwxyZeT/OHko0qSJC0+my1TSZYBJwKPAPYGjkiy93qH/RVwWlXtCxwOvGHSQSVJkhajca5M7Q+sqarLqmodcCpw2HrHFLB9+/XtgO9MLqIkSdLiNU6Z2hH41sj2le2+US8FnpTkSuAM4Lkb+oOSHJVkVZJVa9euvQVxJUmSFpdJDUA/AnhbVe0E/CHwjiQ3+7Or6qSqWlFVK5YvXz6hv1qSJKk/45SpbwM7j2zv1O4b9UzgNICqOge4NXDHSQSUJElazMYpU+cBeybZPcnWNAPMV653zDeBhwIk+R2aMuV9PEmSNHibLVNVdT1wNHAmcCnNrL2Lk5yQ5ND2sD8HnpXkS8C7gadVVU0rtCRJ0mKx5TgHVdUZNAPLR/cdN/L1JcABk40mSZK0+LkCuiRJUgeWKUmSpA7Gus0nbc5ux3yk7wi32BWvOLjvCJKkJcwrU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdbBl3wEk3TK7HfORviPcYle84uC+I0jSxHhlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDV0CXpDG56rykDfHKlCRJUgdjlakkByVZnWRNkmM28Po/JLmw/fVfSX40+aiSJEmLz2Zv8yVZBpwIPBy4EjgvycqqumThmKp6wcjxzwX2nUJWSZKkRWecK1P7A2uq6rKqWgecChy2ieOPAN49iXCSJEmL3ThlakfgWyPbV7b7bibJrsDuwCe7R5MkSVr8Jj0A/XDgfVV1w4ZeTHJUklVJVq1du3bCf7UkSdLsjbM0wreBnUe2d2r3bcjhwHM29gdV1UnASQArVqyoMTNKkuaUy1FoKRjnytR5wJ5Jdk+yNU1hWrn+QUnuCuwAnDPZiJIkSYvXZstUVV0PHA2cCVwKnFZVFyc5IcmhI4ceDpxaVV5xkiRJc2OsFdCr6gzgjPX2Hbfe9ksnF0uSJGlpcAV0SZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjoYq0wlOSjJ6iRrkhyzkWMen+SSJBcneddkY0qSJC1OW27ugCTLgBOBhwNXAuclWVlVl4wcsydwLHBAVV2d5E7TCixJkrSYjHNlan9gTVVdVlXrgFOBw9Y75lnAiVV1NUBVfX+yMSVJkhanzV6ZAnYEvjWyfSVwn/WOuQtAks8By4CXVtW/TyShJEmamd2O+UjfEW6xK15xcC9/7zhlatw/Z0/gwcBOwGeS3L2qfjR6UJKjgKMAdtlllwn91ZIkSf0Z5zbft4GdR7Z3aveNuhJYWVU/r6rLgf+iKVc3UVUnVdWKqlqxfPnyW5pZkiRp0RinTJ0H7Jlk9yRbA4cDK9c75kM0V6VIckea236XTTCnJEnSorTZMlVV1wNHA2cClwKnVdXFSU5Icmh72JnAVUkuAT4FvKiqrppWaEmSpMVirDFTVXUGcMZ6+44b+bqAF7a/JEmS5oYroEuSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUwVhlKslBSVYnWZPkmA28/rQka5Nc2P7648lHlSRJWny23NwBSZYBJwIPB64EzkuysqouWe/Q91TV0VPIKEmStGiNc2Vqf2BNVV1WVeuAU4HDphtLkiRpaRinTO0IfGtk+8p23/oek+TLSd6XZOeJpJMkSVrkJjUA/XRgt6raB/g48PYNHZTkqCSrkqxau3bthP5qSZKk/oxTpr4NjF5p2qnd90tVdVVVXdduvgW414b+oKo6qapWVNWK5cuX35K8kiRJi8o4Zeo8YM8kuyfZGjgcWDl6QJLfHNk8FLh0chElSZIWr83O5quq65McDZwJLANOrqqLk5wArKqqlcDzkhwKXA/8EHjaFDNLkiQtGpstUwBVdQZwxnr7jhv5+ljg2MlGkyRJWvxcAV2SJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDsYqU0kOSrI6yZokx2ziuMckqSQrJhdRkiRp8dpsmUqyDDgReASwN3BEkr03cNx2wPOBL0w6pCRJ0mI1zpWp/YE1VXVZVa0DTgUO28BxLwNeCfxsgvkkSZIWtXHK1I7At0a2r2z3/VKS/YCdq+ojm/qDkhyVZFWSVWvXrv2Vw0qSJC02nQegJ9kCeC3w55s7tqpOqqoVVbVi+fLlXf9qSZKk3o1Tpr4N7DyyvVO7b8F2wN2ATye5ArgvsNJB6JIkaR6MU6bOA/ZMsnuSrYHDgZULL1bVj6vqjlW1W1XtBvwncGhVrZpKYkmSpEVks2Wqqq4HjgbOBC4FTquqi5OckOTQaQeUJElazLYc56CqOgM4Y719x23k2Ad3jyVJkrQ0uAK6JElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA7GKlNJDkqyOsmaJMds4PU/TXJRkguT/EeSvScfVZIkafHZbJlKsgw4EXgEsDdwxAbK0ruq6u5VdU/gVcBrJ55UkiRpERrnytT+wJqquqyq1gGnAoeNHlBV14xsbgvU5CJKkiQtXluOccyOwLdGtq8E7rP+QUmeA7wQ2Bo4cCLpJEmSFrmJDUCvqhOrag/gxcBfbeiYJEclWZVk1dq1ayf1V0uSJPVmnDL1bWDnke2d2n0bcyrwRxt6oapOqqoVVbVi+fLl46eUJElapMYpU+cBeybZPcnWwOHAytEDkuw5snkw8LXJRZQkSVq8NjtmqqquT3I0cCawDDi5qi5OcgKwqqpWAkcneRjwc+Bq4KnTDC1JkrRYjDMAnao6AzhjvX3HjXz9/AnnkiRJWhJcAV2SJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDsYqU0kOSrI6yZokx2zg9RcmuSTJl5OclWTXyUeVJElafDZbppIsA04EHgHsDRyRZO/1DvsisKKq9gHeB7xq0kElSZIWo3GuTO0PrKmqy6pqHXAqcNjoAVX1qar6abv5n8BOk40pSZK0OI1TpnYEvjWyfWW7b2OeCXy0SyhJkqSlYstJ/mFJngSsAB60kdePAo4C2GWXXSb5V0uSJPVinCtT3wZ2Htneqd13E0keBvwlcGhVXbehP6iqTqqqFVW1Yvny5bckryRJ0qIyTpk6D9gzye5JtgYOB1aOHpBkX+BNNEXq+5OPKUmStDhttkxV1fXA0cCZwKXAaVV1cZITkhzaHvZq4LbAe5NcmGTlRv44SZKkQRlrzFRVnQGcsd6+40a+ftiEc0mSJC0JroAuSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgdjlakkByVZnWRNkmM28PrvJbkgyfVJHjv5mJIkSYvTZstUkmXAicAjgL2BI5Lsvd5h3wSeBrxr0gElSZIWsy3HOGZ/YE1VXQaQ5FTgMOCShQOq6or2tV9MIaMkSdKiNc5tvh2Bb41sX9nukyRJmnszHYCe5Kgkq5KsWrt27Sz/akmSpKkYp0x9G9h5ZHundt+vrKpOqqoVVbVi+fLlt+SPkCRJWlTGKVPnAXsm2T3J1sDhwMrpxpIkSVoaNlumqup64GjgTOBS4LSqujjJCUkOBUhy7yRXAo8D3pTk4mmGliRJWizGmc1HVZ0BnLHevuNGvj6P5vafJEnSXHEFdEmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUgWVKkiSpA8uUJElSB5YpSZKkDixTkiRJHVimJEmSOrBMSZIkdWCZkiRJ6sAyJUmS1IFlSpIkqQPLlCRJUgeWKUmSpA4sU5IkSR1YpiRJkjqwTEmSJHVgmZIkSerAMiVJktSBZUqSJKkDy5QkSVIHlilJkqQOLFOSJEkdWKYkSZI6GKtMJTkoyeoka5Ics4HXb5XkPe3rX0iy26SDSpIkLUabLVNJlgEnAo8A9gaOSLL3eoc9E7i6qn4b+AfglZMOKkmStBiNc2Vqf2BNVV1WVeuAU4HD1jvmMODt7dfvAx6aJJOLKUmStDilqjZ9QPJY4KCq+uN2+8nAfarq6JFjvtIec2W7/fX2mB+s92cdBRzVbu4FrJ7U/5AZuyPwg80epUnyPZ893/PZ8z2fPd/z2Vuq7/muVbV8Qy9sOcsUVXUScNIs/85pSLKqqlb0nWOe+J7Pnu/57Pmez57v+ewN8T0f5zbft4GdR7Z3avdt8JgkWwK3A66aREBJkqTFbJwydR6wZ5Ldk2wNHA6sXO+YlcBT268fC3yyNnf/UJIkaQA2e5uvqq5PcjRwJrAMOLmqLk5yArCqqlYCbwXekWQN8EOawjVkS/5W5RLkez57vuez53s+e77nsze493yzA9AlSZK0ca6ALkmS1IFlSpIkqQPLlCRJUgeWKUmSpA5mumjnUpPkhZt6vapeO6ss88T3ffaSPHpTr1fVB2aVZV74Oe9PkgcAe1bVKUmWA7etqsv7zjVESS4CNjrTrar2mWGcqbFMbdp27e97AffmxvW1DgHO7SXRfPB9n71D2t/vBNwf+GS7/RDg84BlavL8nPcgyfHACpr3/RRgK+CdwAF95hqwR7a/P6f9/R3t70f2kGVqXBphDEk+AxxcVde229sBH6mq3+s32bD5vs9eko8BT62q77bbvwm8rar+oN9kw+XnfLaSXAjsC1xQVfu2+748lCski1WSLy683yP7Lqiq/frKNEmOmRrPrwPrRrbXtfs0Xb7vs7fzQpFqfQ/Ypa8wc8LP+Wyta5/QUQBJtu05z7xIkgNGNu7PgDqIt/nG8y/AuUk+2G7/EfD2HvPMiw2972/rL85cOCvJmcC72+0nAJ/oMc888PwyW6cleRNw+yTPAp4BvLnnTPPgmcDJSW7Xbv+I5r0fBG/zjSnJfsAD283PVNUX+8wzL3zfZy/Jo4CFW0yfqaoPbup4defnfLaSPBz4fSDAmVX18Z4jzY2FMlVVP+47yyRZpsbk7I/FIcltq+p/+s4xZEl2pfmsfyLJbYBlC+N5NB2eXzSPkjy9qk7pO8ckDOZ+5TS1sz9eDBzb7lqY/aHZu6TvAEPW3vZ4H/CmdteOwIf6SzR8nl9mK8mjk3wtyY+TXJPk2iTX9J1rTv113wEmxTFT43kU7ewPgKr6TjvjRlOwifV3Atx2llnm0HOA/YEvAFTV15Lcqd9Ig+f5ZbZeBRxSVZf2HWQeJPnyxl5iQBMtLFPjWVdVlcTZH7Pxd8Crges38JpXU6fruqpalwSAJFuyiQX3NBGeX2brexapmfp14A+Aq9fbH5o17AbBMjUeZ3/M1gXAh6rq/PVfSPLHPeSZJ2cneQmwTTtI99nA6T1nGjrPL7O1Ksl7aG5fX7ew01X+p+bfaMYAXrj+C0k+Pfs40+EA9DGNzP4A+JizP6YnyV7AVVX1gw289utV9b0eYs2FJFvQTGH+5Uwn4C3liWKqPL/MTpINDXiuqhrMNH3NnmVqTEl+g2YsSQHnVdV/9xxp8JLsV1UX9J1j3iTZGrgrzWd9dVWt28x/oo48v2jokvwjcGpVDebW3ijL1BjaW0vH0TyvLMCDgBOq6uRegw1ckk8Bv0Ezu+w9VfWVniMNXpKDgTcCX6f5rO8O/ElVfbTXYAPm+WU2kvxFVb0qyevZwDjAqnpeD7HmRpKn0iwCvBfwQZpitarfVJNjmRpDktXA/avqqnb714DPV9Ve/SYbvvYn9sfT/CPcnqZU/U2/qYYryVeBR1bVmnZ7D5rnxN2132TD5fllNpIcUlWnt9/Ub6aqXHV+BpLcAXgMcDiwS1Xt2XOkiXAA+niuAkYXLby23acpa293/GN7leovaH6Ct0xNz7ULRap1GTf97GvyPL/MQFWd3v5uaerXb9MMI9gVGMysSsvUJoysd7QG+EKSD9NcHj4M2NjaGZqQJL9Dc0XqMTTfXN4D/HmvoQYqyaPbL1clOQM4jeaz/jjgvN6CDZjnl9lKcjqbWOajqg6dYZy5k+RVNGuqfZ3mXP6yqvpRv6kmxzK1aQsL5329/bXgwz1kmUcnA6cCf1BV3+k7zMAdMvL192jG7QCsBbaZfZy54Plltv6+7wBz7uvA/TY0S3sIHDMlSZorSbahGa+zuu8s8yTJjjS39355IaeqPtNfosnxytQYkqwA/pKbfwj26S3UHEhyAPBSbnzfQ7MezJ37zDVkSXYHngvsxk0/694CmRLPL7OV5BCaq1RbA7snuSfN7Ek/41OU5BU0g84vAW5odxcwiDLllakxtLNtXgRcBPxiYX9VfaO3UHOgnVn2AuB8bvzHx8KsJ01eki8Bb+Xmn/Wzews1cJ5fZivJ+cCBwKerat9230VVdfd+kw1b+znfp6qu2+zBS5BXpsaztqpW9h1iDv3Y9Y1m7mdV9Y99h5gznl9m6+dV9eOF50+2vKowfZcBWzHyCJ8hsUyN5/gkbwHOwmc5zdKnkrwa+AA3fd9dFX16XpfkeOBj+J7PiueX2bo4yROBZUn2BJ7HgB64u4j9FLgwyfqf80EslmqZGs/TadbF2IobL8MXzTd5Tc992t9XjOwrmkv0mo67A0+meY9HP+u+59Pj+WW2nkszRu064N00z598Wa+J5sPK9tcgOWZqDElWuxqx5kGSNcDePo9vdjy/9CfJMmDbqrqm7yzzoH3u513azdVV9fM+80zSFn0HWCI+n2TvvkPMmyS3S/LaJKvaX69Jcru+cw3cV4Db9x1iznh+maEk70qyfZJtaQb9X5LkRX3nGrokDwa+BpwIvAH4ryS/12uoCfLK1BiSXArsAVxOc2l4YYq+U5enKMn7ab65Lzz+4cnAParq0Rv/r9RFkk8D+9Csej46rsFp41Pi+WW2klxYVfdMciSwH3AMcL7v93S1syifuLC2V5K7AO+uqnv1m2wyHDM1noP6DjCn9qiqx4xs/3WSC3tLMx+O7zvAHPL8MltbJdkK+CPgn6rq50m8qjB9W40uklpV/9X+/zAI3uYbQ7vey87Age3XP8X3bhb+N8kDFjbaRTz/t8c8g9euJ3UFzYnvbJorVM7kmyLPLzP3JprP+LbAZ5LsCjhmavpWJXlLkge3v94MrOo71KR4m28M7VTxFcBeVXWXJL8FvLeqDug52qC1KxO/HVgYJ3U18LSq+lJ/qYYtybOAo4A7VNUe7dTxN1bVQ3uONlieX/qXZMuqur7vHEOW5FbAc4CFH5A/C7xhKIt4WqbG0N5a2he4YGTF3C97j302kmwP4Iyb6Ws/6/sDX3B16Nnw/DJ7SQ4Gfhe49cK+qjqhv0TD1w74/1lV3dBuLwNuVVU/7TfZZHgpeTzrqmmdBb/8UGjKkvxdkttX1TVVdU2SHZL8Td+5Bu660WURkmyJq0NPm+eXGUryRuAJNOtNBXgczXMRNV1nAduMbG8DfKKnLBNnmRrPaUneBNy+vQ3yCeDNPWeaB4+oqh8tbFTV1cAf9phnHpyd5CXANkkeDrwXOL3nTEPn+WW27l9VTwGurqq/Bu7HjWsfaXpuXVX/s7DRfn2bHvNMlLP5xlBVf99+Y7kG2As4rqo+3nOsebAsya0W7qkn2Qa4Vc+Zhu4Y4Jk06+/8CXAG8JZeEw2c55eZW5jE8tN2fNpVwG/2mGde/CTJfguPpkpyLwY0ocgxUxOQ5Jyqul/fOYYmyYuBQ4BT2l1PB1ZW1av6SzXfkrx/veUqNGWeXyYryf8DXk/ziKQT291vqar/11+q4Utyb+BU4Ds0t1d/A3hCVZ3fa7AJsUxNQJIvLgwc1WQlOQh4WLv58ao6s888887P+uz5nk9We4X7z4AH0oxT+yzwz1X1s16DzYF2XamFRyfd5HEySR6+lK/IWqYmIMkFVbVf3znmjT+xz56f9dnzPZ+sJKcB1wLvbHc9EbhdVT2+v1Ra6p9zx0xpKbv15g+RpJu4W1WNPgvxU0ku6S2NFqTvAF04m28ylvSHYAnzsurs+VmfPd/zybogyX0XNpLchwGtxL2ELenzuVemJuPJfQeQZuTFfQeYQ55fJiDJRTTfsLcCPp/km+32rsBX+8ympc8yNYYkjwZeCdyJ5qfEhae6L6zM/ZUe480zf2KfsPb5hy+l+QazJTd+1u9M88XH+ks3TJ5fZuaRfQfQJl3Rd4AuHIA+hiRrgEOq6tK+s+hGSe7mN5rJSvJV4AXA+cANC/ur6qreQg2c5xcNWfvDwkZV1QdmlWWavDI1nu95opudJNeyifvn/sQ+VT+uqo/2HWLOeH7RkB3S/n4n4P7AJ9vthwCfByxTc2RVkvcAHwJ++YTroTTqxaaqtgNI8jLgu8A7aG59HIkrFU/bp5K8muYEN/pZv6C/SIPn+UWDVVVPB0jyMWDvqvpuu/2bwNt6jDZR3uYbQ5JTNrC7quoZMw8zR5J8qarusbl9mpwkn9rA7qqqA2ceZk54ftE8SHJpVf3OyPYWwMWj+5Yyr0yNYaFZa+Z+kuRImkcQFHAE8JN+Iw1bVT2k7wzzxvOL5sRZSc4E3t1uP4Hmod6D4DpTY0hylyRnJflKu71Pkr/qO9cceCLweOB77a/Htfs0JUl+Pclbk3y03d47yTP7zjVknl80D6rqaOCNwD3aXydV1XP7TTU53uYbQ5KzgRcBb1p4RlaSr1TV3fpNJk1WW6JOAf6yqu6RZEvgi1V1956jDZbnF82LJLsCe1bVJ5LcBlhWVdf2nWsSvDI1nttU1bnr7bu+lyRzxJ/Ye3HHqjoN+AVAVV3PyBIJmgrPLxq8JM8C3ge8qd21I82ki0GwTI3nB0n2oJ2un+SxNLPMNF1vBo4Ffg5QVV8GDu810fD9JMmvceNn/b7Aj/uNNN1YsiIAABbySURBVHieXzQPngMcAFwDUFVfo1kuYRAcgD6e5wAnAXdN8m3gcppp+pqu21TVuclNFjr3J/bpeiGwEtgjyeeA5cBj+400eJ5fNA+uq6p1C+fzdgjBYMYZWabGs0NVPSzJtsAWVXVtkkcC3+g72MD5E/vsXQ08CNiLZm2v1cA9e000fJ5fNA/OTvISYJskDweeDZzec6aJcQD6GJJcADxlYcXtJIcDL6iq+/SbbNiS3JnmJ/b703yTvxw4sqr8JjMlSc4HDq2qb7fbvwec6AD06fH8onnQriv1TOD3aX5QOxN4Sw2khFimxtB+U38fzbT8BwJPAR5ZVY4lmYHRn9j7zjJ0Se4NvIHmERD7AS+n+ax/q9dgA+b5RfMiydbAXWnuNqyuqnU9R5oYy9SYktyFZubBN4FHVdX/9hxp8NqB0McDD6D5x/cfwAk+dHe6ktyPZsbNz4CDq2ptz5EGz/OLhi7JwTTrTH2d5srU7sCfDOVZoJapTUhyETcdIHcnmplN1wFU1T595JoXST4OfAZ4Z7vrSODBVfWw/lINU5LTuelnfW+a8WlXA1TVoX3kGjLPL5onSb5Kc8V1Tbu9B/CRqrprv8kmwzK1Ce0CYxvl2J3p2tDChUkucvzO5CV50KZer6qzZ5VlXnh+0TxJcl5V3XtkO8C5o/uWMmfzbcLoySzJPWjGMwB8tqq+1E+qufKxdjDuae32Y2kGLWrCRstSkl8HFk5w51bV9/tJNWyeXzQPkjy6/XJVkjNozudF83iw83oLNmFemRpDkucDzwI+0O56FM1zhV7fX6rhSnItzT+2ANvSrsZNs8js/1TV9n1lG7okjwdeDXya5v1/IPCiqnpfn7mGzPOLhizJKZt6fSgP+rZMjSHJl4H7VdVP2u1tgXMc06ChSfIl4OELV6OSLAc+UVX36DfZcHl+kZY+b/ONJ9z0+WQ3tPs0ZUn2AXZj5LNaVR/Y6H+grrZY77beVfjYqWnz/KLBS7I78Fxufj4fxOQWy9R4TgG+kOSD7fYfASf3mGcuJDkZ2Ae4mBtv9RU33g7R5P17kjOBd7fbTwAGMXV5EfP8onnwIeCtNKue/2Izxy453uYbU5L9aNY7gmaA6Bf7zDMPklxSVXv3nWPetANGRz/rH9zU8erO84uGLskXhryqv2VqDEneUVVP3tw+TVaStwKvqapL+s4yL5K8sqpevLl9mhzPL5oHSZ4I7Al8jHYtNYCquqC3UBPkbb7x/O7oRpJlwL16yjJP/gU4J8l/0/zjC1AOzJ2qhwPrF6dHbGCfJsfzi+bB3YEnAwdy02EbB/aWaIIsU5uQ5Fhg4SnX1yzsBtbRPIBX0/VWmn98FzHAe+yLSZI/o3mK+53b2WULtgM+10+qYfP8ojnzOODOQ3oe3yhv840hycur6thNvP67VXXxLDPNgyTnVNX9+s4xD5LcDtiB5sHGx4y8dG1V/XDkuB2q6upZ5xsyzy+aB0k+BBw11EWALVMTkOSCqtqv7xxDk+QNwO1pZn+M3mN3Nl9P/KzPnu+5hiDJp2lmZ5/HTc/nLo2gX3JNmOnYhuYf3e+P7HNphH75WZ8933MNwfF9B5gmy9RkeHlvCobymIGB8bM+e77nWvKq6uz24d57VtUnktwGWNZ3rklxZWMtWknukuSsJF9pt/dJ8ld955Ik/WqSPAt4H/CmdteONAt5DoJlajPS2Hkzhw1ydsIi8GbgWODnAFX1ZeDwXhPJW06z5/lFQ/Ac4ADgGoCq+hpwp14TTZC3+TajqirJGTRrZGzsmPvOMNI8uU1VnZvc5Pv39X2FGbp2faOLq+qumzjsobPKM3TtqucbtbCYoecXDcR1VbVu4XyeZEsGdAvbMjWeC5Lcu6rO6zvInPlBkj1o/8EleSzw3X4jDVdV3ZBkdZJdquqbGznmhxvar1vkNe3vtwZWAF+iufK3D7AKcFkQDcnZSRbWVXs4zbp2p/ecaWJcGmEMSb4K/DbwDeAnuBL3TCS5M83ihfcHrgYuB46sqm/0GmzAknwG2Bc4l+azDgxn+vJilOQDwPFVdVG7fTfgpVX12H6TSZOTZAvgmTSzswOcCbylBlJCLFNjaGcg3Izf1GcjybbAFlV17Xr7n1pVb+8p1iAledCG9lfV2bPOMi+SXFxV6z9S5mb7pCFL8v6qekzfOW4py9SvIMmdaC7JA7CxWyGaDRcz1BAkeTfNVcB3truOBG5bVUf0l0qarSRfrKp9+85xSzmbbwxJDk3yNZrbTGcDVwAf7TWUwJllE5fkvknOS/I/SdYluWHkuXGajqcDFwPPb39d0u6T5smSvrLjAPTxvAy4L/CJqto3yUOAJ/WcSUv8H98i9U80y0+8l2ZQ9FOAu/SaaOCq6mdJ3gicUVWr+84j6Vfnlanx/LyqrgK2SLJFVX2K5huN+uWVqSmoqjXAsqq6oapOAQ7qO9OQJTkUuBD493b7nklW9ptKmrklfT73ytR4fpTktsBngH9N8n1GZjqpN5/rO8AA/TTJ1sCFSV5FsxSFP3RN1/HA/sCnAarqwiS795pImqB2Dbt/qaojN3HYi2eVZxocgD6GdjbZz2ia85HA7YB/ba9WaUqS3Ap4DLAbI8W/qk7oK9PQtTNXvwdsDbyA5rP+hvZqlaYgyX9W1X1HB+Am+bJLr2hIkvwHcGBVDXJFf69MjaGqRq9CORV/dj4M/Bg4H7iu5yxzYWS5j58Bf91nljlycZInAsuS7Ak8D/h8z5mkSbsM+Fx7C3t0DbvX9hdpcixTY0jyaOCVNM8RCjcu2rl9r8GGb6eqcrzODCU5AHgpsCs3vRp4574yzYHnAn9J8wPDu2kWM3xZr4mkyft6+2sLYLues0yct/nGkGQNcEhVXdp3lnmS5CTg9QsrQ2v62tX+X0BzNfCGhf3e0pakjfPK1Hi+Z5HqxQOApyW5nOandh/jM30/rirXUJuhJHcB/i83Hxt4YF+ZpElLshz4C+B3ueni14P4nHtlahPa23sADwJ+A/gQI2N3quoDfeSaFz7GZ3aSLKwk/3hgGfABbvpZv6CPXPMgyZeAN3Lzq4Hn9xZKmrAkHwPeQ/ODw58CTwXWVtWSnsW3wDK1CUlO2cTLVVXPmFmYOZXkAcCeVXVK+5PNbavq8r5zDU2ST23i5RrKT4+LUZLzq+pefeeQpmnhcz46UzXJeVV1776zTYK3+TahqnykQ4+SHE+zOOpewCnAVjTPLzugz1xDVFUP6TvDHDs9ybOBD3LTq4E/7C+SNHE/b3//bpKDge8Ad+gxz0S5GN8Ykrw9ye1HtndIcnKfmebEo4BDaafRVtV3GOAskMUkyd9t4LP+N31mmgNPBV5EsxzC+e2vVb0mkibvb5LcDvhzmlt9b6GZ7DII3uYbw4aeZr3Un3C9FCQ5t6r2T3JBVe3XLp56jgPQp2cjn/ULqmq/jf03kjTvvM03ni2S7FBVVwMkuQO+d7NwWpI3AbdP8izgGTQ/zWh6liW5VVVdB5BkG+BWPWcapCQHVtUnRya63IQTXDQk7ZjXZ3HzWauDGHtsIRjPa4Bzkry33X4c8Lc95pkXrwEeBlxDM27qOJrnI2p6/hU4a2TyxdNx1f9peRDwSeCQDbxWNDMqpaH4MPBZ4BOMzFodCm/zjSnJ3sDCjKZPVtUlI6/98qqVJifJyaM/tbQPm/5wVT20x1iDl+QgmhIL8PGqOrPPPJKWviQXVtU9+84xLZapCXBMyXQkeRnwa1X17CQ7AB8B3lxVm1qyQlOU5Jyqul/fOYamnd20/mKGPtBbg9FOZPl8VZ3Rd5ZpsExNgIPRpyfJq4DtgXsBr6iq9/ccaa75WZ+8JG8EbgM8hGZM4GOBc6vqmb0GkyYgybU0t60DbEuz/MfPGdgzbi1TE+CVqclab0BugP8HnAv8Ozgwt09+1idvYRHDkd9vC3y0qh7YdzZJ43EAuhaj9QfkfpFmwc5DcGCuhudn7e8/TfJbwFXAb/aYR5q4JI+iGW/843b79sCDq+pD/SabDMvUZKTvAEPiyvOLmp/1yTu9/cbyauACmh8Y3txvJGnijq+qDy5sVNWP2qdcDKJMuQL6GJLskeRW7dcPTvK80VWiAWeXTUGSnZJ8MMn321/vT7JT37nm3JP7DjAkSbYAzqqqH7XjAXcF7lpVx/UcTZq0DfWNwVzQsUyN5/3ADUl+GzgJ2Bl418KLPkNrak4BVgK/1f46vd2nCUtybZJrNvZr4biq+kqfOYemqn4BnDiyfd3CbRBpYFYleW17cWKPJK+leXTSIFimxvOLqrqe5llxr6+qF+GYhllYXlWnVNX17a+3Acv7DjVEVbVdO6vmdcAxwI7ATsCLgf+vz2xz4Kwkj0niLVQN2XOBdcB7gFNpxgo+p9dEE+RsvjEk+QLNN5S/BA6pqsuTfKWq7tZztEFLchbNlah3t7uOAJ7uop3Tk+RLVXWPze3T5LRTx7cFrqf5BjOoKePSOJK8vqqe23eOW8orU+N5OnA/4G/bIrU78I6eM82DZwCPB/4b+C7N+jtP6zPQHPhJkiOTLEuyRZIjgZ/0HWrI2quCW1TV1lW1/chVQmmeHNB3gC68MjWmJFsDd2k3V1fVz/vMMw+SHFBVn9vcPk1Okt1obvUdQDOr7HPA/6mqK/pLNWxJzlr/auuG9klDttTXsBvMSPppSvJgmoe9XkFzCX7nJE+tKh+6O12vB9b/x7WhfZqQtjQd1neOeZDk1jQrn9+xfVzSwpip7WnGrElaIixT43kN8PtVtRogyV1oxvHcq9dUA5XkfsD9geVJXjjy0vbAsn5SDVuSv6iqVyV5Pc0VqZuoquf1EGvo/gT4PzQzVS8Y2X8N8E+9JJL6s6QnYFimxrPVQpECqKr/SrJVn4EGbmvgtjSfz+1G9l9DM25Kk3dp+/uqXlPMkap6HfC6JM+tqtf3nUfq2ev6DtCFY6bGkORk4BfAO9tdRwLLquoZ/aUaviS7VtU3NvH6kp79sdgkWQa8sqr+b99Z5kmSbYEXALtU1VFJ9gT2qqp/6zmaNDFJVtDMiN+V5gflhVmr+/QabEIsU2NoVz9/DvCAdtdngTdU1XX9pdJSH7C4GCU5p6ru13eOeZLkPTSLFz6lqu6W5DbA56vqnj1HkyYmyWrgRcBFNBcnANjUD8xLibf5xlBV1yX5J+Asmg/B6qpa13MsaRouTLISeC8jSyJUlQ+Xnp49quoJSY4AqKqfuoCnBmhtVa3sO8S0WKbGkORg4I3A12kuTe6e5E+q6qP9JpMm7tbAVcCBI/sKsExNz7ok29AO/E+yB+BVbw3N8UneQnNR4pef76H8oGaZGs9rgIdU1Rr45cnuI4Blql/+9D5hVfX0vjPMoeOBf6dZcuVfadb4elqviaTJezpwV2ArbrzNN5gf1CxT47l2oUi1LgOu7SuMfmlJz/5YjJLcmeZ9vS/Nie4cmkU7L+812EAl2QLYAXg0zXse4PlV9YNeg0mTd++q2qvvENPiAPQxJPlnmhkIp9F8g3kc8E3gEzCcy5SLzdBnfyxGSf4TOJEbn4d4OPDcqrpPf6mGLcmqqlrRdw5pmpKcAry6qi7pO8s0WKbG0H4INqZcImE6hj77YzFK8uX1y6oPOp6uJK8AfgC8h5sO+v9hb6GkCUtyKbAHcDnNmKlB/XBsmZqAJMdW1cv7zjE0Sf6jqh6w+SPVVZI7tF++GLgaOJXmKuwTgB2q6ti+sg1dksvZ8Krzd+4hjjQVSXbd0P6h/HBsmZoA1zuajiQPBY5goLM/FpORb+gbGtRffmOfnnYm37Np1rErmnXs3lhV/9trMGnCktwDeGC7+dmq+lKfeSbJAeiT4ayy6Rj07I/FpKp27zvDHHs7zaOS/rHdfmK77/G9JZImLMnzgWdx4/n7nUlOGsqjlLwyNQFemZqOJKuHPPtjMUpyPvBW4F1V9aO+88yDJJdU1d6b2yctZUm+DNyvqn7Sbm8LnDOUMVNb9B1gILwyNR2fT+I3lNl6ArAjsCrJqUn+wNW4p+6CJPdd2EhyH3zgtIYnwA0j2zcwoO+dXpmagCQvqaq/6zvH0Ax99sdi1q5/9Ejgn2lOeqcAr3OG2eS1n/O9aJZbAdgFWA1cj593DUSSFwJPBT7Y7voj4O1V9Q/9pZocy9QmJHk9G5hls6CqnjfDOHNn6LM/Fqsk+wDPAB4BnAn8K83g6Cf78N3J29jnfIGfdw1Fkv1oziXQDED/Yp95JskB6Ju2cKn9AGBvmnVgoFm0c5ALjy0mVfWNIc/+WIzaMVM/At4CvLiqFmZRfiHJAf0lGy7LkuZBkndU1ZOBCzawb8nzytQY2lWhH1BV17fbW9F8Y7/vpv9LdbGB2R+PAgYz+2Mxaseo7cuNq84DUFUn9BZK0pK3/kStJMuAi4Yy0cIrU+PZAdgeWBgvctt2n6brmcB9RmZ/vJLmWXGWqel5Lc2VqQsYWdtLkm6JJMcCLwG2SXLNwm5gHXBSb8EmzDI1nlcAX0zyKZoPwe8BL+010XwY9OyPRWqnqjqo7xCShqF9OsjLk7x8yE9SsEyNoapOSfJRYOFhry+uqv/uM9OcOIVmrM7o7I+Te8wzDz6f5O5VdVHfQSQNyr8l2baqfpLkScB+NDOEBzFm0DFTm5DkrlX11XYGws1U1QUb2q/JGfLsj8UkyUU0M1e3BPYELsPlKCRNSLto5z2AfYC30UxyeXxVPajPXJNimdqEdqn7o9rbe+urqjpw5qHmyIZmegxp9sdi4vR8SdO0MAA9yXHAt6vqrUN6eoi3+TahLVJbAH9VVZ/rO88c+t3RjXb2x716yjJoliVJU3ZtOxj9ScDvtd9bt+o508T4OJnNqKpfAP/Ud455kuTYJNcC+yS5pv11LfB94MM9x5Mk/eqeQDN04JntmOOdgFf3G2lyvM03hiR/TzMl/wPlGzYzQ5/9IUkaBsvUGNqrItvSTM3/X24clLt9r8EGrl1x+8Khzv6QpHnRfh9dKBxb09zi+5+qul1/qSbH23xjqKrtqmqLqtqqqrZvty1S0/fPwE/bR8r8OfB14F/6jSRJ+lUtfN9sv3duAzyG5hw/CF6ZGlOSR9NM0S+aKfof6jnS4A199ockzbMkX6yqffvOMQnO5htDkjcAvw28u931p0keXlXP6THWPBj07A9JmhftBYkFWwArgJ/1FGfivDI1hiRfBX5nYfB5+0394qr6nX6TDVuS3wCeCJxXVZ9Nsgvw4KryVp8kLSFJThnZvB64gubB9Wv7STRZXpkazxpgF2Bh4PPO7T5NUTt99rUj29/EMVOStBRtATy/qn4EkGQH4DXAM3pNNSGWqU1IcjrNGKntgEuTnNtu3wc4t89s82Dosz8kaY7ss1CkAKrq6iSDGC8FlqnN+fu+A8yzqtpu4eskAQ4D7ttfIknSLbRFkh2q6mqAJHdgQB3EMVNaUoY0+0OS5kWSpwAvAd7b7noc8LdV9Y7+Uk2OZWoTkvxHVT1gvdtN4KKdM7GR2R8Pqqr79RRJknQLJdkbOLDd/GRVXdJnnkmyTGnRGvrsD0nSMAzmfuW0JFlGswzCXfvOMocGPftDkjQMPk5mM6rqBmB1u8aRZutmsz8Ax0tJkhYVr0yNZwfg4nZphJ8s7KyqQ/uLNBcGPftDkjQMfmMaz62BR45sB3hlT1nmyWuAc5LcZPZHj3kkSboZy9R4tqyqs0d3JNmmrzDzoqr+Jckqbpz98eghzf6QJA2Ds/k2IcmfAc8G7gx8feSl7YDPVdWTegkmSZIWDcvUJiS5Hc14qZcDx4y8dG1V/bCfVJIkaTGxTEmSJHXg0giSJEkdWKYkSZI6sExJkiR1YJmSJEnqwDIlSZLUwf8PHLQOfJlP9OcAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 720x504 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z4jCp28Gdto7"
      },
      "source": [
        "### Save and load best performing model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZOJrm1UDd0Ry"
      },
      "source": [
        "# Save best performing model to SavedModel format (default)\n",
        "model_5.save(\"skimlit_tribrid_model\") # model will be saved to path specified by string"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ef4xwQ-7d2No",
        "outputId": "d0467a3d-161c-46f4-f7ba-85ad4df6a1df"
      },
      "source": [
        "# Download pretrained model from Google Storage\n",
        "!wget https://storage.googleapis.com/ztm_tf_course/skimlit/skimlit_tribrid_model.zip\n",
        "!mkdir skimlit_gs_model\n",
        "!unzip skimlit_tribrid_model.zip -d skimlit_gs_model"
      ],
      "execution_count": 107,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-27 17:15:13--  https://storage.googleapis.com/ztm_tf_course/skimlit/skimlit_tribrid_model.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 142.251.2.128, 142.250.141.128, 2607:f8b0:4023:c03::80, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|142.251.2.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 962957902 (918M) [application/zip]\n",
            "Saving to: ‘skimlit_tribrid_model.zip’\n",
            "\n",
            "skimlit_tribrid_mod 100%[===================>] 918.35M  37.4MB/s    in 13s     \n",
            "\n",
            "2021-06-27 17:15:26 (70.0 MB/s) - ‘skimlit_tribrid_model.zip’ saved [962957902/962957902]\n",
            "\n",
            "Archive:  skimlit_tribrid_model.zip\n",
            "   creating: skimlit_gs_model/skimlit_tribrid_model/\n",
            "   creating: skimlit_gs_model/skimlit_tribrid_model/assets/\n",
            "   creating: skimlit_gs_model/skimlit_tribrid_model/variables/\n",
            "  inflating: skimlit_gs_model/skimlit_tribrid_model/variables/variables.index  \n",
            "  inflating: skimlit_gs_model/skimlit_tribrid_model/variables/variables.data-00000-of-00001  \n",
            "  inflating: skimlit_gs_model/skimlit_tribrid_model/saved_model.pb  \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3cQTdRCJd6Bz",
        "outputId": "ac70d2b1-e6cb-4d85-e513-e01f6a7e40a3"
      },
      "source": [
        "import tensorflow_hub as hub\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "\n",
        "model_path = \"skimlit_gs_model/skimlit_tribrid_model\"\n",
        "\n",
        "# Load downloaded model from Google Storage\n",
        "loaded_model = tf.keras.models.load_model(model_path,\n",
        "                                          custom_objects={\"TextVectorization\": TextVectorization, # required for char vectorization\n",
        "                                                          \"KerasLayer\": hub.KerasLayer}) # required for token embedding"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named \"keras_metadata.pb\" in the SavedModel directory.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:SavedModel saved prior to TF 2.5 detected when loading Keras model. Please ensure that you are saving the model with model.save() or tf.keras.models.save_model(), *NOT* tf.saved_model.save(). To confirm, there should be a file named \"keras_metadata.pb\" in the SavedModel directory.\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fnJE4shbd8OT"
      },
      "source": [
        "### Make predictions and evaluate them against the truth labels"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "a0mNuHS-eInK",
        "outputId": "640b3b61-2474-4fe2-f87a-d345273a49b4"
      },
      "source": [
        "# Make predictions with the loaded model on the validation set\n",
        "loaded_pred_probs = loaded_model.predict(val_pos_char_token_dataset, verbose=1)\n",
        "loaded_preds = tf.argmax(loaded_pred_probs, axis=1)\n",
        "loaded_preds[:10]"
      ],
      "execution_count": 109,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "945/945 [==============================] - 22s 21ms/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=int64, numpy=array([0, 0, 3, 2, 2, 4, 4, 4, 4, 1])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xxpFEqHjeKjU",
        "outputId": "9fd213cd-be35-4c59-f724-8ca293360388"
      },
      "source": [
        "# Evaluate loaded model's predictions\n",
        "loaded_model_results = calculate_results(val_labels_encoded,\n",
        "                                         loaded_preds)\n",
        "loaded_model_results"
      ],
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 82.83132530120481,\n",
              " 'f1': 0.8272937671199255,\n",
              " 'precision': 0.8268115620164092,\n",
              " 'recall': 0.8283132530120482}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 110
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zzKyDjWYeOzJ",
        "outputId": "c15aedb5-0195-4679-c524-577a79b6371f"
      },
      "source": [
        "# Check loaded model summary (note the number of trainable parameters)\n",
        "loaded_model.summary()"
      ],
      "execution_count": 112,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_18\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "char_inputs (InputLayer)        [(None, 1)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "char_vectorizer (TextVectorizat (None, 290)          0           char_inputs[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "token_inputs (InputLayer)       [(None,)]            0                                            \n",
            "__________________________________________________________________________________________________\n",
            "char_embed (Embedding)          (None, 290, 25)      1750        char_vectorizer[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "universal_sentence_encoder (Ker (None, 512)          256797824   token_inputs[0][0]               \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_3 (Bidirectional) (None, 64)           14848       char_embed[0][0]                 \n",
            "__________________________________________________________________________________________________\n",
            "token_char_hybrid_embedding (Co (None, 576)          0           universal_sentence_encoder[0][0] \n",
            "                                                                 bidirectional_3[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "line_number_input (InputLayer)  [(None, 15)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "total_lines_input (InputLayer)  [(None, 20)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "dense_18 (Dense)                (None, 256)          147712      token_char_hybrid_embedding[0][0]\n",
            "__________________________________________________________________________________________________\n",
            "dense_16 (Dense)                (None, 32)           512         line_number_input[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "dense_17 (Dense)                (None, 32)           672         total_lines_input[0][0]          \n",
            "__________________________________________________________________________________________________\n",
            "dropout_4 (Dropout)             (None, 256)          0           dense_18[0][0]                   \n",
            "__________________________________________________________________________________________________\n",
            "token_char_positional_embedding (None, 320)          0           dense_16[0][0]                   \n",
            "                                                                 dense_17[0][0]                   \n",
            "                                                                 dropout_4[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "output_layer (Dense)            (None, 5)            1605        token_char_positional_embedding[0\n",
            "==================================================================================================\n",
            "Total params: 256,964,923\n",
            "Trainable params: 167,099\n",
            "Non-trainable params: 256,797,824\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tOrPrZhueQ6j"
      },
      "source": [
        "## Evaluate model on test dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eqIXA2wWehvj",
        "outputId": "84866414-1162-46e0-e6fe-4cbd7442f247"
      },
      "source": [
        "# Create test dataset batch and prefetched\n",
        "test_pos_char_token_data = tf.data.Dataset.from_tensor_slices((test_line_numbers_one_hot,\n",
        "                                                               test_total_lines_one_hot,\n",
        "                                                               test_sentences,\n",
        "                                                               test_chars))\n",
        "test_pos_char_token_labels = tf.data.Dataset.from_tensor_slices(test_labels_one_hot)\n",
        "test_pos_char_token_dataset = tf.data.Dataset.zip((test_pos_char_token_data, test_pos_char_token_labels))\n",
        "test_pos_char_token_dataset = test_pos_char_token_dataset.batch(32).prefetch(tf.data.AUTOTUNE)\n",
        "\n",
        "# Check shapes\n",
        "test_pos_char_token_dataset"
      ],
      "execution_count": 113,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<PrefetchDataset shapes: (((None, 15), (None, 20), (None,), (None,)), (None, 5)), types: ((tf.float32, tf.float32, tf.string, tf.string), tf.float64)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mvWqEdnhekLl",
        "outputId": "22afaefe-2c5d-4537-d48b-9379b7898ecf"
      },
      "source": [
        "# Make predictions on the test dataset\n",
        "test_pred_probs = loaded_model.predict(test_pos_char_token_dataset,\n",
        "                                       verbose=1)\n",
        "test_preds = tf.argmax(test_pred_probs, axis=1)\n",
        "test_preds[:10]"
      ],
      "execution_count": 114,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "942/942 [==============================] - 21s 22ms/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(10,), dtype=int64, numpy=array([3, 3, 2, 2, 4, 4, 4, 1, 4, 0])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_jUX9PI8el2H",
        "outputId": "74135155-6f74-4d91-a887-7067e77c8255"
      },
      "source": [
        "# Evaluate loaded model test predictions\n",
        "loaded_model_test_results = calculate_results(y_true=test_labels_encoded,\n",
        "                                              y_pred=test_preds)\n",
        "loaded_model_test_results"
      ],
      "execution_count": 115,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'accuracy': 82.3859299817488,\n",
              " 'f1': 0.8228365555962726,\n",
              " 'precision': 0.8224125471470242,\n",
              " 'recall': 0.823859299817488}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fC9VMwXWenez"
      },
      "source": [
        "### Find most wrong\n",
        "\n",
        "One of the best ways to investigate where your model is going wrong (or potentially where your data is wrong) is to visualize the \"most wrong\" predictions.\n",
        "\n",
        "The most wrong predictions are samples where the model has made a prediction with a high probability but has gotten it wrong (the model's prediction disagreess with the ground truth label).\n",
        "\n",
        "Looking at the most wrong predictions can give us valuable information on how to improve further models or fix the labels in our data.\n",
        "\n",
        "Let's write some code to help us visualize the most wrong predictions from the test dataset.\n",
        "\n",
        "First we'll convert all of our integer-based test predictions into their string-based class names."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WZvrKHwIerNZ",
        "outputId": "3aaa8440-2bc4-4262-c10b-02d75a86a2e9"
      },
      "source": [
        "%%time\n",
        "# Get list of class names of test predictions\n",
        "test_pred_classes = [label_encoder.classes_[pred] for pred in test_preds]\n",
        "test_pred_classes"
      ],
      "execution_count": 116,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 9.54 s, sys: 629 ms, total: 10.2 s\n",
            "Wall time: 8.45 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "y_M_9TVyetNf"
      },
      "source": [
        "Now we'll enrich our test DataFame with a few values:\n",
        "\n",
        "* A \"`prediction`\" (string) column containing our model's prediction for a given sample.\n",
        "* A \"`pred_prob`\" (float) column containing the model's maximum prediction probabiliy for a given sample.\n",
        "* A \"`correct`\" (bool) column to indicate whether or not the model's prediction matches the sample's target label.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "libqHrwpe4AB",
        "outputId": "8b6941f2-9228-43c1-bf3e-8fde00316b26"
      },
      "source": [
        "# Create prediction-enriched test dataframe\n",
        "test_df[\"prediction\"] = test_pred_classes # create column with test prediction class names\n",
        "test_df[\"pred_prob\"] = tf.reduce_max(test_pred_probs, axis=1).numpy() # get the maximum prediction probability\n",
        "test_df[\"correct\"] = test_df[\"prediction\"] == test_df[\"target\"] # create binary column for whether the prediction is right or not\n",
        "test_df.head(20)"
      ],
      "execution_count": 117,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>text</th>\n",
              "      <th>line_number</th>\n",
              "      <th>total_lines</th>\n",
              "      <th>prediction</th>\n",
              "      <th>pred_prob</th>\n",
              "      <th>correct</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>this study analyzed liver function abnormaliti...</td>\n",
              "      <td>0</td>\n",
              "      <td>8</td>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>0.530844</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>a post hoc analysis was conducted with the use...</td>\n",
              "      <td>1</td>\n",
              "      <td>8</td>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>0.319480</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>liver function tests ( lfts ) were measured at...</td>\n",
              "      <td>2</td>\n",
              "      <td>8</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.739303</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>survival analyses were used to assess the asso...</td>\n",
              "      <td>3</td>\n",
              "      <td>8</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.606798</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the percentage of patients with abnormal lfts ...</td>\n",
              "      <td>4</td>\n",
              "      <td>8</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.742463</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>when mean hemodynamic profiles were compared i...</td>\n",
              "      <td>5</td>\n",
              "      <td>8</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.899310</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>multivariable analyses revealed that patients ...</td>\n",
              "      <td>6</td>\n",
              "      <td>8</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.525651</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>abnormal lfts are common in the adhf populatio...</td>\n",
              "      <td>7</td>\n",
              "      <td>8</td>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>0.449698</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>elevated meld-xi scores are associated with po...</td>\n",
              "      <td>8</td>\n",
              "      <td>8</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.523380</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>minimally invasive endovascular aneurysm repai...</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>0.544405</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>the aim of this study was to analyse the cost-...</td>\n",
              "      <td>1</td>\n",
              "      <td>12</td>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>0.486076</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>resource use was determined from the amsterdam...</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.669381</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>the analysis was performed from a provider per...</td>\n",
              "      <td>3</td>\n",
              "      <td>12</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.859642</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>all costs were calculated as if all patients h...</td>\n",
              "      <td>4</td>\n",
              "      <td>12</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.542948</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>a total of @ patients were randomized .</td>\n",
              "      <td>5</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.702793</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the @-day mortality rate was @ per cent after ...</td>\n",
              "      <td>6</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.596260</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>at @months , the total mortality rate for evar...</td>\n",
              "      <td>7</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.884152</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the mean cost difference between evar and or w...</td>\n",
              "      <td>8</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.797754</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>the incremental cost-effectiveness ratio per p...</td>\n",
              "      <td>9</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.806620</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>there was no significant difference in quality...</td>\n",
              "      <td>10</td>\n",
              "      <td>12</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.765841</td>\n",
              "      <td>True</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         target  ... correct\n",
              "0    BACKGROUND  ...   False\n",
              "1       RESULTS  ...   False\n",
              "2       RESULTS  ...   False\n",
              "3       RESULTS  ...   False\n",
              "4       RESULTS  ...    True\n",
              "5       RESULTS  ...    True\n",
              "6       RESULTS  ...    True\n",
              "7   CONCLUSIONS  ...    True\n",
              "8   CONCLUSIONS  ...   False\n",
              "9    BACKGROUND  ...    True\n",
              "10   BACKGROUND  ...   False\n",
              "11      METHODS  ...    True\n",
              "12      METHODS  ...    True\n",
              "13      METHODS  ...    True\n",
              "14      RESULTS  ...    True\n",
              "15      RESULTS  ...    True\n",
              "16      RESULTS  ...    True\n",
              "17      RESULTS  ...    True\n",
              "18      RESULTS  ...    True\n",
              "19      RESULTS  ...    True\n",
              "\n",
              "[20 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 117
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 589
        },
        "id": "ChgZH2-6e51r",
        "outputId": "5f0352b2-c695-46b8-bd43-c5192884ad19"
      },
      "source": [
        "# Find top 100 most wrong samples (note: 100 is an abitrary number, you could go through all of them if you wanted)\n",
        "top_100_wrong = test_df[test_df[\"correct\"] == False].sort_values(\"pred_prob\", ascending=False)[:100]\n",
        "top_100_wrong"
      ],
      "execution_count": 118,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>target</th>\n",
              "      <th>text</th>\n",
              "      <th>line_number</th>\n",
              "      <th>total_lines</th>\n",
              "      <th>prediction</th>\n",
              "      <th>pred_prob</th>\n",
              "      <th>correct</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>16347</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>to evaluate the effects of the lactic acid bac...</td>\n",
              "      <td>0</td>\n",
              "      <td>12</td>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>0.938932</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3573</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>a cluster randomised trial was implemented wit...</td>\n",
              "      <td>3</td>\n",
              "      <td>16</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.934583</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13874</th>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>symptom outcomes will be assessed and estimate...</td>\n",
              "      <td>4</td>\n",
              "      <td>6</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.929205</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29294</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>baseline measures included sociodemographics ,...</td>\n",
              "      <td>4</td>\n",
              "      <td>13</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.915164</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>835</th>\n",
              "      <td>BACKGROUND</td>\n",
              "      <td>to assess the temporal patterns of late gastro...</td>\n",
              "      <td>0</td>\n",
              "      <td>11</td>\n",
              "      <td>OBJECTIVE</td>\n",
              "      <td>0.907444</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7823</th>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>at @ year , mortality rates in the pi and ppci...</td>\n",
              "      <td>8</td>\n",
              "      <td>10</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.825351</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8638</th>\n",
              "      <td>METHODS</td>\n",
              "      <td>this study is registered with clinicaltrials.g...</td>\n",
              "      <td>5</td>\n",
              "      <td>9</td>\n",
              "      <td>RESULTS</td>\n",
              "      <td>0.824962</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2605</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>circulating epc ( cells positive for cd@ , cd@...</td>\n",
              "      <td>4</td>\n",
              "      <td>10</td>\n",
              "      <td>METHODS</td>\n",
              "      <td>0.824808</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3859</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>this suggests that generalisation of habituati...</td>\n",
              "      <td>9</td>\n",
              "      <td>11</td>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>0.824272</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10089</th>\n",
              "      <td>RESULTS</td>\n",
              "      <td>qualitative findings in the community trial al...</td>\n",
              "      <td>10</td>\n",
              "      <td>19</td>\n",
              "      <td>CONCLUSIONS</td>\n",
              "      <td>0.823939</td>\n",
              "      <td>False</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>100 rows × 7 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "            target  ... correct\n",
              "16347   BACKGROUND  ...   False\n",
              "3573       RESULTS  ...   False\n",
              "13874  CONCLUSIONS  ...   False\n",
              "29294      RESULTS  ...   False\n",
              "835     BACKGROUND  ...   False\n",
              "...            ...  ...     ...\n",
              "7823   CONCLUSIONS  ...   False\n",
              "8638       METHODS  ...   False\n",
              "2605       RESULTS  ...   False\n",
              "3859       RESULTS  ...   False\n",
              "10089      RESULTS  ...   False\n",
              "\n",
              "[100 rows x 7 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xKUfDdr-e8Cj",
        "outputId": "5cf8c04e-9dce-46fb-92da-ddedb43d5543"
      },
      "source": [
        "# Investigate top wrong preds\n",
        "for row in top_100_wrong[0:10].itertuples(): # adjust indexes to view different samples\n",
        "  _, target, text, line_number, total_lines, prediction, pred_prob, _ = row\n",
        "  print(f\"Target: {target}, Pred: {prediction}, Prob: {pred_prob}, Line number: {line_number}, Total lines: {total_lines}\\n\")\n",
        "  print(f\"Text:\\n{text}\\n\")\n",
        "  print(\"-----\\n\")"
      ],
      "execution_count": 119,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Target: BACKGROUND, Pred: OBJECTIVE, Prob: 0.9389324188232422, Line number: 0, Total lines: 12\n",
            "\n",
            "Text:\n",
            "to evaluate the effects of the lactic acid bacterium lactobacillus salivarius on caries risk factors .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: RESULTS, Pred: METHODS, Prob: 0.9345834255218506, Line number: 3, Total lines: 16\n",
            "\n",
            "Text:\n",
            "a cluster randomised trial was implemented with @,@ children in @ government primary schools on the south coast of kenya in @-@ .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: CONCLUSIONS, Pred: METHODS, Prob: 0.9292047619819641, Line number: 4, Total lines: 6\n",
            "\n",
            "Text:\n",
            "symptom outcomes will be assessed and estimates of cost-effectiveness made .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: RESULTS, Pred: METHODS, Prob: 0.9151636958122253, Line number: 4, Total lines: 13\n",
            "\n",
            "Text:\n",
            "baseline measures included sociodemographics , standardized anthropometrics , asthma control test ( act ) , gerd symptom assessment scale , pittsburgh sleep quality index , and berlin questionnaire for sleep apnea .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: BACKGROUND, Pred: OBJECTIVE, Prob: 0.9074438810348511, Line number: 0, Total lines: 11\n",
            "\n",
            "Text:\n",
            "to assess the temporal patterns of late gastrointestinal ( gi ) and genitourinary ( gu ) radiotherapy toxicity and resolution rates in a randomised controlled trial ( all-ireland cooperative oncology research group @-@ ) assessing duration of neo-adjuvant ( na ) hormone therapy for localised prostate cancer .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: RESULTS, Pred: METHODS, Prob: 0.905343234539032, Line number: 3, Total lines: 13\n",
            "\n",
            "Text:\n",
            "data were collected prospectively for @ months beginning after completion of the first @ group clinic appointments ( @ months post randomization ) .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: METHODS, Pred: OBJECTIVE, Prob: 0.9049059748649597, Line number: 0, Total lines: 7\n",
            "\n",
            "Text:\n",
            "to determine whether the insulin resistance that exists in metabolic syndrome ( mets ) patients is modulated by dietary fat composition .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: BACKGROUND, Pred: OBJECTIVE, Prob: 0.9047386050224304, Line number: 0, Total lines: 9\n",
            "\n",
            "Text:\n",
            "to compare the efficacy of the newcastle infant dialysis and ultrafiltration system ( nidus ) with peritoneal dialysis ( pd ) and conventional haemodialysis ( hd ) in infants weighing < @ kg .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: METHODS, Pred: RESULTS, Prob: 0.9013400673866272, Line number: 5, Total lines: 7\n",
            "\n",
            "Text:\n",
            "at this time , an as@ response was achieved by @ ( @ % ) and @ ( @ % ) patients in groups @ and @ , respectively ( p < @ for all ) .\n",
            "\n",
            "-----\n",
            "\n",
            "Target: CONCLUSIONS, Pred: RESULTS, Prob: 0.8980521559715271, Line number: 7, Total lines: 9\n",
            "\n",
            "Text:\n",
            "pdt was associated with a significant decrease in bleeding scores ( p = @ ) as well as inflammatory exudation ( p = @ ) .\n",
            "\n",
            "-----\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tHLDP3xde_H6"
      },
      "source": [
        "# Make example predictions\n",
        "\n",
        "Okay, we've made some predictions on the test dataset, now's time to really test our model out.\n",
        "\n",
        "To do so, we're going to get some data from the wild and see how our model performs.\n",
        "\n",
        "In other words, were going to find an RCT abstract from PubMed, preprocess the text so it works with our model, then pass each sequence in the wild abstract through our model to see what label it predicts.\n",
        "\n",
        "For an appropriate sample, we'll need to search PubMed for RCT's (randomized controlled trials) without abstracts which have been split up (on exploring PubMed you'll notice many of the abstracts are already preformatted into separate sections, this helps dramatically with readability).\n",
        "\n",
        "Looking at the large chunk of text can seem quite intimidating. Now imagine you're a medical researcher trying to skim through the literature to find a study relevant to your work.\n",
        "\n",
        "Sounds like quite the challenge right?\n",
        "\n",
        "Enter SkimLit 🤓🔥!\n",
        "\n",
        "Let's see what our best model so far (`model_5`) makes of the above abstract.\n",
        "\n",
        "But wait...\n",
        "\n",
        "As you might've guessed the above abstract hasn't been formatted in the same structure as the data our model has been trained on. Therefore, before we can make a prediction on it, we need to preprocess it just as we have our other sequences.\n",
        "\n",
        "More specifically, for each abstract, we'll need to:\n",
        "\n",
        "1. Split it into sentences (lines).\n",
        "2. Split it into characters.\n",
        "3. Find the number of each line.\n",
        "4. Find the total number of lines.\n",
        "\n",
        "Starting with number 1, there are a couple of ways to split our abstracts into actual sentences. A simple one would be to use Python's in-built split() string method, splitting the abstract wherever a fullstop appears. However, can you imagine where this might go wrong?\n",
        "\n",
        "Another more advanced option would be to leverage spaCy's (a very powerful NLP library) sentencizer class. Which is an easy to use sentence splitter based on spaCy's English language model.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MYmLVGClfFe4",
        "outputId": "01b98597-0d22-438a-e2fc-b9e380c9ec59"
      },
      "source": [
        "import json\n",
        "# Download and open example abstracts (copy and pasted from PubMed)\n",
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/skimlit_example_abstracts.json\n",
        "\n",
        "with open(\"skimlit_example_abstracts.json\", \"r\") as f:\n",
        "  example_abstracts = json.load(f)\n",
        "\n",
        "example_abstracts"
      ],
      "execution_count": 121,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-06-27 17:20:54--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/skimlit_example_abstracts.json\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 6737 (6.6K) [text/plain]\n",
            "Saving to: ‘skimlit_example_abstracts.json.1’\n",
            "\n",
            "\r          skimlit_e   0%[                    ]       0  --.-KB/s               \rskimlit_example_abs 100%[===================>]   6.58K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-06-27 17:20:55 (94.6 MB/s) - ‘skimlit_example_abstracts.json.1’ saved [6737/6737]\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'abstract': 'This RCT examined the efficacy of a manualized social intervention for children with HFASDs. Participants were randomly assigned to treatment or wait-list conditions. Treatment included instruction and therapeutic activities targeting social skills, face-emotion recognition, interest expansion, and interpretation of non-literal language. A response-cost program was applied to reduce problem behaviors and foster skills acquisition. Significant treatment effects were found for five of seven primary outcome measures (parent ratings and direct child measures). Secondary measures based on staff ratings (treatment group only) corroborated gains reported by parents. High levels of parent, child and staff satisfaction were reported, along with high levels of treatment fidelity. Standardized effect size estimates were primarily in the medium and large ranges and favored the treatment group.',\n",
              "  'details': 'RCT of a manualized social treatment for high-functioning autism spectrum disorders',\n",
              "  'source': 'https://pubmed.ncbi.nlm.nih.gov/20232240/'},\n",
              " {'abstract': \"Postpartum depression (PPD) is the most prevalent mood disorder associated with childbirth. No single cause of PPD has been identified, however the increased risk of nutritional deficiencies incurred through the high nutritional requirements of pregnancy may play a role in the pathology of depressive symptoms. Three nutritional interventions have drawn particular interest as possible non-invasive and cost-effective prevention and/or treatment strategies for PPD; omega-3 (n-3) long chain polyunsaturated fatty acids (LCPUFA), vitamin D and overall diet. We searched for meta-analyses of randomised controlled trials (RCT's) of nutritional interventions during the perinatal period with PPD as an outcome, and checked for any trials published subsequently to the meta-analyses. Fish oil: Eleven RCT's of prenatal fish oil supplementation RCT's show null and positive effects on PPD symptoms. Vitamin D: no relevant RCT's were identified, however seven observational studies of maternal vitamin D levels with PPD outcomes showed inconsistent associations. Diet: Two Australian RCT's with dietary advice interventions in pregnancy had a positive and null result on PPD. With the exception of fish oil, few RCT's with nutritional interventions during pregnancy assess PPD. Further research is needed to determine whether nutritional intervention strategies during pregnancy can protect against symptoms of PPD. Given the prevalence of PPD and ease of administering PPD measures, we recommend future prenatal nutritional RCT's include PPD as an outcome.\",\n",
              "  'details': 'Formatting removed (can be used to compare model to actual example)',\n",
              "  'source': 'https://pubmed.ncbi.nlm.nih.gov/28012571/'},\n",
              " {'abstract': 'Mental illness, including depression, anxiety and bipolar disorder, accounts for a significant proportion of global disability and poses a substantial social, economic and heath burden. Treatment is presently dominated by pharmacotherapy, such as antidepressants, and psychotherapy, such as cognitive behavioural therapy; however, such treatments avert less than half of the disease burden, suggesting that additional strategies are needed to prevent and treat mental disorders. There are now consistent mechanistic, observational and interventional data to suggest diet quality may be a modifiable risk factor for mental illness. This review provides an overview of the nutritional psychiatry field. It includes a discussion of the neurobiological mechanisms likely modulated by diet, the use of dietary and nutraceutical interventions in mental disorders, and recommendations for further research. Potential biological pathways related to mental disorders include inflammation, oxidative stress, the gut microbiome, epigenetic modifications and neuroplasticity. Consistent epidemiological evidence, particularly for depression, suggests an association between measures of diet quality and mental health, across multiple populations and age groups; these do not appear to be explained by other demographic, lifestyle factors or reverse causality. Our recently published intervention trial provides preliminary clinical evidence that dietary interventions in clinically diagnosed populations are feasible and can provide significant clinical benefit. Furthermore, nutraceuticals including n-3 fatty acids, folate, S-adenosylmethionine, N-acetyl cysteine and probiotics, among others, are promising avenues for future research. Continued research is now required to investigate the efficacy of intervention studies in large cohorts and within clinically relevant populations, particularly in patients with schizophrenia, bipolar and anxiety disorders.',\n",
              "  'details': 'Effect of nutrition on mental health',\n",
              "  'source': 'https://pubmed.ncbi.nlm.nih.gov/28942748/'},\n",
              " {'abstract': \"Hepatitis C virus (HCV) and alcoholic liver disease (ALD), either alone or in combination, count for more than two thirds of all liver diseases in the Western world. There is no safe level of drinking in HCV-infected patients and the most effective goal for these patients is total abstinence. Baclofen, a GABA(B) receptor agonist, represents a promising pharmacotherapy for alcohol dependence (AD). Previously, we performed a randomized clinical trial (RCT), which demonstrated the safety and efficacy of baclofen in patients affected by AD and cirrhosis. The goal of this post-hoc analysis was to explore baclofen's effect in a subgroup of alcohol-dependent HCV-infected cirrhotic patients. Any patient with HCV infection was selected for this analysis. Among the 84 subjects randomized in the main trial, 24 alcohol-dependent cirrhotic patients had a HCV infection; 12 received baclofen 10mg t.i.d. and 12 received placebo for 12-weeks. With respect to the placebo group (3/12, 25.0%), a significantly higher number of patients who achieved and maintained total alcohol abstinence was found in the baclofen group (10/12, 83.3%; p=0.0123). Furthermore, in the baclofen group, compared to placebo, there was a significantly higher increase in albumin values from baseline (p=0.0132) and a trend toward a significant reduction in INR levels from baseline (p=0.0716). In conclusion, baclofen was safe and significantly more effective than placebo in promoting alcohol abstinence, and improving some Liver Function Tests (LFTs) (i.e. albumin, INR) in alcohol-dependent HCV-infected cirrhotic patients. Baclofen may represent a clinically relevant alcohol pharmacotherapy for these patients.\",\n",
              "  'details': 'Baclofen promotes alcohol abstinence in alcohol dependent cirrhotic patients with hepatitis C virus (HCV) infection',\n",
              "  'source': 'https://pubmed.ncbi.nlm.nih.gov/22244707/'}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 121
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 241
        },
        "id": "xiN8yduPfLe2",
        "outputId": "536630cf-d89c-4140-b105-84d1da6e7804"
      },
      "source": [
        "# See what our example abstracts look like\n",
        "abstracts = pd.DataFrame(example_abstracts)\n",
        "abstracts"
      ],
      "execution_count": 122,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>abstract</th>\n",
              "      <th>source</th>\n",
              "      <th>details</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>This RCT examined the efficacy of a manualized...</td>\n",
              "      <td>https://pubmed.ncbi.nlm.nih.gov/20232240/</td>\n",
              "      <td>RCT of a manualized social treatment for high-...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Postpartum depression (PPD) is the most preval...</td>\n",
              "      <td>https://pubmed.ncbi.nlm.nih.gov/28012571/</td>\n",
              "      <td>Formatting removed (can be used to compare mod...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Mental illness, including depression, anxiety ...</td>\n",
              "      <td>https://pubmed.ncbi.nlm.nih.gov/28942748/</td>\n",
              "      <td>Effect of nutrition on mental health</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Hepatitis C virus (HCV) and alcoholic liver di...</td>\n",
              "      <td>https://pubmed.ncbi.nlm.nih.gov/22244707/</td>\n",
              "      <td>Baclofen promotes alcohol abstinence in alcoho...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                            abstract  ...                                            details\n",
              "0  This RCT examined the efficacy of a manualized...  ...  RCT of a manualized social treatment for high-...\n",
              "1  Postpartum depression (PPD) is the most preval...  ...  Formatting removed (can be used to compare mod...\n",
              "2  Mental illness, including depression, anxiety ...  ...               Effect of nutrition on mental health\n",
              "3  Hepatitis C virus (HCV) and alcoholic liver di...  ...  Baclofen promotes alcohol abstinence in alcoho...\n",
              "\n",
              "[4 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 122
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "G277VTmZfiFG",
        "outputId": "de1738a4-d856-4068-f384-476240fc27bc"
      },
      "source": [
        "# Create sentencizer - Source: https://spacy.io/usage/linguistic-features#sbd \n",
        "from spacy.lang.en import English\n",
        "nlp = English() # setup English sentence parser\n",
        "sentencizer = nlp.create_pipe(\"sentencizer\") # create sentence splitting pipeline object\n",
        "nlp.add_pipe(sentencizer) # add sentence splitting pipeline object to sentence parser\n",
        "doc = nlp(example_abstracts[0][\"abstract\"]) # create \"doc\" of parsed sequences, change index for a different abstract\n",
        "abstract_lines = [str(sent) for sent in list(doc.sents)] # return detected sentences from doc in string type (not spaCy token type)\n",
        "abstract_lines"
      ],
      "execution_count": 123,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['This RCT examined the efficacy of a manualized social intervention for children with HFASDs.',\n",
              " 'Participants were randomly assigned to treatment or wait-list conditions.',\n",
              " 'Treatment included instruction and therapeutic activities targeting social skills, face-emotion recognition, interest expansion, and interpretation of non-literal language.',\n",
              " 'A response-cost program was applied to reduce problem behaviors and foster skills acquisition.',\n",
              " 'Significant treatment effects were found for five of seven primary outcome measures (parent ratings and direct child measures).',\n",
              " 'Secondary measures based on staff ratings (treatment group only) corroborated gains reported by parents.',\n",
              " 'High levels of parent, child and staff satisfaction were reported, along with high levels of treatment fidelity.',\n",
              " 'Standardized effect size estimates were primarily in the medium and large ranges and favored the treatment group.']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 123
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mFwOQJKHfkf8"
      },
      "source": [
        "Now our abstract has been split into sentences, how about we write some code to count line numbers as well as total lines.\n",
        "\n",
        "To do so, we can leverage some of the functionality of our preprocess_text_with_line_numbers() function.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WSARr-Zhfrqs",
        "outputId": "3bc0df73-66dc-4132-9ff6-ea5af57cecbf"
      },
      "source": [
        "# Get total number of lines\n",
        "total_lines_in_sample = len(abstract_lines)\n",
        "\n",
        "# Go through each line in abstract and create a list of dictionaries containing features for each line\n",
        "sample_lines = []\n",
        "for i, line in enumerate(abstract_lines):\n",
        "  sample_dict = {}\n",
        "  sample_dict[\"text\"] = str(line)\n",
        "  sample_dict[\"line_number\"] = i\n",
        "  sample_dict[\"total_lines\"] = total_lines_in_sample - 1\n",
        "  sample_lines.append(sample_dict)\n",
        "sample_lines"
      ],
      "execution_count": 124,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[{'line_number': 0,\n",
              "  'text': 'This RCT examined the efficacy of a manualized social intervention for children with HFASDs.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 1,\n",
              "  'text': 'Participants were randomly assigned to treatment or wait-list conditions.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 2,\n",
              "  'text': 'Treatment included instruction and therapeutic activities targeting social skills, face-emotion recognition, interest expansion, and interpretation of non-literal language.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 3,\n",
              "  'text': 'A response-cost program was applied to reduce problem behaviors and foster skills acquisition.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 4,\n",
              "  'text': 'Significant treatment effects were found for five of seven primary outcome measures (parent ratings and direct child measures).',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 5,\n",
              "  'text': 'Secondary measures based on staff ratings (treatment group only) corroborated gains reported by parents.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 6,\n",
              "  'text': 'High levels of parent, child and staff satisfaction were reported, along with high levels of treatment fidelity.',\n",
              "  'total_lines': 7},\n",
              " {'line_number': 7,\n",
              "  'text': 'Standardized effect size estimates were primarily in the medium and large ranges and favored the treatment group.',\n",
              "  'total_lines': 7}]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 124
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n4lh68-lft89",
        "outputId": "01ed5073-0581-47bd-dc26-b403bca6f6e3"
      },
      "source": [
        "# Get all line_number values from sample abstract\n",
        "test_abstract_line_numbers = [line[\"line_number\"] for line in sample_lines]\n",
        "# One-hot encode to same depth as training data, so model accepts right input shape\n",
        "test_abstract_line_numbers_one_hot = tf.one_hot(test_abstract_line_numbers, depth=15) \n",
        "test_abstract_line_numbers_one_hot"
      ],
      "execution_count": 125,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(8, 15), dtype=float32, numpy=\n",
              "array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0.]],\n",
              "      dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 125
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "58wtRUj8fwCZ",
        "outputId": "f88e4f04-2cd4-4fd9-ac71-3b5e0d79222f"
      },
      "source": [
        "# Get all total_lines values from sample abstract\n",
        "test_abstract_total_lines = [line[\"total_lines\"] for line in sample_lines]\n",
        "# One-hot encode to same depth as training data, so model accepts right input shape\n",
        "test_abstract_total_lines_one_hot = tf.one_hot(test_abstract_total_lines, depth=20)\n",
        "test_abstract_total_lines_one_hot"
      ],
      "execution_count": 126,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(8, 20), dtype=float32, numpy=\n",
              "array([[0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.],\n",
              "       [0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
              "        0., 0., 0., 0.]], dtype=float32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 126
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LqzcyoOkfyVK",
        "outputId": "4d957d5b-2640-481a-9f70-1235dee63ce4"
      },
      "source": [
        "# Split abstract lines into characters\n",
        "abstract_chars = [split_chars(sentence) for sentence in abstract_lines]\n",
        "abstract_chars"
      ],
      "execution_count": 127,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['T h i s   R C T   e x a m i n e d   t h e   e f f i c a c y   o f   a   m a n u a l i z e d   s o c i a l   i n t e r v e n t i o n   f o r   c h i l d r e n   w i t h   H F A S D s .',\n",
              " 'P a r t i c i p a n t s   w e r e   r a n d o m l y   a s s i g n e d   t o   t r e a t m e n t   o r   w a i t - l i s t   c o n d i t i o n s .',\n",
              " 'T r e a t m e n t   i n c l u d e d   i n s t r u c t i o n   a n d   t h e r a p e u t i c   a c t i v i t i e s   t a r g e t i n g   s o c i a l   s k i l l s ,   f a c e - e m o t i o n   r e c o g n i t i o n ,   i n t e r e s t   e x p a n s i o n ,   a n d   i n t e r p r e t a t i o n   o f   n o n - l i t e r a l   l a n g u a g e .',\n",
              " 'A   r e s p o n s e - c o s t   p r o g r a m   w a s   a p p l i e d   t o   r e d u c e   p r o b l e m   b e h a v i o r s   a n d   f o s t e r   s k i l l s   a c q u i s i t i o n .',\n",
              " 'S i g n i f i c a n t   t r e a t m e n t   e f f e c t s   w e r e   f o u n d   f o r   f i v e   o f   s e v e n   p r i m a r y   o u t c o m e   m e a s u r e s   ( p a r e n t   r a t i n g s   a n d   d i r e c t   c h i l d   m e a s u r e s ) .',\n",
              " 'S e c o n d a r y   m e a s u r e s   b a s e d   o n   s t a f f   r a t i n g s   ( t r e a t m e n t   g r o u p   o n l y )   c o r r o b o r a t e d   g a i n s   r e p o r t e d   b y   p a r e n t s .',\n",
              " 'H i g h   l e v e l s   o f   p a r e n t ,   c h i l d   a n d   s t a f f   s a t i s f a c t i o n   w e r e   r e p o r t e d ,   a l o n g   w i t h   h i g h   l e v e l s   o f   t r e a t m e n t   f i d e l i t y .',\n",
              " 'S t a n d a r d i z e d   e f f e c t   s i z e   e s t i m a t e s   w e r e   p r i m a r i l y   i n   t h e   m e d i u m   a n d   l a r g e   r a n g e s   a n d   f a v o r e d   t h e   t r e a t m e n t   g r o u p .']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 127
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "h0aAr32yf0mn",
        "outputId": "6e32b408-d3e0-4565-fad4-5124c4dbe124"
      },
      "source": [
        "# Make predictions on sample abstract features\n",
        "%%time\n",
        "test_abstract_pred_probs = loaded_model.predict(x=(test_abstract_line_numbers_one_hot,\n",
        "                                                   test_abstract_total_lines_one_hot,\n",
        "                                                   tf.constant(abstract_lines),\n",
        "                                                   tf.constant(abstract_chars)))\n",
        "test_abstract_pred_probs"
      ],
      "execution_count": 128,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 2.16 s, sys: 51 ms, total: 2.21 s\n",
            "Wall time: 2.19 s\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "M8oLi2bof26Y",
        "outputId": "a1fc6e43-2575-474f-97cb-ae052d576873"
      },
      "source": [
        "# Turn prediction probabilities into prediction classes\n",
        "test_abstract_preds = tf.argmax(test_abstract_pred_probs, axis=1)\n",
        "test_abstract_preds"
      ],
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(8,), dtype=int64, numpy=array([3, 2, 2, 2, 4, 2, 4, 4])>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 129
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PGuhUB0kf5GA"
      },
      "source": [
        "Now we've got the predicted sequence label for each line in our sample abstract, let's write some code to visualize each sentence with its predicted label.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w07KS7bnf7Dr",
        "outputId": "eaee2ee0-098e-4488-deaa-b46931ccbf25"
      },
      "source": [
        "# Turn prediction class integers into string class names\n",
        "test_abstract_pred_classes = [label_encoder.classes_[i] for i in test_abstract_preds]\n",
        "test_abstract_pred_classes"
      ],
      "execution_count": 130,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['OBJECTIVE',\n",
              " 'METHODS',\n",
              " 'METHODS',\n",
              " 'METHODS',\n",
              " 'RESULTS',\n",
              " 'METHODS',\n",
              " 'RESULTS',\n",
              " 'RESULTS']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 130
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L1GV3Z7yf81u",
        "outputId": "92506289-0578-47b2-cb63-0beb92e61e7d"
      },
      "source": [
        "# Visualize abstract lines and predicted sequence labels\n",
        "for i, line in enumerate(abstract_lines):\n",
        "  print(f\"{test_abstract_pred_classes[i]}: {line}\")"
      ],
      "execution_count": 131,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "OBJECTIVE: This RCT examined the efficacy of a manualized social intervention for children with HFASDs.\n",
            "METHODS: Participants were randomly assigned to treatment or wait-list conditions.\n",
            "METHODS: Treatment included instruction and therapeutic activities targeting social skills, face-emotion recognition, interest expansion, and interpretation of non-literal language.\n",
            "METHODS: A response-cost program was applied to reduce problem behaviors and foster skills acquisition.\n",
            "RESULTS: Significant treatment effects were found for five of seven primary outcome measures (parent ratings and direct child measures).\n",
            "METHODS: Secondary measures based on staff ratings (treatment group only) corroborated gains reported by parents.\n",
            "RESULTS: High levels of parent, child and staff satisfaction were reported, along with high levels of treatment fidelity.\n",
            "RESULTS: Standardized effect size estimates were primarily in the medium and large ranges and favored the treatment group.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UVG96O4Pf-gt"
      },
      "source": [
        "Nice! Isn't that much easier to read? I mean, it looks like our model's predictions could be improved, but how cool is that?\n",
        "\n",
        "Imagine implementing our model to the backend of the PubMed website to format any unstructured RCT abstract on the site.\n",
        "\n",
        "Or there could even be a browser extension, called \"SkimLit\" which would add structure (powered by our model) to any unstructured RCT abtract.\n",
        "\n",
        "And if showed your medical researcher friend, and they thought the predictions weren't up to standard, there could be a button saying \"is this label correct?... if not, what should it be?\". That way the dataset, along with our model's future predictions, could be improved over time.\n",
        "\n",
        "Of course, there are many more ways we could go to improve the model, the usuability, the preprocessing functionality (e.g. functionizing our sample abstract preprocessing pipeline) but I'll leave these for the exercises/extensions.\n",
        "\n"
      ]
    }
  ]
}