processing_sequences.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "18Xm7z99z7rt"
   },
   "source": [
    "# <center>Modeling Sequences</center>\n",
    "\n",
    "This Jupyter notebook (which is part of the hands-on tutorial on pythonic data science and scinentic computing) extracted materials mostly from [16-17] and their jupyter notebooks, plus the TensorFlow documentation [18]. In addition, I also used other materials from different sources as cited in the following to briefly explain the theory, while focusing more on implementation.\n",
    "\n",
    "## Basics\n",
    "\n",
    "- Sequences data are typically variable-sized and not *independent and identically distributed* (***i.i.d.***), such as\n",
    "    - text data,\n",
    "    - time series,\n",
    "    - nested sequences (e.g., videos), etc.\n",
    "- Types of time series\n",
    "    - by the number of values per time step:\n",
    "        - ***Univariate***: e.g., a series with ***dimensionality***=1 and the shape [batch size, time steps, 1].\n",
    "        - ***Multivariate***: e.g., measurement of revenue, debt.\n",
    "    - by their tasks:\n",
    "        - ***forecasting***\n",
    "        - ***postdicting*** (filling the missing values by local prediction)\n",
    "- When a sequence model uses the previous time-steps to predict the current state, it has the *autoregressive* property:\n",
    "    - Given the input sequence data $X = x_{1:N} = \\left \\{  x_1, x_2, ..., x_N\\right \\}$, the output sequence $Y = y_{1:M} = \\left \\{  y_1, y_2, ..., y_M\\right \\}$ can be generated by the *chain rule of probability* that defines the output conditional probabilities $p(\\mathbf{y}|\\mathbf{x})$ using $\\mathbf{y}_{<t}$ (i.e., output sequence data before the time step $t$):\n",
    "\\begin{equation}\n",
    "p(\\mathbf{y}|\\mathbf{x})=\\prod_{i=1}^{T}p\\left ( y_{t}|\\mathbf{y}_{<t}, \\mathbf{x} \\right ),\n",
    "\\end{equation}\n",
    "    - This **autoregressive model (AR)** is a sequential function as each output sequence generation requires generation of the previous outputs ($\\mathbf{y}_{<t}$), making it slow comparing to parralized non-AR models; but non-AR models may have sub-optimal performance [19].\n",
    "    - The AR forecast problems are solved in statistics by techinques such as the ***weighted moving average*** or ***autoregressive integrated moving average (ARIMA)***, which may require us to first remove or handle components of systematic (that can be modeled) and non-systematic (e.g., noises) patterns, e.g., by decomposion of them. The systematic components include\n",
    "        - ***Level***: the average value in the series.\n",
    "        - ***growth***: the average increase or decrease observed in a period,\n",
    "        - ***trends***: e.g., a continuous increase in the number of clients monthly;\n",
    "        - ***seasonality***: e.g., the numbers that may change in each season (which can be handled by ***differencing*** that considers them by theirs differences at different days);\n",
    "        - ***cyclicity***: similar to *seasonality* but without a fixed frequency.\n",
    "    - Patterns like trends and seasonality, make the series non-starationary (time-dependent) and therefore more complicated for most statistical methods. Thus, they are usually converted to stationary data by techniques like differencing, complicating their implimentation [15].\n",
    "    - Alternatively, such treatments in data with machine learning might be useful but not required. Common techiniques include:\n",
    "        - *Recurrent AR* by **recurrent neural network (RNN)**, which approximates a function AR models by a recursive function that while sequentially computes the outputs but typically determine what information should be used for the next step as its *memory*. Its major challenges are:\n",
    "            - ***Unstable gradients***, which can be alleviated by e.g., **recurrent dropout** and **layer normalization**.\n",
    "            - ***limited short-term memory***, which can be extended by **LSTM** and **GRU cells**.\n",
    "        - *Deep AR* by **convolusional neural networks (CNN)**, which uses feed-forward neural networks (FFNN) with easier training due to the possible application of non-saturating activation function and batch normalization, that handle very long sequences by considering hierarchical structure of sequences [20].\n",
    "        - **Transformers**, which avoids recurrence (to redcuce repititions) and convolusions (to reduce very deep architectures) for long sequences, allow very efficient and parrallel implementation, mostly by different attension mechanisms (i.e., learnable methods that dynamically favor different parts of input data) [21].\n",
    "\n",
    "## Recurrent neurons and layers\n",
    "### Intuition\n",
    "- As it has a **memory cell** (i.e., a ***recurrent neuron*** or ***layer***), it is not a FFNN.\n",
    "- At time step $t$, i.e., a ***frame***, it can take pattern of typically previous 10 steps (or 100 steps for more powerful models) using the state of the previous step $\\mathbf{h}_{\\left(t\\right)}$ (stands for “*hidden*”). Figure below shows the cell and its *unrolled* network through time:\n",
    "\n",
    "<img src=\"img/rnn1.jpg\" width=\"300px\">\n",
    "<center>The cell’s hidden state.</center>\n",
    "\n",
    "In basic (not complex) cells, $\\mathbf{h}_{\\left(t\\right)}=\\mathbf{y}_{\\left(t\\right)}$:\n",
    "- The input connections has weight vectors $\\mathbf{w}_x$ and $\\mathbf{w}_y$ and the ReLU and tanh can be used.\n",
    "- For a vectorized layer of recurrent neurons with a minibatch, we have:\n",
    "\n",
    "$\\mathbf{Y}_{\\left(t\\right)}=\\emptyset\\left(\\left[\\begin{matrix}\\mathbf{X}_{\\left(t\\right)}&\\mathbf{Y}_{\\left(t-1\\right)}\\\\\\end{matrix}\\right]\\left[\\begin{matrix}\\mathbf{W}_x\\\\\\mathbf{W}_y\\\\\\end{matrix}\\right]+\\mathbf{b}\\right)$\n",
    "- where:\n",
    "  - $\\mathbf{Y}_{\\left(t\\right)}$ is of $m\\times n_\\text{neurons}$ dimension ($m$ and $n_\\text{neurons}$ are # mini-batch’s instances and neurons, respectively).\n",
    "  - $\\mathbf{X}_{\\left(t\\right)}$ is of $m\\times n_\\text{inputs}$ dimension.\n",
    "  - $\\mathbf{W}_x$ is of $n_\\text{inputs}\\times n_\\text{neurons}$ dimension.\n",
    "  - $\\mathbf{W}_y$ is of $n_\\text{neurons}\\times n_\\text{neurons}$ dimension.\n",
    "  - $\\mathbf{b}$ is of $n_\\text{neurons}$ dimension.\n",
    "  - $\\emptyset$ is the ReLU or tanh fuctions.\n",
    "\n",
    "### Input and output sequences\n",
    "E.g. of different types of RNNs:\n",
    "- ***Sequence-to-sequence***: stock prices time series.\n",
    "- ***Sequence-to-vector***: sentiment score.\n",
    "- ***Vector-to-sequence***: caption generation by an input image.\n",
    "- ***Encoder-decoder***: machine translation\n",
    "\n",
    "### Backpropagation\n",
    "After unrolling, we use ***backpropagation through time (BPTT)***.\n",
    "\n",
    "<img src=\"img/rnn2.jpg\" width=\"300px\">\n",
    "<center>BPTT.</center>\n",
    "\n",
    "\n",
    "### Baseline metrics\n",
    "For comparison, we can start with some baseline models, e.g.:\n",
    "- ***Naive forecasting***: prediction by the last values.\n",
    "- Fully connected NNs: e.g., a simple regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 166725,
     "status": "ok",
     "timestamp": 1671395724329,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "q7uaYZOTFpaU",
    "outputId": "32699eb2-3f1e-4558-aa85-62d876d9e8c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading package lists... Done\n",
      "Building dependency tree       \n",
      "Reading state information... Done\n",
      "The following package was automatically installed and is no longer required:\n",
      "  libnvidia-common-460\n",
      "Use 'apt autoremove' to remove it.\n",
      "The following packages will be REMOVED:\n",
      "  libcudnn8-dev\n",
      "The following held packages will be changed:\n",
      "  libcudnn8\n",
      "The following packages will be DOWNGRADED:\n",
      "  libcudnn8\n",
      "0 upgraded, 0 newly installed, 1 downgraded, 1 to remove and 18 not upgraded.\n",
      "Need to get 430 MB of archives.\n",
      "After this operation, 1,392 MB disk space will be freed.\n",
      "Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  libcudnn8 8.1.0.77-1+cuda11.2 [430 MB]\n",
      "Fetched 430 MB in 7s (62.0 MB/s)\n",
      "(Reading database ... 124016 files and directories currently installed.)\n",
      "Removing libcudnn8-dev (8.1.1.33-1+cuda11.2) ...\n",
      "update-alternatives: removing manually selected alternative - switching libcudnn to auto mode\n",
      "\u001b[1mdpkg:\u001b[0m \u001b[1;33mwarning:\u001b[0m downgrading libcudnn8 from 8.1.1.33-1+cuda11.2 to 8.1.0.77-1+cuda11.2\n",
      "(Reading database ... 123993 files and directories currently installed.)\n",
      "Preparing to unpack .../libcudnn8_8.1.0.77-1+cuda11.2_amd64.deb ...\n",
      "Unpacking libcudnn8 (8.1.0.77-1+cuda11.2) over (8.1.1.33-1+cuda11.2) ...\n",
      "Setting up libcudnn8 (8.1.0.77-1+cuda11.2) ...\n",
      "\u001b[33mWARNING: Skipping tensorflow-text as it is not installed.\u001b[0m\n",
      "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
      "Collecting tensorflow-text\n",
      "  Downloading tensorflow_text-2.11.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.8 MB)\n",
      "\u001b[K     |████████████████████████████████| 5.8 MB 29.5 MB/s \n",
      "\u001b[?25hCollecting tensorflow\n",
      "  Downloading tensorflow-2.11.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (588.3 MB)\n",
      "\u001b[K     |████████████████████████████████| 588.3 MB 23 kB/s \n",
      "\u001b[?25hRequirement already satisfied: tensorflow-hub>=0.8.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow-text) (0.12.0)\n",
      "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.6.3)\n",
      "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.2.0)\n",
      "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.21.6)\n",
      "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.3.0)\n",
      "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.1.0)\n",
      "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (14.0.6)\n",
      "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (2.1.1)\n",
      "Collecting tensorflow-estimator<2.12,>=2.11.0\n",
      "  Downloading tensorflow_estimator-2.11.0-py2.py3-none-any.whl (439 kB)\n",
      "\u001b[K     |████████████████████████████████| 439 kB 71.8 MB/s \n",
      "\u001b[?25hRequirement already satisfied: setuptools in /usr/local/lib/python3.8/dist-packages (from tensorflow) (57.4.0)\n",
      "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.15.0)\n",
      "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (4.4.0)\n",
      "Collecting flatbuffers>=2.0\n",
      "  Downloading flatbuffers-22.12.6-py2.py3-none-any.whl (26 kB)\n",
      "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.28.0)\n",
      "Requirement already satisfied: wrapt>=1.11.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.14.1)\n",
      "Requirement already satisfied: gast<=0.4.0,>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (0.4.0)\n",
      "Requirement already satisfied: protobuf<3.20,>=3.9.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.19.6)\n",
      "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (1.51.1)\n",
      "Collecting keras<2.12,>=2.11.0\n",
      "  Downloading keras-2.11.0-py2.py3-none-any.whl (1.7 MB)\n",
      "\u001b[K     |████████████████████████████████| 1.7 MB 50.8 MB/s \n",
      "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.8/dist-packages (from tensorflow) (21.3)\n",
      "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow) (3.3.0)\n",
      "Collecting tensorboard<2.12,>=2.11\n",
      "  Downloading tensorboard-2.11.0-py3-none-any.whl (6.0 MB)\n",
      "\u001b[K     |████████████████████████████████| 6.0 MB 59.7 MB/s \n",
      "\u001b[?25hRequirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.8/dist-packages (from astunparse>=1.6.0->tensorflow) (0.38.4)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (0.4.6)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (0.6.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (2.23.0)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (2.15.0)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (1.8.1)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (1.0.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.12,>=2.11->tensorflow) (3.4.1)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (4.9)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (5.2.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (0.2.8)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow) (1.3.1)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.8/dist-packages (from markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow) (5.1.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.12,>=2.11->tensorflow) (3.11.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.12,>=2.11->tensorflow) (0.4.8)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (3.0.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (2022.12.7)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (2.10)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.12,>=2.11->tensorflow) (1.24.3)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.12,>=2.11->tensorflow) (3.2.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.8/dist-packages (from packaging->tensorflow) (3.0.9)\n",
      "Installing collected packages: tensorflow-estimator, tensorboard, keras, flatbuffers, tensorflow, tensorflow-text\n",
      "  Attempting uninstall: tensorboard\n",
      "    Found existing installation: tensorboard 2.9.1\n",
      "    Uninstalling tensorboard-2.9.1:\n",
      "      Successfully uninstalled tensorboard-2.9.1\n",
      "  Attempting uninstall: flatbuffers\n",
      "    Found existing installation: flatbuffers 1.12\n",
      "    Uninstalling flatbuffers-1.12:\n",
      "      Successfully uninstalled flatbuffers-1.12\n",
      "Successfully installed flatbuffers-22.12.6 keras-2.11.0 tensorboard-2.11.0 tensorflow-2.11.0 tensorflow-estimator-2.11.0 tensorflow-text-2.11.0\n",
      "\u001b[K     |████████████████████████████████| 1.1 MB 33.3 MB/s \n",
      "\u001b[K     |████████████████████████████████| 5.8 MB 29.4 MB/s \n",
      "\u001b[K     |████████████████████████████████| 7.6 MB 53.1 MB/s \n",
      "\u001b[K     |████████████████████████████████| 182 kB 62.0 MB/s \n",
      "\u001b[?25hColab only includes TensorFlow 2.x; %tensorflow_version has no effect.\n"
     ]
    }
   ],
   "source": [
    "# initialization\n",
    "\n",
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "# Is this notebook running on Colab or Kaggle?\n",
    "IS_COLAB = \"google.colab\" in sys.modules\n",
    "IS_KAGGLE = \"kaggle_secrets\" in sys.modules\n",
    "\n",
    "if IS_COLAB:\n",
    "    # Install the most re version of TensorFlow to use the improved\n",
    "    # masking support for `tf.keras.layers.MultiHeadAttention`.\n",
    "    !apt install --allow-change-held-packages libcudnn8=8.1.0.77-1+cuda11.2\n",
    "    !pip uninstall -y -q tensorflow keras tensorflow-estimator tensorflow-text\n",
    "    !pip install tensorflow-text tensorflow\n",
    "    !pip install -q tensorflow_datasets\n",
    "    !pip install -q -U tensorflow-addons\n",
    "    !pip install -q -U transformers\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "    IS_COLAB = True\n",
    "except Exception:\n",
    "    IS_COLAB = False\n",
    "\n",
    "# TensorFlow ≥2.0 is required\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "assert tf.__version__ >= \"2.0\"\n",
    "\n",
    "if not tf.config.list_physical_devices('GPU'):\n",
    "    print(\"No GPU was detected. LSTMs and CNNs can be very slow without a GPU.\")\n",
    "    if IS_COLAB:\n",
    "        print(\"Go to Runtime > Change runtime and select a GPU hardware accelerator.\")\n",
    "\n",
    "import tensorflow_text\n",
    "import tensorflow_datasets as tfds\n",
    "import numpy as np\n",
    "import os\n",
    "from pathlib import Path\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"rnn\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "C7UzyGMxFpaW"
   },
   "outputs": [],
   "source": [
    "# Dataset generation:\n",
    "\n",
    "def generate_time_series(batch_size, n_steps):\n",
    "    freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1)\n",
    "    time = np.linspace(0, 1, n_steps)\n",
    "    series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10))  #   wave 1\n",
    "    series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20)) # + wave 2\n",
    "    series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5)   # + noise\n",
    "    return series[..., np.newaxis].astype(np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "71tHi_lWFpaW"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "n_steps = 50\n",
    "series = generate_time_series(10000, n_steps + 1)\n",
    "X_train, y_train = series[:7000, :n_steps], series[:7000, -1]\n",
    "X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1]\n",
    "X_test, y_test = series[9000:, :n_steps], series[9000:, -1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "moqf0s9EFpaW",
    "outputId": "a2dc2a55-be7a-4e06-d740-2ccdfef44830"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((7000, 50, 1), (7000, 1))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "45zRk_upFpaX",
    "outputId": "b1624029-8198-45af-f109-ac3e8acb8941"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving figure time_series_plot\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1sAAAEYCAYAAACqb+zuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAACGN0lEQVR4nO3dd3yb1b348c+RvOK9ncSJ7TjDWWQ5IWYnzEJLodAFFOgAfhe67m17S3t76b5dt+3tYpRCC6VAS1soEPZIgJA4w9nLe8R2vOUhb0nn94ckx3G8rfFI+r5fL70SS48enyPJR8/3jO9RWmuEEEIIIYQQQniWyd8FEEIIIYQQQohgJMGWEEIIIYQQQniBBFtCCCGEEEII4QUSbAkhhBBCCCGEF0iwJYQQQgghhBBeIMGWEEIIIYQQQniBBFtCCCGEEEII4QWGDLaUUl9QSu1VSvUrpR6b4Nj/UEo1KKU6lFJ/VEpFDnssWSn1nFKqWylVrZS62euFF0IIIYQQQggMGmwB9cAPgT+Od5BS6irgG8BlQA6QC3xv2CH3AwNABnAL8KBSaoUXyiuEEEIIIYQQZ1Baa3+XYUxKqR8C87TWnx7j8aeAKq31f7l+vgx4Ums9WykVA1iAlVrrEtfjTwB1Wutv+KQCQgghhBBCiJAV5u8CzNAK4PlhPx8EMpRSKUAWYHcHWsMev2S0Eyml7gLuAoiKisrPysryTol9yOFwYDIZdfBycqQOxhCsdSgpKWnRWqdN5vnB1kYEw3sKwVEPqYMxSBtxpmB9TwNRMNQjWOswqTZCa23YG86phI+N83g58IFhP4cDGueUwouAhhHH3wlsm+j3LlmyRAeDrVu3+rsIMyZ1MIZgrQOwV0+jbQqGNiIY3lOtg6MeUgdjkDbiTMH6ngaiYKhHsNZhMm1EYIeYYAXih/3s/n/XKI+5H+/yQbmEEEIIIYQQIS7Qg62jwOphP68GGrXWrUAJEKaUWjzi8aM+LJ8QQgghhBAiRBky2FJKhSmlogAzYFZKRSmlRltf9mfgc0qp5UqpJOC/gccAtNbdwLPA95VSMUqpC4DrgCd8UgkhhBBCCCFESDNksIUzaOrFmdb9U67//7dSKkspZVVKZQForV8FfgZsBapdt+8MO889wCygCXgauFtrLSNbQgghhBBCCK8zZDZCrfV3ge+O8XDsiGN/CfxyjPO0Add7sGhCCCGEEEIIMSlGHdkSQgghhBBCiIAmwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIFhgy2lVLJS6jmlVLdSqlopdfMYxz2klLIOu/UrpbqGPb5NKdU37PFi39VCCCGEEEIIEarC/F2AcdwPDAAZwBrgJaXUQa310eEHaa3/Dfg3989KqccAx4hzfUFr/YhXSyuEEEIIIYQQwxhyZEspFQPcCNyntbZqrbcDLwC3TvJ5j3u/lEIIIYQQQhjP7spWfv5aMUXVFn8XJeQprbW/y3AWpdRaYIfWetaw+74GXKK1vnac590GfBdYqF0VU0ptA1YACigGvqW13jbKc+8C7gJIS0vLf+aZZzxVHb+xWq3Exsb6uxgzMrIOZRY7J9rsLE02syjJ7MeSTV4wvg+BaLQ6bN68uUhrvX4yzw+2NiIY3lMIjnpIHYxB2ogzBet7GoimWo8yi50f7+7DriHcBPduiPL7NVMwvBfTbSOMOo0wFugYcV8HEDfB824H/qzPjCDvBY7hnJL4SeBFpdQarXX58CdqrR8GHgbIy8vTmzZtmn7pDWLbtm0Eej2G12FXRSs/eX0XDq2JCLPz5B0F5Gcn+beAkxBs70Ogmmkdgq2NCIb3FIKjHt6sQ1G1hcKKVgpyU7zaXsr7IG2EEQVDHWDq9Ti6tQy7dqYosDmgPzGbTZsWeal0kxNo78XPfgYbNsDmzafvG1mHrVsB5mZMdC5DTiMErED8iPviga5RjgVAKTUfuAT48/D7tda7tNZdWut+rfXjwPvANR4ur/CBx3ZUYXNoHBoGbQ4KK1r9XSQh/Kqoqo37t5bJNBExqqJqCzf/oZCfv1bMLY8UyudEiBCxdPawsQkFBbkp/itMgNqwAT7+cXdAdbatW52PQ3fPROcyarBVAoQppRYPu281cHSM4wFuwzn1sGKCc2ucUwpFgKlq6R76f3iYSRoPEdIefrecGx/aKRfSYkzvl7XQb3OggQHpoBIiZFh6BgFYMScerSE+yqgT2Yxr82Z45pnRAy53oOWcKdwx5kCQmyGDLa11N/As8H2lVIxS6gLgOuCJcZ52G/DY8DuUUolKqauUUlFKqTCl1C3AxcBrXiq68JKKZivHG7pYNsfZW/PFzYsCYgqhEJ5md2h+/loxP3r5BODsPeoflAtpcbYWa//Q/5VS0kElRIjYVtxEelwkj332XMJMin/sq/V3kQLSaAHX8EBr+BTD8Rgy2HK5B5gFNAFPA3drrY8qpbJc+2VluQ9USp0HzAP+PuIc4cAPgWagBfgicL3WWvbaCjB/L6rFbFL88dMbmJc0i3dLW/xdJCF8qqjaws9ePcH192/nd1vLuHxZOlFhziZcA0nREf4toDCUps4+/llUS352EqsyE1AK5ifPmviJQoiAZrM7eLekmU15aaTFRbIpL53n9tVhs4/cFUlMxvCA649/zJlyoAUGDra01m1a6+u11jFa6yyt9VOu+2u01rFa65phx+50Hdc14hzNWusNWus4rXWi1rpAa/2Gr+siZsZmd/DPolo256UxJ2EWtxZks6uyjeOnOv1dNCF8oqjawk0PF/LAtnIO13Xy+c0LeeT2DTx5ZwFfvHQRs+Mj+eUbJdS39/q7qMIgfvLKCQbtml98bDW/vmktDofmj9ur/F0sIYSX7T/ZTmefjU156QB8NH8eTV39vFcmndTTtXkz3H03PPFEDnffPbVACwwcbAnh9m5pM01d/Xxs/XwAPrFhPlHhJv68s8q/BRPCR3aWtzDg6pU0KYiOcM6/z89O4qtX5vHE5zbSN2jnzj/vpWfA5s+iCgPYW9XGs/vruOOiBeSkxrAgNYarz5nDXwqr6egd9HfxhBBetPVEE2aT4sLFqQBcujSdpOhw/lEkUwmna+tWePBBuPXWKh58cOykGWORYEsY3t/2nCQ1NoJLlzp7aRKjI/jI2kye219He8+An0snhPdlxEcBzkArYpTkMIsz4vjtTWs5dqqTOx7by/1bSyVhRoiyOzTffv4ocxKi+MKlp1M937NpIdZ+G38prPZj6YQY37biprM24i2qtkjW1SnYVtzM+uwk4qPCAed3xnVrMnnjaCMdPdLZMlXD12h99rNVYybNGI8EW8LQOvs1bx1v4oZ18wg3n/643n5+Dn2DDv6256QfSyeEb7hHI+66OHfM/eU2L03n1oJsdlS08vPXSiRDYYh6encNx0518l/XLBsaAQVYMTeBTXlpPLq9kt4Bux9LKMToCita+PSf9vC7rWXc+OAOrv7Vu9zx+B4+8fud/OJ1ybo6GQ0dfRw71Tk0hdDto/nzGLA7eOFQvZ9KFphGS4YxXpbCsUiwJQzt/XobNofmY/nzzrh/6ex4Ni5I5onCauwOPcazhQgOuyrbyE6J5htXLxs3C2dGfCTgTJghe9GFnm3FTfzwpWOsmBvHh1bNOevxezYtoq17gL/tqRnl2UL411O7TneeKqCr38aOslbZX3MK3ilpAmDz0rQz7l8xN56ls+NkKuEkFVVb+PIv67jho45Rk2EMD7ggIW60cwwnwZYwLK0179UNsjYrkcUZZ3+WP31+DrWWXt463uiH0gnhGw6HZk9VGxsXJE94bEFuKibXLoKyF11oebe4mc8+toe+QQdlTd3sq2k/65hzFySzPjuJP7xXyaBkJhMGU2tx7g1rVhAZbuLXn1zLE3dsHNoYVdq0iW0rbmZOQhR5I66ZlFJ8NH8eB0+2U9o44bZQIa2o2sLHHtrBEy92EnfNHuJzRx9NdQdcEBM90Tkl2BKGdeBkO/VWzcddiTFGumJ5BqkxEfzPS8dlaoEIWsWNXbT3DLJxwcQXGfnZSXzm/BwAfvWJNbIXXYiotfTw788cwD3Ib7OPPQJwz+aF1LX38qWn90u7KQyjsbOPAyfbuWFtJl+5Mm9ounR+dhLnZCYwOyFqzCnUwmnQ7uC90hY25aWhlDrr8evXZmJW8K1/HZa//XG8faIRh4b4jRWEz2sZdzTVOeJVP2GPvwRbwrAe2FqGWcG8pNH3hjlY20F77yDVbT3c9LDM5RbByd3Qb8ydeGQL4MZ8Z+dEd7+sywkFR+o6+MgDO+gbtBNhNmFW448AJESFo4BXjjTIGhhhGM/tr8Oh4QuXLuLzmxedEVStnJdA/6BdAq0J7K2yYO23nbVey626tQcN7K60yN/+ONx7VpomaEunImziQ4TwvZ3lLbxx3Dn3+M4/7x21R6uwohWHdnblDtgdbDlYL42xCDq7KtrITJzFvKQJZyoAsHR2HAmzwtlV2cqNI9Y6iuBRVG3h6d01bDlYT3JMBP/6/AV09dkorGilIDdlzLawsLJt6P/uNTDSbgp/0lrz970nWZ+dRG5a7FmPZydHY+kZpLNvcCjDnjjbtpImws2KCxaljvp4YUUrrksm+dsfhzuB0BcuXcQlS9I98hpJsCUM6fkDpzPmjNUoFOSmEBFmYsDmwKHhxUP13L1pIemuNNlCBDqtNbur2tg8Rk/laEwmxYacZHYNu6gWwcW9yfWA3YECvn/dSpa41mhMdGFQkJtCmFkxaNeYzbIGRvjf/pPtlDd389Mbc0d9PDvF2dFU09rDyswEXxYtoGw70cyGnGRiI0e/tJe//ck5XNfBgtQYvnJFnsfOKdMIhSG507ybGHsYNz87iSfvKOCrV+bxvx9dRc+Anc8+vofuftnUNVS8faLxrD1Zgklpk5W27oFJTyF0K8hNprq1h4aOPi+VTPhTYUXrGZtcF09hwXt+dhIP3JwPwC0bs6RnW/jd3/fWEhVu4ppzzs6gCZCVHAM4p8GJ0b12tIHixi4Wp589MuiWn53Erz+xFoDPXZAjf/tjOFLXwTkeDuplZEsYUnNXP7PjI7kww8FNl28Ys1FwL6AFSImN4I7H93L7H3ezKS+N8xamSmMSRIqqLRRWtLJqXgKnOvr4y85qDtV1APDQO+U8dedGzp1EEolAssu1XqtgivVyJ9PYVdnKdWsyPV4u4V8FuSkonCn+p7Om4IoVGaTHRWLplk3hhX/1DdrZcrCea1bOIW6MKYJZrpGt6rZuXxYtYBRVW/jCU/sA+Ouek3x4TeaY1z5XrZxNVLiJQbtsmTOaFms/9R19EmyJ0HDgZDsbc1P40OyOSQdMly7N4HMXLuAP71Wyt9pCVHiZZC8KEkXVFm7+QyH9ttPpqhNmhQ9dcNocmi8+vZ9Hb98QVNNMCivbmJMQxfzk0ZPEjGX53HjiIsMorGiTYCsILUiNQQMXLU7l3y9fMq02btW8RA7Vdni+cAKA1j5NUbVFvn8m8NrRBrr6bXx0/djrS2Mjw0iNjaBGRrZGVVjROhQ8uTORjvW5M5sUC9NiKW2y+rKIAeOwqwPX09cRMo1QGE5DRx8NnX2smZ845ecmurLIAAzIBohBo7CidSjQUsCnNmbxx9vXExnuyr5mVvTbHHz4d9v50tP7+NWbJQE/tVBrza4K5/5ao6XxHY/ZpFifk8SuSvn8B6PdrvV4X75s8bQv5lfPS6CipZuO3kFPFk24dA1obvmDZHybyN/31jIvadaEo/dZydEyjXAM7pFumNxI96L0WMok2BrVkVp3sBXv0fNKsCUM58BJ55fTdIKtgtwUIlzrvcJMShaABomCYRv6Roab+Mi6eeTnJPPkHQV85co8/nrXebzzn5u5bFkGLxw8xa/eLA341LYVLd20WPun/RnemJtCRXM3TV2ybivY7KpsJTLMxDnzpt/7usrVvh6pk9EtbxkYZ78zAXXtvbxf3sKN6+ZhMo3foZSdEkNNmwRbo1k2x5kc57zc5EnN5lmcHktde6+sbx/F4boOclNjxpzSOl0SbAnD2X+ynQizieVzp96zkJ+dxB9ucy7+vulcWfwdLNLinBkmr1ieccaXSX520tCeLAmzwlkzPxH3d/ZggI9snt5fa5rBlitA3VMZuAGnGN3uyjbWZSURGWae9jlWuabJHKxt91CpxEgmJR1+4/ndW6Vo7dyuYiJZydHUd/TSb5P9A0c6UteJBu64KHdS1zyL0p2vd3mzjG6NdLiuwytLESTYEoZzoKadZXPjp30hcUleOqmxEfQNOiY+WASEohrntKn/mGB9ins7gIk2dg0EuyraSI+LJCdlcvtrjbQyM4HoCLNMJQwyHb2DHDvVOeUMlSMlxUSQlRzNoZMysuUNJmB9TpJ0+I2hqKqNv+45CcB/PHNgwlkI2SnRaA0n23p9UbyAcvBkO+BchzkZi1wZC0sbJdgarsXazykvJMcASZAhDMbu0Byu6+Dj6+fP6Dy5abHSaxNE9lW3ExNhJm+CHlD3dgATbexqdFprdlW2sjE3ZcrrtdzCzSbys5PYVSH7bQWTvVVtaH064+RMrJqXwL4AnmprZLPClAQG4/jXgXrc+fAms8Hu0F5bbd1DwYJwOljbTmbiLNLiIid1fHZKNOFmJUkyRnAnx5jJ9OyxyMiWMJSSxi56BuzTWq813KL0WMqarWgt6U2DQVG1hTVZiZgnmNcPZ04tDFTVrT00dvYPTQWcroLcFIobu2iTFN9BY1dlGxFmE2uzEmd8rtXzEqnv6KO5q3/mBRNniDA71yS1WOW1HY2lx9kmTXYWguy1NbaDte1TumYKN5tYkBojSTJGOOxKjrFiGktYJmLYYEsplayUek4p1a2UqlZK3TzGcZ9WStmVUtZht01TPY8whgOu4fAZB1tpsbT3DNIqF5kBr7vfxomGTvKzAjd4mqpn9tYAEB81s8kH57qCNXf2OhH4dlW2sXp+AlHh01+v5bbK1YN7uK59xucSZ4oMc3YMHZI1cWexOzR7qtpYn53EV67Mm1RSh9TYCKIjzBJsjdBq7edkWy+r509tNMaZkXDym6GHAm8lxwADB1vA/cAAkAHcAjyolFoxxrE7tdaxw27bpnke4WcHatpJig4fmjIwXe5pBtJzE/gOnmzHoWFtAI9UTUVRtYXfv1MJwNf/eWhGGRVXzUsgMswk67aChLXfxpG6Do9MIQTnuj6TgoOybsvjIkyg5LUd1e7KNho7+7nt/JxJz0JQSpGVHC0ZCUdw75W3epLrtdwWpcdR09ZD36AkHHE7UtfhlSmEYNBgSykVA9wI3Ke1tmqttwMvALf64zzCdw6cbGf1/MRpr1NxWyjBVtDYV+MMNtbND41gq7CiFbtr+utMMypGhplZlyXrtoJFUbUFu0PPODmGW0xkGIvSY2X0xQtMyjnDQl7bs71wsJ5Z4WYuX5Y+peflpMRQ3drtpVIFpgMn2zGpqW/Cuzg9FoeGyhZ5PQGau7yXHAOMmyBjCWDXWpcMu+8gcMkYx69VSrUAbcATwI+11rapnEcpdRdwF0BaWhrbtm2bcSX8zWq1BlQ9em2aksYelsX1DZV7unXQWhNphnf2n2BeX6VnCzpFgfY+jMafdXh9fx9zYxT7d78/o/PMtA6+aiPsLc69TxTO9QyR7dVs21Y77fNlmAbYeWqQL/3hdValmlmU5Jx+FgyfSwiOeky2Dv8oGcCsoKf6CNvqZtYh5ZYe1s/eSitbt26dUSdXKL0PYxnZRqwP6/PIa+sv3nhPbQ7NC/t7WJ1qZveO7VN6ruoZoLp1kLe3bsU0ydczGD6XMHY9th50fj/u2Tm117K9y5mt+fltu2mc45tQwMjvxcFm5/eurbmSbdtqxjxu2nXQWhvuBlwENIy4705g2yjH5gILcI7SnQMcA7451fMMvy1ZskQHg61bt/q7CFPyflmzzr53i956onHovpnU4UO/eU9/6pFCD5RsZgLtfRiNv+pgtzv06u+9pr/+94MzPtdodQD26mm0Ud5sI94tadLZ927R//n3A3pvVduMz/fY+5U6+94tOufeLTrvv18eOmcwfC61Do56TLYONz7wvr7+/u0e/d1/3uH8fJxs657ReYL1fZhJG/G4h15bf/HGe/rW8Qadfe8W/cbRhik/9y+FVTr73i26ztIz6ecEw+dS69Hr4XA49Jrvvab/8+8Hpny+3gGbXvCNLfoXr53wQOkmx8jvxa/fLNE539iiO3sHxj1uum2EIacRAlZgZDqQeOCs1Xxa6wqtdaXW2qG1Pgx8H/joVM8j/M9TyTHcFqXHUi7TCANaRUs37T2DrMtO9HdRfMadfvZb1yz3SEbFdlfWL03gb/QcynoH7BysbR9KeuIp7r153Gs/hOesltf2LC8cqCdhVjgXL0mb8nOzJSPhGWotvVh6Blk9jWumqHAz2SkxlMkWOYDze3eBl5JjgEHXbAElQJhSavGw+1YDRyfxXI1zBs5MzyN87EBNOwtSY0iMjvDI+Ralx1Lf0Ud3v80j5xO+N7ReK4QyER6u7SA7JZqEaM80+hcuTsOdMT/QN3oOZftrLAzaNQUeSo7htnROHOFmxUFZW+Rx8tqeqXfAzuvHGrl65WwiwqZ++Tl8ry1xuoN6qskx3BamxcrGxi6Hazu8tl4LDBpsaa27gWeB7yulYpRSFwDX4VyPdQal1NVKqQzX/5cC9wHPT/U8wr+01hw4ObW9IiayMM3ZCyabGweu/TUW4qPCWJgWOptYHq7zbKOfn53EDesyUcAfblsf0PuPhbLCyjZMCtbnePb9iwwzs2xOPIcka57HyWt7prdONNIzYOfDq+dO6/lzEqIIMykZ2XI5eLKdyDATebPjpvX8xRmxVLZ0M2h3eLhkgaW5q5+GTu8lxwCDBlsu9wCzgCbgaeBurfVRpVSWay+tLNdxlwGHlFLdwMs4g6sfTXQeX1VCTM6pjj6auvo9GmxJ+vfAV1RtYW1WEqZJbGYcDCzdA9Raej3e6F+/Zh4a5/42IjDtrmxlxdwEr0xzWTUvgSN1HTjk8+Fx8tqe9sKBetLjItk4zdH1MLOJeUmzqJb074BzM+OVmQmEm6d3Kb84PRabQ4d88HrENXU/JIMtrXWb1vp6rXWM1jpLa/2U6/4a7dxLq8b189e01hmu43K11t/WWg9OdB5hLJ5erwWQnRJDmElJsBWgOnoHKW2yTmsk5mc/g61bxz/G+fjcjGkVzkvc67U8vdfHuuxEwkyKXbK5cUAqrGhhT5VlxvsPjmXVvES6+m1USBpoj5PX1qmjd5Btxc18cNUczDPoPMtKiaEmxIMDAJvdweG6jmlPIYThHdKhncbgUG0HSsGKUAy2RGg5cLKdiDATy+aMzGcyfeFmE9kp0TKNMEAdONmO1tNbr7VhA3z842MHXFu3Oh+HbkN9a7uDranumTKR6IgwVs1LYJckxwg4RdUWbnt0D3aH5rWjDTPa5Hos7gu2X75R7JXzh7LTSTLa/VoOf3v43XIG7A6WZExvyptbdnI0Va3d7uzSIauk0UrfoIPV86f/XeGenh/q67beK20mKTqc4gbvBZ0SbAlDeK+kmZSYiKGLTU9ZmBYrI1sBal+1BZNiWl8mmzfDM8+MHnC5A61nngHoMFSX3qFaZ5KYeC9MFduYm8Kh2g56BiRhTCAprGgdWlPhcGivZJPs7HNOBnn5cAO3PFIoAZcHLUqPJTrCHNIZCYuqLTy4rRyA7714dEafr+yUaLr6bLT3DE58cBBzJ12ZyWygmMgwMhNnURrC10hF1Rb2Vlto6x70atsnwZbwu92VrRxv6KKho8/jH/ZF6bFUt/aE/ALQQLSvxsKSjLhpr1EZLeAaHmht3uzBwnrIkbpOj49quW1ckIzNodlX3e6V8wvvKMhNGVqz6K1skruHTS+V7QE8y2xSrJybENIZCQsrWnEvWZvp5ysr2TmVNtTXbR082U5idPjQ6zFdi9JDu0P63ZKmof97s+2TYEv43bP76gDv7AO0aGgBaGjPlw80dofmQE37jDPnDQ+4/vjHHEMHWq3Wfurae1nlpWBrfU4yZpNiV6VcSAeS/Owkls2OY05CFE/eUeCVbJIFuSlD62hkewDPWzUvgWP1nSHb6bfKtQZVMfPPV3aKe6+t0P5OP1jbwap5iSg1s+RRi9NjKW+2hmzypDmJswAwKe+2fRJsCb8bsDm/gMxe+LBLRsLA9PyBOrr6bSTHzHzPtc2b4e674Ykncrj7bmMGWuC99VpusZFhrJwbz64KSZIRaBq7+rlwUarX0vbnZyfxhc0LAfjxR1bJ9gAetmp+Iv02h1fXhBhZbGQYANevzZxxh4F7JCeUk2TsKGvhREMnGXEz/35cnBFLv81BrSU0X88wkzMM+swFC7zWmQUSbAkDKG7sYvmcOL5yZZ7HP+zuBaDlzaHdCxZIiqot3PvPQwA8/G7FjKeVbt0KDz4It95axYMPTpyl0F8O17qDLc8liRlpY24KB0620zdo99rvEJ7V0TNIc1c/izO8u9fch1Y59z5yhHjiAW9Y7RrZCdV1W+41QV++bPGMv99nRZhJj4sM2WmERdUWPv2nPWgN/zpQP+Pvx0XpzoQlodohXdZkJdys+ObVS73aySTBlvCrpq4+jtZ38qHVc/n85kUe/7DHRIYxJyEqZBuSQORMCOC84LPZZzatdPgarc9+tmrMpBlGcLiug9zUGK/so+S2cUEyA3YH+2vavfY7hGeVNTtHQ9yj9N6Sm+ZM5ODpJEXCORoTG2nmr7trQjL5SHmTlYgwE/NnuL7ILSeE078PT5hj90DCHHe7EqpJMsqauliQGkPYNPcqmywJtoRfvVvSAsAlS9K89jtCfQFooCnITcE9DX0m00pHS4YxXpZCfztc1+Hx/bVGWp+TjFLIuq0A4m67FqXNLGX2RMwmxfI58UMbfArP2VfTTs+AnUN1HSGZ7bG0yUpuasyM9tcaLislmuq20JytMjxhToQHll0kzAonKTqcFw/OfJQsEJU1Wb3ekQUSbAk/e6ekmbS4SJZ7cH+tkRamOReAOkJ0AWigWZeVyKxwM6vnJUx7Wul4WQeHB1yQ4N0r2Elq7urnVEefV3ewB+cX6/I5sm4rkJQ1WYkMM5GZNMvrv2tlZgJH6ztDdrG8txRWtKI9lI0vEJU2dbF4hvtrDZedHE1jZ39ITofOz05i+Zw4ZnsoYU5RtYWO3kGO1neGXEdA36CdmraeoamU3iTBlvAbu0PzXmkzlyxJm3FGnfEsSo+lZ8DOqc4+r/0O4TlNXf30DNi5Yd28aX+R7NkzftZBd8AFMZ6Z1zJD7tEEbwdbABsXpLCvxsKgXFAHhNImKwvTYj02KjCeczIT6B20UyEbwXtUQW4KYWbn+xdmDq1sjz0DNmotvSz24OhBVoqz2f7pqydCKjhwa+zs54KFnkmY48m0/IGmsqUbh/b+FG2QYEv40aHadtp7Br06hRBOJ8mQqYSBwZ2xayYJAb7+9YmzDjofr2+c9i/xoMN1HSgFK3wRbOUm029zUNkRmmmoA42vprkAQ9NYZd2WZ+VnJ/H961YC8B9XLAmpbI8Vzd1ojUeDrV7XiNZj71eF3GhMR+8gTR5MmFOQm0KYKTQ7Ak5P0ZZgSwSxd0qaMSm4cFGqV3+P+0KlXIKtgFDS6Ay28jw47cToDtU6k2O4UyR707k5yQCcaAu9KTiBpnfATl17r8+CrYVpsUSFmyTY8oJrzpkDOPeaCiWlTTPvPBup3tILeGdvTqPzdICQn53E1z+QB8B3rl0eUh0BZU1WTApy02K8/rsk2BJ+s624mdXzE0nywF5K40mNjSBhVjhlMjUmIJQ2WkmJiSAlNtLfRfGZI3XODSp9ISkmgqWz4yiWYMvwyputaB9NcwFJkuFNCbPCSY2NoCLEtiEpbbQSZlJDmxF7wiV56UNBa6htwu3uNPZk8Hr5sgwAIsLMHjtnIChrsjI/OZqocO/XW4It4ReW7gEO1rZ7fQohgFKK2QlRvFvSHFLTDQJVcWMXS0JoVKupq4+Gzj6vbWY8mgWpMZxoc7BbshIamrsX25NTsCZyjitJhiQU8rzc1FgqWkKr06+0ycqC1BjCPZhaOz87iXMXJJEUHe7VjWiNqKzZmUZ/XpLnlhvPT47GbFJUhthns6zJ6pMphCDBlvCT98pa0Nq7Kd/diqotlDVaqbX0htz87kCjtaa0sYslXt7A1UjcowirvJz23a2o2sKbxxuxa/jUI7vl78HAypqsmD08KjCRlZkJ9AzYqWgJrREYX8hNiwm5ka2yJqtXNuS+YFEa7b2DLJ0dOh1zAKWNXR5Now8QbjYxP2kWVS2hs3eZze6gsqWbRT661pBgS/jFO8XNJEWH+2TqlDPbjrOXNtTmdweauvZeugfsHk0TbHSvHmkAYMDmm4QVhRWtQ6m9B2a4abTwrrImK9kp0USE+e6r2p0kQ6YSel5uWgyt3QN09Az6uyg+0W+zU93a7ZXU2ivmxqM1HD/V6fFzG1lZs9Ur3485qTFUhlAHS01bDwN2h4xsieDlcGjeKWnmosVpPklnXJCbQrjrYsVkUiE1vzvQlDY6pzHkhUhvZVG1hX8U1QLwucf3+GSUqSA35YyL98xE7+/fJKantKnLZxcDboskSYbX5Ka6kjWFyHQtd2ptb0yDXTHX2SlwtD50gq3eATu1ll6vtAkLUmOoau1G69CYPjyUaMRHU7QNG2wppZKVUs8ppbqVUtVKqZvHOO52pVSRUqpTKVWrlPqZUips2OPblFJ9Simr61bsu1qI0Rxv6KTF2u+TKYTgnN/9l8+dS5hJceXyjJCa3x1oil2ZCJf4YJNBI/DHHif52Uk8eUcB1+aGE2FW7Chv8frvFFM3aHdQ3drjlSlY4wkzm1g2J16CLS9wZz0LlamE7s4zb1zQZsRHkhITwdH60PmcejNhzoLUGHoG7DR19Xv83EbkTpgW8sEWcD8wAGQAtwAPKqVWjHJcNPDvQCqwEbgM+NqIY76gtY513fK8V2QxGU/tqgEgITrcZ7/z3AUprMtO4qQrZawwppLGLjLiI3362fAn9zothW+zauVnJ3Hjkgg+eW4W/9pfT5Ns+G041a3d2BzaZxcDw52TmcAxSZLhcfOTowkzKcpDJDNuqSu19oJUz685VEqxfG58SI1suT833uiAyXGtCw2VqYRljVZmx0cRF+Wbaw1DBltKqRjgRuA+rbVVa70deAG4deSxWusHtdbvaa0HtNZ1wJPABb4tcegpqmrjd2+XTnna096qNp7a7Qy2vvDUPp8uzl+fncTR+k56Bmw++51iakobrSGViTA6wply9iPrMv2SVetzFy5g0OHgsR1VPv29YmKn99Px/d/DyrkJWPttVLaGxoWXr4SbTWSlRFMRIsFWWVMX2SkxXkutvWJuAiWNXT5b7+pvpY3OhDk5XkiY4w6IQybYavbdZvEA3t9Bc3qWAHatdcmw+w4Cl0ziuRcDR0fc92Ol1E+AYuBbWuttI5+klLoLuAsgLS2NbdvOOiTgWK1Wr9Sj1GLjR7v60UC4qYR7N0SxKGnixrRrQPM/u3pxTwkeGHTw9Jt76Fo49j5bnqxDRKcNu0Pz+IvvsCzFd/tJeOt98CVf1MGhNcWnetg8P8wrv2umdfBGG7G1xrlQ/vy4Nroq29lWOeNTTprVaqXy8B7y0808tr2cVWGniAoLvC1Xg/Xv6/XyAQDqT+yjtcy370tfp3MPtmfeKOS8uZO7TAjW92EqJtNGJKg+Dlf3BMRrNdPX40BlDxnRJq/VVbXbGLRrnn55K9nxo3+nB8PnEpz12FlaSdos2LH9XY+f36E1YQre3X+COT0VHj8/GOe90K5rjYsyp36tMd06GDXYigVGTsTtAMbt4lNKfQZYD9wx7O57gWM4pyR+EnhRKbVGa10+/Lla64eBhwHy8vL0pk2bZlJ+Q9i2bRtTrUdRtYXCilYKclPG7GX/82N70DQBYHNAf2I2mzYtGve8uypa+cZfD9DSC2Emhdaa8DATN12+Ydze/OnUYSxregb41b43sCdlsWnTYo+cczI8WQd/8UUdqlu7GXhtG5euX8amDVkeP/9M6+CNNuLt548QF1nHjR/YjFK+vaB2vx4JuRY+8sAOTs3K4TMXLPBpGTwhWP++nmvYT2aihasu3+zz8gzaHfxw92s4EuayadPyST0nWN+HqZhMG7Gz9zh/2l7FRRdf4pMEUTMxk9dj0O6g6fVXuX59Dps2LfVswVyymq08ePAdZs1dwqb180c9Jhg+l+CsR7sDVmXHsmnTeq/8jpz972CfFeO18xvlvahr76X/tbfZtG4pmwqyp/Tc6dbBqMGWFYgfcV880DXWE5RS1wM/AS7XWg+t+NZa7xp22ONKqZuAa4Dfeqy0QaKo2sInH97JoF0TGWbiqTvPntZ0pK6Dd0uaMSlwaNDAuqzEMc+5p6qNX71Ryo7yFnJSY/jX5y+g35UIYLyAzhsSoyNYlB4r+woZVHGD8887lNK+n2joYsnsOJ8HWsOtzUpiQ04Sj26v5NaCbMI8uPmomL6yJt9OcxkuXJJkeM3C1FgG7A5qLT0+3T/N19xrDr2Z4CUnJYaYCDPHQmDdls2hqW7t5aoVs732O9wZCYOdrzMRgkHXbAElQJhSavjww2rOnh4IgFLqA8AfgGu11ocnOLfGuR5djFBY0cqg3TnHr9/m4Pn9dWc83t1v40tP7yclNoJHP72BT2xw9iSN9YW8u7KVT/x+J++Xt6AU/OD6lazMTCA/O4nPb17kl6yA67OTKKq2yMJvAyp1NYDeSBNsRFprihu6DJHm/s6Lcqm19PIffzsgnREG4HBoyn28pmCkczLjOVonSTI8LVQyErozES72YmZZk0mxbE58SGQkbOzRXg9encFWT9D/zUuw5aK17gaeBb6vlIpRSl0AXAc8MfJYpdSlOJNi3Ki13j3isUSl1FVKqSilVJhS6haca7pe834tAo/7Ile5bn8vqmV3ZdvQ49994SiVrd386hNr2ZyXzk9vXMWmvDTu31pOR+/ZmzT++JUTQ2mtFXDwZLvX6zCRddlJdPbZhtJ+CuMoaewiM3GWz7ID+VtjZz8dvYMsNUCwlRwTgQJePHSKWx4plIDLz+rae+kbdPg52Eqgq99GdVuP38oQjHJdeyQFe0bCsiYrSsFCL+8Tt2JufEhkzqy3OpOAeDNhTk5KDAM2B/UdwZ21uaypi6TocFJixs4X4GmGDLZc7gFmAU3A08DdWuujSqks135Z7kUd9wEJwMvD9tJ6xfVYOPBDoBloAb4IXK+1lr22RlHj+lL93EUL+P2t+cxNjOK2P+7ikfcq+MJT+/h7US1f2LyI8xaeTk/99auW0tk3yEPvnLEEjr8UVrO/ph2zSWFWvk1rPZ71rtE0uZg0nuKGLpb4eE8hfzrR4Jz6kmeAaZO7hnWqDPhovy8xtjIDjPKuzHRuS/CL14ulvfSg5JgIEqPDqQjyrG+lTVbmJc1iVoR3k1GtmJtA94A96DsFTnU7g62F6d6bepqTGg1AVUtwv5buKdq+nL5v1DVbaK3bgOtHub8GZwIN989jrh7WWjcDG7xRvmC05dApVsyN578/6FwQvS47iY8+uIMfvnQcAKXgosWpZzxn+dx4rl+TyR+3V3L7eTnMTohiR3kL333hKJvz0rhn8yJ2V7b5fH3WWBakxpAcE8HeKgs3nev5JAxiemx2BxXN3T7b6NoI3GvUjDCNsCA3hcgwE302B0opQ3SMhDJ/THMZqbvfuUXGS4dO8ebxRr9sTRCsclNjgj79e2mT1atTCN2Wz3Uu7z9a3+GV/byMot7qIDNxFtER3rtsz011tjeVLVYuHHGtFyy01pQ2Wbl6pffWvo3GyCNbwodOtvVw4GQ7H1o1d+i+1NhIPrz69M8K2FN1dg/nV65YgkNrfvVmCdWt3dzz5D5yUmP4zU1r2ZCT7Lf1WaNRSrEuK4mi6raJDxY+U9Xaw4DdEVLJMYobnBs4J0b7birDWPKzk3jyzgKWzY5jVriJc1yjGpNVVG3h/q1lMgLiIWVNVlJjI/z62XC39RoYlNFOj8pNiw3qNVt215pDX4zMLsmII9ysgn5z43qr9zc4z4iPZFa4mcogHtlq7R6gvWeQRT7oCBhOgi0BwEuHTwHwoVVzzrj/krx0osJNmBVEjDEVcH5yNJ8qyOZve05y3f3vY7M7eOS29YZde7M+J4mq1h5arP3+LopwKW10jfKEULB1oqGLvNkjk676T352EvdevRRrv523TzRN+nlF1RZu/kMhv3i9WNZ7eUhpU5fX17pMpCA3hXCzc5qN2WyMaeDBIjcthqaufrr6zl7rHAxOtvUwYHOw0AfBVkSYicXpcUEdbNkdmlPdDq8Hr0opslOigzojob9mDUiwJQDYcqie1fMTmZ8cfcb9+dlJPHlHAV+5Mm/caSQXL05FA+09gwzYNK3dAz4o9fTIui3jKWl0Lqb257QpX7LZHZQ1Ww2RHGO4CxelkhYXybP7aif9nMKKVvptDhxaRkA8QWvNiYYu+m12v7ZR+dlJPHrbBkwKrlqeYZjZCcHg9HSt4Lyo9XVmWWeSjA60Ds4kGXWWXgYdvvl+zE2LoSpIP5cgwZbwo8qWbo7UdXLtiFEtt8mkaj92qgv3/ox2h7EvuFZmJhBhNkmwZSAljV1kJUd7fTG1UVS1djNgcxhuJC/MbOL6NXPZWtxE2yQ7TDITZw393yTrvWbsrRNN9AzYOXiyw+8jhRfnpXHZsgx2VbZhD/Jsb760MMjTv28rdo6MW13r/rxtxdx4WqwDNHUF52yV0ibnzA9fBAg5KTHUtPVgszu8/rv8YUd5C+FmRUO7bzMuSrAl2HKwHoBrzhk92JqMgtwUIsJMhso8OJaocDMrM+Ml2DKQksYunyymNooTBkqOMdIN6+YxaNdsOVQ/qeN3VbYRblakxUaQGBPO6nlTW+8lzvTiAefrbpS1UtevyaSpq9/v5QgmWSnRmBRBmSSjqNrCX3efBODOP+/1yffsCtca02Ddb8uXozE5qTHYHJpaS/Clfy+qtvDqkQYG7ZpbHt3l02tACbYEWw6dYn12EnOH9VBP1WSnGxrF+pxkDtd20Ddo93dRQt6AzUFlSzd5s0NjCiE4k2OYTcqQ0yaXzYln2Zx4/rmvbsJjLd0DPLuvlo/mz+dHN6yiuWuAFycZpInRnXLtcWOUjqvLlqUTGxnG8wcm/jyIyYkMMzM/OZryIJyuVVjRit01nc9XnQXL5sSjFBytC851W2VNVuIjlE8S5uS6MjpWBuG6rcKKlqG9X33dkeXxYEsp9Vul1Iuj3B+vlPquUmrZsPv+Qyl1SCklQZ+flDZ2UdzYdVZijOmYzHRDo1iXlcSA3cGRuuDsCQskLx06hc2hCTeHTjNwoqGLnJRoosKNOW3yxnWZHDzZPtSjOpandtfQb3PwmQtyuGxpOnkZcTywtTzoNxj1lvaeAfafbOfaVXMM03EVFW7mqhWzeeVwg3ROeZAz/XvwXdBuyHF+XhW+6yyIjQwjJyUmaJNklDZZyYz1zZ5QOe5gKwg/m+6kQ778bLp59OpGKbUQ+H/A90Z5eD3wHZwbDbs9BKQDt3uyHGLyXjx0CqVmNoUwEOVLkgxDKKq28PV/HgTggW3lIfN+FDd0sdRAmQhH+vCauZgUPLd/7EQZg3YHT+ys5qLFqSzJiMNkUtyzeSGlTVbeON7ow9IGj5cOn2LQrvl/lyw0VMfV9Wvn0tVvY+sUslSK8eWmxVLZYg26jonkmEgArl4526edBcvnxnP0VPB1nmqtKWnoot/um+uVlJgI4iLDgjoj4S0F2T7vyPJ0V/K/Awe11ntHeWwt0A8cc9+hte4F/gx8zcPlEJNQVNXGE4VVLJ8TR3p8lL+L41NpcZHMjo/k70W1IXOBb0SFFa3Y7M6LDbvd/+tTfKG730ZNW48h12u5pcdFcfGSNJ7bVzfmxeArRxpo6OzjMxfkDN33wXPmkJUczQPbyoM2M5g3PbevjiUZsayYa6xA/LzcFFJjI3n+gEwR9ZSFabH0DTo41dnn76J4VIlrG497fNxZsGJuPCfbevnl68VB9Z3+5rEmegbtVHQ4fJIwRynFgrSYoMyUeai2gzCT4r8/uMznHVmTCraUUouUUoNKqe+NuP9BpVSXUmq9UioS+BTw1CjPPw78HIgEBpVSWin1D9fDfwWWK6XOn1FNxJQUVVu46ZFdWLoHKWmwBlXjNBlF1RaauwYoa7Jy8x9kbyB/KchNcY7pY4z1Kb7gvhgxcrAFzkQZ9R19FFaOHgD/cXslC1Jj2LQkfei+MLOJf7tkIQdPtrOjPPgDZ0+qae1hb7WF69dmopRvpgxNVpjZxLWr5/D2iSY6eoNzbyhfyx3KSBhcSTJONDgzE/t6PWpkmPNy9ndby/yexdOTXjp8uoPDV+uMclJignJk63BdB0sy4vwyfX9SwZbWugx4BPgPpVQqgFLq28BngY+4RrIKgETgvVFOcRtQAbwInOe6fdX12AGgE/jAdCshpq6wopVBmzO1p0PrkBhRGK6wohWNs+e93+agsKLFzyUKTSsz4zGbFBtykgyxPsUXil2ZCI22x9ZIVy7PIDrczI9eOn7Whcu+GgsHTrbz6fNzMJnODAxuzM8kPS6S+7eW+bK4Ac+dgOK6NZl+Lsnorl+TyYDdwatHTvm7KEHBHWyVT7AuMtAUN3SSkxrj8wvapk5n2vdg2++v07XxtQnfdUjmpMZQZ+ml3xY8azS11hyp6+CcTP9ky53KNMLvAWbgXqXU53Cuv7pVa/2m6/ECnNlqD43y3IPAPOBtrXWh61YNoLV2uJ5TMM06iGkoyE1BhdiIwnDuVPXuy8RBu0x58oej9Z3Y7JrPXbggJAItcPb8RkeYmZ8UPfHBfnS0vpN+m4Mj9Z18/Pc7eX7/6Wx0f3q/irioMD6aP++s50WGmbnr4lx2lLfyrecOB00PszdprXlufx0bFySfsW+Zkayal0BOSrRMJfSQtNhIosPNPH+wPqj+RkoarX7ZP/Dy5RmAf5IfeIvDoTlU28kFC1O4YXG4zzokc1NjcGg42dbj9d/lK7WWXiw9g6z009Ykkw62tNYNwK+ALwK/B76ktX5m2CFzgU6t9Wg7Ya4AIoB9Y5y+2fV84SP52Umkx0WyOD02ZEYUhnOnqv/qlUtYkhHLn96voinI5s4Hgn2ui4x1WaHz+Stp7GKxK6GEkQ0f/bU7NF/+2wE+/vud/Pjl42w5VM/mvDRiIsNGfe6yOc41R0/uqgmqKT3eUtnpoKKlmxvWGXNUC5xrOa5bk8mO8lZ++soJeU9naF9NO702O/tr2oPmb6R3wE5Va7dfpkhvyEkmIy6SvNlxQXNNc7iugxZrPx9dP48PLYzwWZ2GMhK2BE+w5c48vSoARrYASnGuu9qptb5/xGNROBNgjGYdzlGvA2M83gsYszsvSHX2DdLQ2c91a+YGRaM0HfnZSXzh0sU8+Kl8+gbt/Ndzh2VRv48VVVuYnzwrpBK0FDd0sdQPPb9TNXyj8sgwE7cWZFPd0s3v361Aa3jtaOOYF4gHTrYPjRr3DwbPlB5v2VlvIyLMxAdWGjsr7KJ050XYQ++UB02A4C+FFa24v26CZdpbaVMXWvtvivRi13qcYLmmeftEEyYFlwxbF+sLC1Kcf+dPFlYHzd/4oTpncgx/rZWedLCllLoU54jWTuACpdTqEYe0AmN9wtcC5VrrsTZBSAZk0YwPuaP8lX6K8o1kYVos/3lVHm8eb5rURq7CM7TW7KuxkB9Co1rNXf20dg8YPjkGnLlR+VN3FvCD61dy63nZuAfkbONkjyzITSEy3Pn1ooE5CaETTE/VoN1B4Skbly9LJ2FW+MRP8KOaNueGy5rgCRD8pSA3hTDXH1O4OTimvbnXoy7xU2fS/ORZQTX17e0TTazNSiI5xvubGQ9X5krasq2kOWg6VY7UdZA32z/JMWDy2QjXAf/CmSRjE1AD/GjEYSeAcKXU2ZP4YTnDUr6PYgFQPJmyCM9wB1v+WixoNJ+9YAHn5iTz7eeP8JNXzk4IIDyvrr2Xxs5+1gVJL+RkBEpyDLeRG5WftzB1aLRrvHUR7kDt85sXkhgdzoPbyukdCJ7F1p60vbSFrgH4yNrRvjqNZXiAEBYkAYK/5Gcn8eXLFgPww4+sDIrRmOKGLiLDTGS7RkZ8bV5SNK3dA3T32/zy+z2psbOPw3UdXLrUt6NawBmdKMHQqaK15rAfk2PAJIItpdQi4BXgdeCLrjVZ3wOuUUpdPOzQd13/njvKadqB1Uqpq5RSBUqpoRZaKZUILBn2fOEDh+s6mZsQRUpspL+LYggmk+L287PpGbDz0DsVfOL3O3nlsDPrVlG1hfu3lkkA5mFFIbhe683jDQABm+Vp+GjXROsi8rOT+M+rlvLbm9ZS2mTlf14er78tdD26vZJwE8RHjb7+zUjys5P4xceck1puOz8nKAIEf3IndfBXb7unFTd2sTgjFrOf1qNmJTuTDtVaev3y+z3JvYH4Zct8H2wV5KZgdmVQC4ZkI7WWXtp7Bv06k2vc1l0pNRtnkHUcuMWVORCcGxF/HfgJcD6A1rpKKbUbuBZ4dsSpvg08inN0LAq4CNjueuyDwADw3AzrIqbgSF2HTCEcoaq1B4VziozNobn7yX0sSI3mZFsvdocm3Gziv65ZyrI58ZQ3Wzl+qos18xNZPT+BcLOJEw1dFDd0csGitKGLkKJqC1vKB4hbYJELkxH2VVuIjjAHzCjPTBVVW/jzzmoA7n5yX8Au4s7PTppSuS9anMZdF+fy8LsVXLw4jStXzPZi6YyjqNpCYUUrBbkpo75edofmd1vL2F7mnEF/+592B8Rn4sNr5vLrt0s5Xj/WqgAxWQtS3XttBceeRsUNXVy0OM1vv3++K9gy+qbxk/H2iSYyE2f5JbNjfnYSt52fzZ/er+KBW9YZvk2ayGF3cgw/ZSKECYItVwbC3FHutwPLRnnKg8CvlVKf11r3DDv+CLBxjF/zKeDvWuszximVUsk4A7Qrca7n+qbW+qwNk13H/gdwL84kG/8E7tZa90/1PKGiq2+QypZublhr3MxX/uBeZzJocxBmNvHJDfN543gjNodzFfOA3cF3Xzyzd/6JwuqzzvN/b5SyMjOe2fGz2FrchN2h2VJVGBAXUr60r6adNfMTCTNPNU9PYNpe2oxjxIL4UPk8fO3KPHaUt/CVZw5wa0EOly/PCOq6F1W18YmHC12dNIpHP72BixanUVRt4f2yZnoG7Lx6pIGq1tPrSwLlM6GU4gMrZvP7dyto7xkgMdq360mCSVS4mczEWUGxsbGle4Cmrn6/dp7NT3LmWQv0dVt9g3a2l7Vw47p5ftvg/PJlGfzp/SqiwgJ/1PVQbQfhZv8lx4CpZyOcyBNAHXDPZA5WSq0BNuOcljjS/ThHvDKAW4AHlVIrRjnHVcA3gMuAHJzB4fDzTeo8oeSoq0fSX/sNGNXIhADfu24lv71pHZFhJkwKIsJM/OC6FXxi/byhJAEmBdeumsM158weyr6mgRbrAO+VNWNzaFlMPoqeARvHTnWG1BRCd55L0wTrnYJRRJiJuy7Oxdpv58EQyGT3+M7qob/9Abvmtkd3c9kvtvHx3+/kl2+U8tA7FZhNiq9ftYSocJNPNyz1hA+snI3doXnzeJO/ixLwctNiqGgJ/JGt4kZXcgw/XtAmx0QQHWHmpCWwg61dlW30DNj9sl7LbXFGLHD6fQ1k7uQYkX4MHD06SVxrbVdKfRZnqvfJmA18RmtdNvxOpVQMcCOwUmttBbYrpV4AbsUZWA13O/Co1vqo67k/AJ4EvjHF84QMSY4xtpFTpPKzk3jqzoIzpgMVVVt4/mA9gzYH4WEmPn3BAsA57O++73c3O/8EPvbQDhw6sC6kfOHgyQ7sDm34XnxP2lfTTmpsBJ8+P4fzFqaGVN0BTradXkcRKKM402Htt7G9tAWFM7A2m0xct2YOOyvasLuGNk0KbliXyT2bF7MxN5Wn39zDTZdvCJjX45zMBOYmRPHqkYZRN7YWk5ebGsM/99WhtfbbKIYnGCH5j1KKrOTogB/Zevt4I1HhJs5b6L9rhrTYSJKiwykJ8GDLnRzjmnP8O31dGXFfIaXUWmCH1nrWsPu+Blyitb52xLEHgR9prf/m+jkV5ybJqUDWFM5zF3AXQFz6/PzkzzzolboJIYyj+qcfKtJar5/MsdJGCBF6pI0QQoxnUm2E1tpwN5wJNBpG3HcnsG2UY8uBDwz7ORznjJ2cqZxn+G3JkiU6GGzdunXU+zf/fKu+4/E9vi3MNI1Vh0BxuLZdZ9+7Rf9j70l/F2VGPP0+fPZPu/Vlv9jm0XNOZLQ6AHv1NNqoqbYRD20r09n3btEVzdbpV8DD/PG39W5Jk17wjS36S0/v89g5jdRGlDR06oXffEn/598PjPr43qo2/bu3S/XeqrYz7jdSHSZrZ3mLzr53i95ysF5rHZh1GMkfbcS7JU06+94tekdZy8wr4GFTeU9veOB9/bGHdnivMJP03ReO6GX3vaIdDofWOvA+lyUNnTr73i36ycLqM+73Rz3u+9dhvfLbrw69ljPljzpsOVivs+/dog+dbPfI+abbRhh1ZboViB9xXzww2njmyGPd/++a4nlCgrXfRmVLt0wh9JHlc+KJi4B3Spr9XRTD0FpTVGNhXVaiv4viE1pr/rmvlnVZiUPZx0LVRYvTKMhN4VgQZrLTWvPt548SExnGvR9YOuoxI/ctC2QbcpJJiYng1aMN0z6HbKsBuWnOtTEVLYGbJENrTUlDlyEyy2YlR9MzYKete8DfRZmWt1wp3zcv9V9WR7fFGXF09ds41dHn76JM26G6dsLNiiWzY/1aDqMGWyVAmFJq8bD7VgNHRzn2qOux4cc1amd2w6mcJyQcretAa1mv5Ssmk2JlqpntZS04HMabsusPFS3dtPcMBsUF52Qcre+kpNHKDetkbQvApUvTKW2yBvy6ipFeOFjPzopW/vOqvJDYv9BsUlyxPIO3jzfSNzj1feP2VLXxyYd38r+vFXPzH4I7Ycp45sRHERVuCuj073XtvXT121jihzTlI81POp3+PRA9v7+O9LhI6tv9H+C4084HcpIMIyTHAIMGW1rrbpx7dX1fKRWjlLoAuA5ntsOR/gx8Tim1XCmVBPw38Ng0zhMS3PsNyB5bvnNOahht3QMcqe/wd1EMYZ/roipUgq1/7qslwmzi2lVz/V0UQ9jsyrC1tTh4Mtm9V9rMfz17mIVpMdx0bpa/i+MzV62cTfeAnR3lLZM6vqjawm/eKuV/Xz3BnX/ey6Dd2QHVb3Pw1vFGbxbVsEwmxYLU2IBO/+5OomCEkS33XlsnA3Bj43eLmzne0EVzV78hMrYucWUkLA3QYEtrzeHaDs7JTPR3UYwZbLncg3PfrCbgaZx7Zx1VSmUppaxKqSwArfWrwM+ArUC16/adic7ju2oYy5G6DmbHR5EWF/w9r0axMsXZo/KuTCUEYF+NhfioMHJT/Tus7wuDdgcvHKjn8uXpJESH+7s4hpCbGkN2SjRvnwj8YMvh0Px970lu/+NuugfsnLT0cuBku7+L5TPnL0whLjKMV49MPJVwb1Ubn/j9Tn75Rgn3bysnOtxEuFkNbaPx/IE6mrr835vvD4Ge/v1Eg//TvrvNTw7cvbb+sa8WwDDbxSRGR5AeF0lxQ2B2BNS09dDZZzPETC6Ppn73JK11G3D9KPfXALEj7vsl8MupnCdUHa7rkFEtH4uPVKyYG8+7JS184dLFEz8hyBVVW1iXnYTJFLhpjifrneJmWrsHuGGtTCF0U0qxOS+dp3fX0DtgZ1ZEYG2a+U5xE3/dcxJrv41j9Z20DlsbYrcHb0r70USGmbl0WTqvHD6Fbb6JuAWWMev+0LsVQxvEmxTcUpBNQW4qhRWtJESF86NXjnPzH3bx9J0FIdcZuDA1hlcOn6LfZvf7dKfpKGnoYm5CFPFR/u9Qio4IIzU2IiCDrd4BGwBmA+3FmDc7LmDTvz9/oB6AcLP/rzWMPLIlPMzab6OipZuVmSNzhghvu3hJGvtqLHT1Dfq7KH71bkkzJY1W5sRH+bsoPvHs/lpSYiK4JM//i52N5NKl6fTbHJOefmYUrx1t4PY/7eGVIw28V9rCisx4vnTpIiLDTIa6QPKlJemxdPXbea5scMypT23dA+wsa3HuO+baIL4gN3UoYcinzsvmT5/eQJ2ll4888D4/e/WE36dQ+dKCtBgcGmpaAy9AAOfIVp4BRrXc5iVFB9zGxlprjp3qYuOCJL5yZR5P3lFgiE6bJRlxlDZ1Bdya86JqC79+qxSA+54/4vf2RIKtEHKsvlOSY/jJxYvTsDk0O8r9Oy3An4qqLdzx+B4A/rmvzu+Nn7e9U9zEa0caKchNJtwsTe1wG3OTiY4wB9RUwlZrP9949tDQz2YFGxek8JUr83jqzgJDXSD50qDrImy8qU//+1oxfTYHv/z4mjFfp425KXzzmqXUWnp5YFu5Idas+Ip7SnV5ACbJGLQ7KG+2GmIKodv85OgzNlEPBFWtPdS193Lt6kxDZSzNy4ijb9ARcMFrYUXr0CbyRpiSKVcAIcSdHEOCLd/Lz04iJsIc0ingCytahxbE2x3+b/y8yRlY7sWuNW8cbwqZi8bJigwzc+GiVLaeaHLvf2hoXX2D3P6n3XT32Ygwnz2KFUwp3afqosVpRAxN01Fnjewdqm3nr3tq+PT5OVy/dvwLya4+G+4zDRjgAslXctOcW0JUBuC6raqWbgbt2hDJMdyykmdR196Lze7wd1Em7b1S57XBRYtT/VySMy12JckobgisqYQFC5IBUBhjxoEEWyHkSF0H6XGRpIfIFC4jiQgzcd7CVN4taQ6Ii0tvKMhNQbmupIzQ+HlTYUXrUI+/ex2PONOlS9Op7+gzfFrhvkE7dzy+lxOnuvj9ret5+q7QHcUaTX52Ek/fdR5Lk0zYtaZqWMDgcGi+88JRUmIi+fLlE69XLchNITLMeVmi1NmBW7CKiwonLS4yIDMSvuJKjuIwUFwzPykau0MH1P5Q75a0kJUcTXaKsfZiXOxK/x5o67bmJDoTpVy2LN0QbbUEWyHkcF2HjGr50SVLUqm19AZk76UnrJ6XQLjZxLqsREM0ft60dn4iYJxeNSNyp4B/67hxpxLurmzlg795j12Vbfzi46vZvDQ9pEexxpKfncR/bohi44JkvvWvw0O94P/cV8v+mna+cfXSSSVPyM9O4sk7C1iYFkNSdHjIbHwOziydgZaRcPi6mG89d9gwI/in078HxtS3QbuDneUthhvVAoiNDGNe0iyKGwOrI+BYfScAd29aaIi2WoKtEPF+WQtlTVZSYiP8XZSQdfESZ5KEUE0Bf/xUF/02B5++YIEhGj9v6nVt8vqx9fOCPrCcroz4KFbMjWerQddtFVVbuPkPuyhv7ibMpJjn2ixVjM5sUvz2prXERoZzz5NFnOro5aevnmBdViI3rM2c9Hnys5P43IW5tFgHKAmwC7yZyE0LvL22XjvScHpdjIFG8LPcwVaAZCTcX9NO94CdixYbM5FSXkYcJQE2jfBofSdKwdLZxkgIJ8FWCCiqtvCZPzkTEzy3P/gTExhVdopzf6F3SwMrA5un7K1uA2B9CAQeb59oIibCzA+uXymB1jguXZpOUbWFn79WbLh2qbCidShVudbaMBeSRpYeH8VvblpDZUs3V/7yHVqsA9x0btaUt3m4bJlz1PPNENroeGFaDJaeQSzDthIwun01zr9Zo2XinJMQhdmkAiZJxvbSZswmxXkLjfH6jbQ4I46KFiuDAbQG7tipDhakxBATaYwdriTYCgHOxATOPxK7Qy4a/OnixWlsL23hN2+VGO7iEpwbj/76zRJ2lrfQN2inb9BOYUULW8oHZlzevdUW5iZEMdc1lzpYaa3ZVtzMBYtSA3LPHF+amzgLDdy/tcxw2eeWz3X2iMpU0Kk5f2Eqn9wwn65+5+judNIuZ8RHcU5mAm+FULDlTpJR0RIYo1tvHmtkb7WF287LNtwaxjCziTkJUQEzjfDd0hZWz0sgYZb/9ykbTd7sWAbtZ67HNLqj9Z0sm2uMUS0w8KbGwnMKclMwmRR2h3btbyIXDf6SmTSLAbuDX71ZygPbyg31BbWnqo2P/34nY+Xv2FJVOO3yaq0pqrKwwZUhKJiVNlmpa+/lC5cu8ndRDK/V2g+cmTbcKH8PdlfmzE+em8VH8+cZplyBYG7iLBQze18vW5bOr98qpcXaT2ps8G9yPDz9e362/9vJomoLW8oHRt2oum/Qzve2HGVReiz3fWi5Ibe2mJ8U7ZxGONvfJRlfe88Ah2rb+eKlEyeQ8ZclriQZxY1dQwkzjKyjd5BaSy83b8zyd1GGGO8vRHhcfnYSSzJimZsYZaiL+1Dk3iHeoY2x98Nwj++oGgq0FHDx4lQuHrZgdyblrWvvpaGzLySmELrXIG2SjYwndN7CVEwGzVC5t9pCuFnxnWuXS5s5RectTCUyfGYbPV++LAOtMeyaPk+blzSLcLOiwg97bRVVW7h/axlF1RYcDs0bRxu56eFC/lk6+kbVD71Tzsm2Xr7/4RWGDLTAuW6rJgCmEe4ob8Wh4eIlxkuO4bYwLRaTImDWbbmTYyyfIyNbwoe0dqZAvXrlHLlo8LOLl6Tzm7fK0Bjr4tLu0OyrsaAAk+sC6cuXLwFgV2Ub/TYHzCAVs/vLOhQ+f1uLm1g6O445CcE9XdIT8rOTuLUgm8d3VvO7m9YZ6vNRVN3GirkJRIXLVNCpys9O4sk7CiisaKUgN2Va7+uKufHMjo/ireNNfGz9fC+U0ljCzCaykqN9niTDmQimkH6bAwWEmdXQfogAfYMOHttRydr5iZhMiprWHh7YVs61q+dy/iLjBgjzk2fRYu2n327sv9/3SpuJiwxj9bxEfxdlTFHhZnJSYgImYc2xU85ga8Vc42TflmArBDR39dPeM0iea3M64T/52UlcvCSVPVUWnvjsuYa5uHzxYD317X3855VLhoIqd9meurOArz5ZSK1VMzthenu0FVVbiIkwG2rjS2/o7Btkb5WFOy/O9XdRAsZH1s3j8Z3VzoDeIAZsDg7WdnBbQba/ixKw8rOTZtS+KaW4dFk6z++vo99mD4n1j7lpsT5P/15Y0Tr0t6eBNfMTWZOVxOPvVw2t9X7x4Ckqmrv5yNpMnt5dgwn41jXLfFrOqXKnf2/pNe6+llpr3i1p4fxFKYQZdITQbUlGXMDstXW0voO0uEjS4owz/djY767wCPemoUuC/EI3UFy6NIOeATuzDZIowmZ38Ju3Slk6O467Ny06aw+h/Owkvrg2EpNS/ObN0mn9jr1VFtZmJRn+C2Wmtpe2YHNoLnXtISUmtmJuPLPCzeypavN3UYYcqe9gwOZgfY4xOkNC1eXL0ukesFNYYZzPhjflpsVQ3dqNzYdZ32aFuzaRBqLCTXzj6mV865plPH1XATcuDueZfzuPX3xsNU1dffzwpeOUN3djc2jq2o09Rc8dbDX3GKcTZ6Sq1h7q2nu50KAp34dbMjuOqtZu+lzbmhjZsfpOVhgoOQZIsBUS3BtM5gXAwsZQsGqec2j70Ml2/xbE5YWD9VS0dPPvly8eM0VzyiwTN2/M4h/7aqc8zcXab+NEQ6dhRvG8aeuJJuKjwoY2NRYTCzebWJuVaKhgq6jKOe11XQh8Zo3s/IWpRIWbQiYr4cJUZ9a3WotvApl+m52/FNYwOz6Sf7988RlruvOzk/jQwgg25CRzY/48bi3Iwf3tEAhbIcx37YvXbOCRrfdKnXtuXmzAzYxHWpIRi0NDWZOxpxL22+yUNVkNtV4LJNgKCSWNXaTGRpASAhmdAsGyOfGEmxUHazv8XRRsdge/fbuMZXPiuXL5+GmbPr95ERFmE/83xdGt/TUWHJqgHyVwODTbSpq5eEla0I/gedr6nGSOn+qkq2/Q30UBnNNes5KjSY+b3rRZ4RlR4WYuXJTGW8eb0GOlSQ0ivk7//uj2Sipauvnxjav48uVLxu0Qu2DRzJOe+FJqbASzws20GHhk64UD9STMCqfFavy91dyd9Q9uKzPUFh0jlTZasTn00NYdRiFXBCGguNE6lLpT+F9UuJmls+M5aICRrecP1FM5waiWW1pcJJ+9MIcXD9YPZfuZjL1VFkzKuRYgmB071UlzVz+b82QK4VSdm5OMQ8O+mnZ/FwWtNXurLSGROTMQXL4snbr2Xk4ESCa0mchNc66r9kVGwvr2Xn77VhlXLs+YVJvlTnpitD21xqKUYl7SLJoMOrL1flkLe6stdPSOnvHRaCw9zoDw5cMNhi6v+9rESMkxQIKtoOdwaEobuyTYMpjV8xM4XNeBw+G/L4Ldla384KVj5KREc+XyjEk9566LFhIfFca3nz8ylCp4IkXVFvJmxxMXZcwNGz3FnaL6Ekn5PmVrsxIxmxR7Kv0/lbCmrYcWa79MITQI9/rHH7103LAXeJ6SHBNBbKSZFw7We72uP3zpGA6tue9Dyyf9nPzspLPW9BpZVnK0YRNk/G5r2dD/jbYNzGj2uKZWD987z4iO1ncQE2Em27Vmzygk2Apyde299AzYJdgymFXzErH223w2XWSkomoLtzyyi/aeQeraeyc9opAQHc6HVs9lb7WFX7xePGEPl83uYH9NaIwSvF3cxOp5CSGxAaunxUSGsWJuvCHWbe11XVQE+7TXQHHS0otS8F5Zi6F71D2hqNpC94CdQ7UdXq3rH7dX8PLhBm5YmzmUSCIYzU+OprnHYbgpqA0dfRRVtWFSBMy0zILcFMyu2S/hZuOW99ipTpbNiZ9wpo6vGS7YUkolK6WeU0p1K6WqlVI3j3Ps7UqpIqVUp1KqVin1M6VU2LDHtyml+pRSVtet2De1MA53qs682ZL23UjcU+oOnvTPuq3CitahvVQcjqktdk53BRMO7UyRPd5zTzR00T1gD/oL163FTeyvaSdPMn5O2/rsZA6cbGfAzyngi2osxEWGsSRd3ksjKKxodXanY+wedU8orGgd2ljeW3XdVdHKD7YcB+C5/XVBHbyCps8O75Q0+7sgZ/j568WA4v6b1wXMtMz87CS+8YGlANz7gaWGLK/DoTlW32m49VpgwGALuB8YADKAW4AHlVIrxjg2Gvh3IBXYCFwGfG3EMV/QWse6bnneKbJxudO+L5aRLUNZmBZLdISZQ7Xtfvn97mx5iqn3ql20JI3IMGfT4dDQO04q2GDdzLijX7O3qo2ypi5+8spxPvfYHgD+dcD703+C1bkLkui3OThc59/EMUVVFtZmJxmuZzRUFeSmEO5KOGMyTX9j9UDgrKvzc2f20ujB3/acdMeuDNqDN3gtqrbw5K4aAO76c5Fh2uWj9R38c18tn74gh6vPmRNQ0zJv2phFmEnRbO33d1FGVdPWQ/eA3XBp38FgwZZSKga4EbhPa23VWm8HXgBuHe14rfWDWuv3tNYDWus64EngAt+VeGJF1ZZJr23xhpKGLuYmRBEf5OtlAo3ZpFiZmcABP2UkVMr5hX5DfuaUe9Xys5N46s4CvnzZYlZlJvC7t8t4cFv5qFM19lZbmB0fRaZB9hTzFEu/5qMP7eTyX77LQ+9U4F56Zw/iixdvy89OBmCvH6cSdvQOUtLUFRLTXgNFfnYSj392AyYFHzxnbsBcmE5HfnYSv7t5HQC3FWR7pa6droyfgTJ9bboKK1qxuxpmowSVWmv+56XjJM4K5/ObF/m7OFMWGxnG6vmJvF/u/9dyNEddyTGWzzFWcgyAsIkP8aklgF1rXTLsvoPAJZN8/sXA0RH3/Vgp9ROgGPiW1nrbaE9USt0F3AWQlpbGtm2jHjauMoudE212liabWZho4nCLnd/s78fugHATfH1DFIuSzFM+73RZrVaKyntJjVTTqo8RWK3WgC2721h1SNYDvFk7yJtvbyXMx73oL5Y7Mwtdmmihq7KdbZXjHz9aHdaGwzkrNI84zPz01RO8faCU9FmK1elmliQ5m5Z3jncTH6F49F9v+/SzP5qZfpaGtxERs51flPkZZjZkmPnjkQFsDucFTGR7Ndu21XqiyF5lxL+t2dGKV/aWkqdPTvo5nqzHoWYbWkNYew3bttV55JyTYcT3Yqq8XYfcBBNHqk6xbVu7136HJ9uI6V5HRGhNUqTiSHkN27Y1Tbsso3Foze7yXpYlm1iRYmZpspmuyoNjtv+B/LmMbLcTpmBAa0AZol0+0GRjR3k/tyyLYP+u96f0XKO8F5lhA7xYPsjLb2wlOnxq1y3ersMrJQOYFZwq3kdrmXeuqaZbB6MFW7HAyK7+DmDCOXBKqc8A64E7ht19L3AM57TETwIvKqXWaK3LRz5fa/0w8DBAXl6e3rRp05QKXlRt4WdvFjJgc6AYJCrcfMb0KpuG/sRsNm3yXW/GW29vpbGnj2vWZrNp0zKf/V5P2rZtG1N9L4xmrDpYk+t5tWo/6UvWsmpeok/L9OeqPSxO7+GDV0yuH2O89+HSTZovPb2fLYdPAfBSpY3Z8ZAaG0nnAHQNaH6+b8Dv89Jn+lka3kZEzlmso8JN/NcNG8nPTuKKaguFFa0U5KYETM+7Ef+2Lm45yOvHGrn44ksmPY3Pk/Uoer0Ys6mc2z90CTGRvvt6NOJ7MVXersPO3uP8cXslBRdcRFS4dzpuPNlGTOc6wu38+n0cONnu8ddzf42Fztd28P0rVnP92swJjw/kz+UmYO06C9/9eyGHWxx88NLzmZPgmxkWRaN8HwzaHfzgV++SmxrGdz918dDU2MkyynsROb+VF8oLCc9czqZJZjF283YdHqvczeKMPq687GKv/Y7p1sGn0whdCSv0GLftgBUYOdkyHhh3gw2l1PXAT4CrtdYt7vu11ru01l1a636t9ePA+8A1Hq2UyzvFTUMLuzWwbE4cn7kgZ2j+tdaw2sf7DDX2aAbsDslEaFCrXQGWrzc3djg0+2osrMvyTFBgMimWzY3DfW2sgMToCFq7naNnRk8VOx1JkeqM4DHQUiIb1YacZNp7Bilr9k+Wzr1VFpbNifNpoCUm59ycZAbtmgMG2J/Q2/Kzk6hr7+VUR69Hz/vm8UbMJsWmENmeIj87iU/mORM6bT3hmyQZ75Y087GHdvC/rxXzsYd28NVnDvD8gTq+9dxhypu7+dj6eVMOtIxkXXYikWEmdpS3THywjxk1OQb4ONjSWm/SWqsxbhcCJUCYUmrxsKet5uypgUOUUh8A/gBcq7U+PFERcF4LetyxU865oiYFUeEmvvXB5Xzn2hX89a7z+MT6eQD8ZWe1T1OQ1lmdwZ9kSDOmeUmzSI6J4JCPLx4qWrpp7xn0aGBQkJtKRJjJOY0u3MT/fOQcfnfzOqLCTUG5NiAhUklg5QUbcpzrtvyRAt5md3DgZDvrXWvHhLGsz05GKQyxF5u3uduWfdXtHj3vm8ea2JCTRGJ0hEfPa2RzYxWZibN4+4Rnp2SOpqtvkHv/eWhoDa9Dw7P76/jyXw/wzF7nFMZfv1VqmGQd0xEZZmZDTjI7ymbeeerJnAbNXf00dfWzfI4xgy1Ddd9prbuVUs8C31dK3QGsAa4Dzh/teKXUpTiTYnxEa717xGOJODMUvgPYgE/gXNP1754u9/tlLbx5vInr1sxlSUbcGUPH+dlJ5GcnsTgjjh++dJwHtpX7bGFkbZcDpWBRuqR9NyKlFKvmJXDQxxkJ97kaNk9u2pqfncSTdxScNXVitPuEGEt2SjRpcZHsqWzjlo3ZPv3dz+2ro3fQTnJM6FyIBpKE6HDyMuLYbYC92Lxt+dx4osJN7K1u44Or5njknCfbeihu7OK/PxiYSwqmSynFpUvT+EdRLX2Ddq9NQe3ut/HZx/bQ1NlHuFnhcGjCw0w89pkNvHqkkcd3VJ0xyyOQvw/PX5TCz14tprmrn7S46e0rubuylVse2YXdoYkIM01pmcFo0zRfOOhcY2vUUUNDBVsu9wB/BJqAVuBurfVRAKVUFs41WMu11jXAfUAC8LI7uxrwntb6aiAc+CGwFLADJ4DrtdYe3Wuru9/GN549xILUGH5646ox/5A/d+ECDtV28PPXi1kxN55NeemeLMao6qwOclJivNa4iJlbPS+Rd0qasfbbiPXR1KV9NRYSo8PJTY3x6HndHQsT3SfEWJRSbMhJYk+Vb3t+i6otfPM558SI+7eWccGiVPncGtCGnGSe3VeLze4gzKAXVZ4Qbjaxal7iUMeYJ7x5vBGAK6a4ziYYXLo0nScKq9ld2cbFS6Y/hXJ3ZSs7y1u5cHHaGe1D74CdOx7fS1G1hd/etI7ZCVFnBAPhZjN/3VPDoM0RFLM8LliYChSzs6KVD6+eO61z/O7tsqG9PgemEIC+cvgUn39qHw7tnKa2MjOepOgItpc5pzX+6OXjrMxMMFz7bbhgS2vdBlw/xmM1OJNouH/ePM55moENni7fSP/7WjEn23p55v+dN25Qo5TipzeuorTJyj1P7uOWjVl8YOUcr34gaq0OVufIqJaRrZ6fgNZwpK7DZw1wUbVzvZbsIySMaENOMi8fbuAnrxzniuWzffKlWVjRis0198dmD/ye52C1YUEyTxRWc+xUp8+TCvna+uwkHn63gt4BO7MiZt5h+ubxRhanx5Kd4tlOtkBw3sIUosJNvH2iadrBVlG1hU8+XIhDw6/eKuXaVXO4dnUmCvjhS8eoau3h159cMzQSObz9GGvmR6BamZlAXFQYO8paphVsVbV0s7OiFaWc+QwcGtp7B9FaM2zgBDg9irV8bjzvljTz+I6qoWmaGmi1DlDT1jN0n1Hbb8MFW4Fkb1Ubj++s4rbzsjl3wcTz/GdFmPnSpYu4+8l9/OG9Sp4orPZahra+QTuN3Zo8SY5haO4LhkO17T4Jtjp6Biltsk4qE5UQ/uAe4f39OxU8tqPKJ1ks011TYaazybfwnXNda/p2V7YFfbCVn52EzaE5VNvOxhl+Hjt6B9lV0cadF+d6qHSBJSrczPkLU9la3MR39PKzLugn49l9tacv8jW8dLiBFw6eGno83KyYlxQ95vODaZaH2bW5+I5p7LelteZb/zpMVJiZX358DQdqLbxf1sof3q3g0EkL5y5IYVNeOudkJrC1uIkvPr2fQZtjKOHCZcvSea+0BZvdOUr4W9e+dLc8UmjokUMJtqZpZ3kL9zy1j5SYCL7+gaWTfl5FSzcKvD53t7zZigaWSHIMQ0uNjSQzcZbPMhLuO+mclrI2K9Env0+IqWrs7AO830YOt7uyjcgwE//v4lwuyUsPmouiYDM7IYr5ybPYU9XGHRcFd+Cw1pUtdm+1ZcbB1jslzdgcmsuXhd4UQrfNS9N5+0QTFS3dLEyb2owfrTUHXYms3Amf/vTpDTx/oJ6/7TmJxpnl14gjKt5y/sIU3jjWyMm2HuYnjx1kjvTc/jreL2vlB9ev5IoVGVyxIoOvXqH53otHeXxnNbsqLfz27bJRn/uZC3L49rUrRl2zZfSRQwm2pqGo2sKtj+7G5tBEmE0UN3RN+s0tyE0hPMzEgM2BydU74A0ljc5s+ZL23fjWzE8casi9bV+1BbNJDaWdF8JozluYikmV4NC+GWVqtfbz/MF6PrF+Pl+5Ms+rv0vM3IacZN4pbh51ylEwSY6JIDctxiPrtt481khqbARrfLz9jJFcujSd+4CtJ5qmHGxtL2vhSH0nn70gh5TYyKEL+ogwM/86UGfoERVvuWBRKgA7ylv4RHLWpJ7TNaD5wZZj5Gcnccu5p59jMinS46MwKYbWYl2wKJWVmfE8ur1yKInGB1c5pywG4vrw4F1h6kVbTzQNze+3O6a2f1B+dhJP3bGRmAhn+kxvfTiKG6yYFeSE4PzsQLNqXgK1ll7+97UTXk8JW1Qt+wgJY8vPThrKRHj/zeu8/gX69O4aBmwObj/ft9kPxfScm5NMa/cA5c3d/i6K163PTqKoxjKjLWMG7Q62Fjdx6dJ0zCG8TjczcRZ5GXFTTgGvteZ/XysmM3EW91699Iz9FN1rsb5yZZ5PpjsbyeL0WFJjI6c0lfCvJwaw9tv48Q3nnLVmvCA35YztY/7jiiV84+pl/PWu8/hqELy+EmxNw0lLD+DcU2s6vRnrc5L50Kq5HK7tGNoI2dNKGruYE6OICJO32OiiXYufH9hazi2PFHot4HLvI5Tvoc2MhfCWG9Y51xT2DXqnfXQbtDt4orCaixansihdZgEEgg0L/LcXm6/lZyfR3jNIRcv0A8s9lW109dlCegqh2+al6eyubKOrb3DSz3ntaAOHajv48uWLiQw7O1FJqG5or5Ti/IXOdVuT6Qz40/ZK3q+3DW2RNNJYgWuwvL5yJT4KS58e84K3qbOP1442cPHi1BlF25ctS6er3+a1L4xDrr2bAnnzvFDR3uts+IevUfGGEw1d9AzYPbq/lhDesDIzgegIM7sqvfO34PbqkQYaO/v5zAU5Xv09wnNyU2NIjY0Iqc2Ni2awFcKTu6oxm5TPthYxskuXpmNzaLaXtkzqeLtD8/PXS1iYFsMNklTqLOcvTKG5q58fbDk27rXm+2UtfH/LMQC2HDw15rHBEliNRoKtUXQMaD758M5RPxC/21qGza75/nUrZ/ShuHBxKhFhpqG9Lzxpe1kLLdYBaq3aqyMlwjPOX5iK2bX2wJvzvvfXOD8HwdiQieASbjaRn53ErgrvXlA/tqOKnJRoNi3x/r6HwjOUUqzPTg6JzY1zU2NJjA6f9nd4UVUbLx9uwO7QfPbxPSF/LbAuK5HoCDP3by2b1Gvx3P46ypqsfO3KvKDe1226EmaFA/Cn96vGvNZ0ODTffv4o7rGvQbv3OpSNTD49Yxi0a/5RVHvGfSfbenh6dw0fWz+fnBluCBsdEcYFC1N463jTjOZjj+blQ6fTkXpzpER4Rn52Et+4xpnR8itXLPFaMFRUbSE9zpn9UAijK8hNobixi7buAa+c/1BtO0XVFm4/P0f2nAswGxYkU2vp5VRHr7+L4lUmk2JdlnPd1nT8c1/d6YtcuRbgYG0HfYN2jtR3TtgRXVjRwvdfPEpuagwfWDnbh6UMHO7prZrTGxOP9LPXiilvthJmUpgI3a01JNgagwKe31/H0frTKbl//VYpSim+dNkij/yOy5dnUNPWQ1mT1SPnczO5RklC+YMdaG4/L4f4qDCKGzz7WRiuqMZCfnZSUGfwEsGjINe9p5J3LhAf21FFTISZj+bP88r5hfcM328r2OVnJ1HWZKW9Z+qdDqVNzqzE5mmuLw82hRWtuPu2+wYd7CgffTphUVUbn3pkN519NmotveyrafddIQNIQW4KUa68AA4NIy8t/llUy0PvlHPLxiz+dlcBNywOD/hEF9MlwdYokiIVD92aT0J0OLf/cTeVLd2UNXXx7L5abi3IZk6CZ0YGLlvqXLD65vGpZceZyKmOXuYmRoX0BzvQRISZuGL5bN441uCVpClNnX2cbOuVz4IIGOdkJhIVbqLQC1MJ3zreyPP767kkL424qHCPn19417I5cUSFmfjT+5VBPzVunSuh0UTrYkYqb7ayt9rCx/IzQzJb3mgKclOIDDfhjgl2lrUyaD/z+7bfZuc7LxyddsbpUJKfncSTdxbwpcsWsTg9ll++XsLrRxsA2FvVxjefPcz5C1P47odXkJ+TzIcWRoTsZ1CCrVEkRCquWjGbJz63EYeGjz20gzse30uE2cQ9mxZ67PfMTohiZWa8R9dtORyavdUWLlqUFtIf7EB0zTmz6eyzjdnbNhP/KDoJICnfRcCICHOt2/Lw6EVRtYX/90QRdq1563hT0F+sB6ODtR0M2B0cONkR9OuS3csMnt1XN6W6PvJeBRFmE/devSxokw5MlTvj3deuyuPT52ezo6KVu/+yj36bHYAWaz+femQXR+o7CTMpGRGchPzsJL5yRR7/vOd8VmYm8Pmn9vE/Lx3jtkd3kxwTzgO3rCNc1rtJsDWeRemxfPMDS2mxDlDV2oPNoalq7fHo77hsaQb7aiy0Wvs9cr7SJisdvYOsz5GGNdBcuDiV2MgwXjnc4NHzFlVb+MUbpQB878WjQX1hIoLLxgUpnGjopKNn8qmaJ1JY0TrUa20L0cXagW74dLBgX4u037Xh/VSy1TZ39fPPfXXcmD+P1NhI7xYwwLgz3n33wyv5wfUrefN4I5/4fSH//dxhPvCrdzlU28Fvb1rL3/7feTIiOAXxUeH8+XPnkp0Swx/eq6Rn0I6lZzAk9sObDAm2JtBk7ce9dlpr7fFG/fJlGWgNW4ubPXI+dyr5c117kYjAERlm5vJl6bx2rOGsqQ0zUVjRit11cRnsFyYiuGxckIzWeDTz3EZX26iQXutAVZCbQpjZ+cVsNgX3e1iQm0KY6yIkzDy5uj6+o4pBu4M7L8r1dvEC2q0F2Xx+00IOnGznL7tqaLEO8IPrVnDt6rlBnYbcW+KjwrnmnNPJRKQz6zQJtiYwfFdrb3wxr8yMJyM+krc8NJVwT1UbaXGRZCVHe+R8wreuPmcO7T2DHk15vd71ZSEXlyLQrJ6fSGSYyaNf2LFRzqm0V6+cLb3WASo/O4mHPpUPwM0b5wf1e5ifncTDt+WjFFy9cs6Ede0ZsPFEYTVXLs9gwQyzJoeC6MiwoTVcZgXNVu9kPw0VlyxJJyrce9fMgUoWcEzAPce3sKKVgtwUjzfqSikuXZrBCwfq6LfZR92hfCr2VlnYkCMZ5wLVJUvSiI4w8/KRU1y4ONUj54wKd36mrl09l9vPzwnqCxMRXKLCzazNSvTo5sZ7XBvEfuPqZWSlSKdUoLpsWQZzE6Jo7fbcFFOjunRpBpvz0tlV2YrDocfdquCZPSfp6B3krotlVGsy3EkzBm0OCQ48wNvXzIFKRrYmwdvDyVcsT6d7wM5/P3dkRutp6tp7qWvvZUOOTCEMVFHhZi5dms5rRxqGpv7NlHsz429es1QaPhFwNi5I4Vh9J519nrmoLqpqIz0ukvnJst9coDtnXgKHa9v9XQyf+MjaTE519I07ymuzO3hkeyX52UnkZ8t1wGS4gwNZn+U5MgXzbBJsGcAs18jD34tqZ5RZaa9rXYMEW4HtmnPm0No94LHe/P0n25kdH+WxLQuE8KWNuck49On2bab2VFnYkJMso/9BYNW8RKpae6aVQOVnP4OtW8c/xvn43IxpFc7DrlieQVxkGM/urxvzmAe2lVNr6eWyZek+LFngk+BAeJvhgi2lVLJS6jmlVLdSqlopdfM4x35aKWVXSlmH3TZN51z+NHzDvJkkMNhT1UZsZBhLZ8d5qGTCHzblpRFhVvzs1WKPZA7cX9PO2qzEmRdMCD9Yl5VEhNnkkXWM9a7Rf7moCg6r5iUAcLiuY8rP3bABPv7xsQOurVudj0O3Z1MQT1NUuJmrz5nNK4dP0TtgP+vxvVVt/N+bJQD85q1SyTorhIEYLtgC7gcGgAzgFuBBpdSKcY7fqbWOHXbbNoNz+UVBbgoRrn0ITCY17TnDe6ssrM1KJEz2NAhox091YXNoDpxsn/EeMi3WfmraeiTYEgErKtzMmvmJHkmSsdf1tySj/8FhVWYiAAenMZVw82Z45pnRAy53oPXMMwAdXTMtp6d8ZO08ugfsvH7s7O1BfvNWWcikwxci0BjqqlwpFQPcCNyntbZqrbcDLwC3+vNc3pafncTTd24kOSacBakx0+p17egZpLixSy4igsDwPWQGZvilud81aro2S3ryReDamJvM4boO/u+Nkhl1PuytaiM6wsyyOTL6HwwSosPJSYnmcO3UR7Zg9IBreKC1ebMHC+sBGxckMzchimf3nTmVsKjawvayZkwKyQInhAEp9+7kRqCUWgvs0FrPGnbf14BLtNbXjnL8p3GOXvUCbcATwI+11rZpnOsu4C6AtLS0/GecXVo+9XrVIE+dGOB750eRHT+1rIQHmmz8al8/926IYlmK87lWq5XY2FhvFNVnQrEOZRY7P93Tx6ADTAr+69woFiVNL0vlP0oGeKVykAcujybSPP01KsH6PmzevLlIa71+Ms83QhvhSYH0nrrbRgWEm+DrG07/TUylHt9+v5e4CPjPDcZavxhI78VY/FWHBw/0Udru4Jebpp9Zcv/+RL73veV84ANVvPpqDt/5zjHWrm0HjNdG/L14gJcrB/m/zbNIjDTRPaj59vu9KAW3r4igusPB0mTztL8zhpPPpXEEQz2CtQ6TaiO01oa5ARcBDSPuuxPYNsbxucACnCN05wDHgG9O51zDb0uWLNH+0N49oJd862X9zWcPTfm5P3nluF74zZd0T79t6L6tW7d6sHT+Eap12FvVps//8Zt68/++PaPf/cnf79Qf+s17MzqH1sH7PgB79TTaKn+1EZ4USO/pr94o1tn3btHZ927Rud/Yon/3dunQY5OtR0fvgF7wjS36l68Xe6mU0xdI78VY/FWHP7xbrrPv3aKbOvtmdJ777tManP8OZ7Q2oqShU2ffu0X/4d1y7XA49L89sVcv/OZLel91m8d/l3wujSMY6hGsdZhMG+HTaYRKqW1KKT3GbTtgBeJHPC0eGHXOtNa6QmtdqbV2aK0PA98HPup6eErnMoKE6HCuXT2X5/fXYe23Tem5eyrbWJmZwKyImfdmCf/Lz07i5o3ZVLT00NTVN61z2B2ag7WSHEMEvgsXp+HeWmi6U6T217Tj0LJeK9ick+lOktE+7XNs3QoPPgi33lrFgw9OnKXQnxZnxLEyM57n9tfx5K4aXjnSwH9elSdTxYUwMJ8GW1rrTVprNcbtQqAECFNKLR72tNXA0cn+ChjaDHym5/KLWzZm0T1g51/jpHcdqW/QzqHaDs5dIBcRweSSJWkAvFvSMq3nlzR20TNgZ518CYsAl5+dxEfz56GAh29dP611rXur2jCbFGuk8yGorMxMQCk4NM11W8PXaH32s1VjJs0wko+sncfR+k6+/fwR1sxP5M6LZANjIYzMUAkytNbdwLPA95VSMUqpC4DrcK7FOotS6mqlVIbr/0uB+4Dnp3Muo1gzP5EVc+P5S2G1e+rjhA7XdTBgd7Be0hkHlRVz40mLi2RbcdO0nn86OUai5wolhJ/cuG4eGui3Oab1/D1VbSyfE09sZJhnCyb8KiYyjEVpsdMKtkZLhjFelkKjyElxrk9zaDh+qpP9J9v9WyAhxLgMFWy53APMApqAp4G7tdZHAZRSWa69tLJcx14GHFJKdQMv4wyufjSZcxmVUopbNmZzoqHrjP23xrO70rn/zHqZHhNUlFJcsiSN90pbsNmnfoG5v8ZCckwEWcnTXzguhFGsyUokKtzEjvKpj/QO2h0cONnO+hzpkApGq+Ylcqi2Y9IdlDB+1sHhARckGC515YmGrqEpPDa7pHkXwugMF2xprdu01tdrrWO01lla66eGPVajnXtp1bh+/prWOsN1bK7W+tta68HJnMvIrlszl9jIMJ7cVT2p49863khyTASVLd1eLpnwtU15aXT0Dk5rH5n9J9tZOz8RpaafhVAIo4gMM7MhJ5md5VO/sDxa30nfoIP12dIhFYxWzUugxdrPqY7Jr2/ds2f89O7ugAtiDNdbVZCbQmS4SdK8CxEgDBdsCee0iI+szeSFg/X84vXicfeV2V3Zyr6adtq6B2a8Aa4wnosWORMDbD3RPKXndfQMUtZklSmEIqictzCFEw1dtFj7p/S8vVXu0X8Z2QpGq+Y5k2RMZSrh178+8T5azsfrG6dfMu/Iz07iyTsK+MqVeTx5R8G01jAKIXxHgi2DWpuViM2u+e3bZdzyh9GDqLr2Xv79rweGfpZd44NPQnQ4+dlJbCuZ2rqtA66RMMlQJYLJ+QtTAabczu2paiMrOZqM+ChvFEv42bI58YSZFIemMQMgUOVnJ/H5zYsk0BIiAEiwZVDDp0P02Rz8dXfNGY+/dbyRD/7mPSw9g4SblUwnCGKb8tI5Utc5pRTw+2ssKHW6x1eIYLByrjPBxY4pTCXUWrO3yiKjWkEsKtxM3uw4DtdNLyOhEEJ4k6RlMqiC3BSiwk0M2BxoDX8vqqVnwM61q+fwx+2V7K6ysHxOPPffso627gEKK1opyE2RXq4gdMmSNP73tWLeLWnho/nzJvWc/TXtLEmPIy4q3MulE8J3wswmNi6Y2rqtlw6forV7gPTYSC+WTPjbqnmJvHz4FFprWacqhDAUCbYMyj0nu7CilfzsRPZUWvjN26W8dPgUAGaT4r4PLWNBagwLUmMkyApiw1PATybYcjg0B062c/XK2T4onRC+dd7CFN460UR9ey9zE2eNe2xRtWVoqvWfdlRxxYrZ0lYGqVXzEnh6dw3VrT3kpMb4uzhCCDFEphEamHtOdkFuKl+8bDGfOX/B6Qe1nnRqeBHYppoCvrK1m47eQUmOIYKSe93WZEa3Xj/WgM3hTAcuKbKD21CSDJlKKIQwGAm2AshVK2cTJeleQ9LmvHQ6egc5MInNK5/bXwc4U2ULEWyWzo4jKTp8wnVbWmuKqpyJhUzSZga9JRlxhJsVf95RJVl5hRCGItMIA8jwqYWyPiu0XLg4FbNJsa24edzNq4uqLTywtQyAbzx7iPnJ0fI5EUHFZFKctzCFneUt425i++qRBvZWW7jtvGwy4qOkzQxyh2o7sDk0e6st3PJIoaREF0IYhgRbASY/O0m+QEJQwqxwlqTH8veik2xemj7mZ+Dp3TW4Zk0NbQUgnxcRbM5bmMrLhxuobu0Z9fHOvkG+88JRls2J574PLSfcLJM4gl1hRSu42r5+afuEEAYi30BCBICiagulTVYaO/u56eHR913rG7SzvbQZBTLVVAS18xc6P9djTSX8+WvFNFv7+fEN50igFSIKclOIDHe+11o7EwsJIYQRyLeQEAGgsKIVh2vK1IDdwetHG8465oFt5TR09vOda5fzlSvzZBqNCFq5qTFkxEeyo7zljPuLqi1867nD/HlnNbefl8Oa+Yn+KaDwOfc0+0+fn4NZwcuuzL1CCOFvMo1QiABQkJtCRJhz3zWHhlePNvDlyxcTHeH8E65otvLQtnKuWzOXT1+wYIKzCRHYlFKcvzCVt443EtmriFtgoWfAxh2P76Xf5szYedmydD+XUviae5r9rAgzD24r54Z180Yd3S+qamNnRSvnLUyVDikhhNfJyJYQAcDda/vVK/P41geXcbKth68+cxCHQ6O15tvPHyUy3MS3PrjM30UVwifmJkbR2Wfjn6WD3PjgDm59dPdQoGVSzoQJIjR96dLFzE+exbeeO0y/zX7GY8/uq+Vjv9/JL14v4ZZHRp+SLYQQniQjW0IEiDOSo2j4n5eP89u3y1iQFsP2sha+f90K0uOi/FtIIfxAAavnJ3K0vgO7QxMh6xVD2qwIM9+/biWf+dMefv9OBV+6bDFNXX388vUS/rbnpDuPhiQREkL4hARbQgSgOy5awPGGTv7vzRIiwkwsTIvhlo3Z/i6WED5z6dIMHt1eycCgg4hwE/d9aDmAbI0hAOfehB9aNYffvF3K9tJmDtZ24NCaD66awxvHGrHZHZJESAjhExJsCRGAlFJ8LH8ez+2vY8Dm4KSllwMn2+UCU4QM99Tap9/cw02Xbxj67MvfgHC7fk0mWw6dYneVBZOC3960lg+umktRtUWCciGEz0iwJUSA2lfTjsK5tYzdLtNhROjJz06ia2GEfO7FqIobuzApcGjnVNMq175ssl+lEMKXJEGGEAHKnaFQ9tQSQoizSRsphDACw41sKaWSgUeBK4EW4Jta66fGOPYh4FPD7goHBrTWca7HtwEFgM31eJ3WOs9LRRfCp9zTqGQ6jBBCnE3aSCGEERgu2ALuBwaADGAN8JJS6qDW+ujIA7XW/wb8m/tnpdRjgGPEYV/QWj/itdIK4UcyHUYIIcYmbaQQwt8MNY1QKRUD3Ajcp7W2aq23Ay8At07huY97t5RCCCGEEEIIMTGltZ74KB9RSq0FdmitZw2772vAJVrrayd47m3Ad4GF2lUp1zTCFTjXxhYD39Jabxvj+XcBdwGkpaXlP/PMMzOtjt9ZrVZiY2P9XYwZkToYQ7DWYfPmzUVa6/WTeX6wtRHB8J5CcNRD6mAM0kacKVjf00AUDPUI1jpMqo3QWhvmBlwENIy4705g2ySe+xbw3RH3bQTigEjgdqALZzA27rmWLFmig8HWrVv9XYQZkzoYQ7DWAdirp9FWBUMbEQzvqdbBUQ+pgzFIG3GmYH1PA1Ew1CNY6zCZNsKn0wiVUtuUUnqM23bACsSPeFq8K0ga77zzgUuAPw+/X2u9S2vdpbXu11o/DrwPXOO5GgkhhBBCCCHE6HyaIENrvWm8x13rrsKUUou11qWuu1cDZyXHGOE2nNMPKyYqAs4phUIIIYQQQgjhVYZKkKG17gaeBb6vlIpRSl0AXAc8McFTbwMeG36HUipRKXWVUipKKRWmlLoFuBh4zQtFF0IIIYQQQogzGCrYcrkHmAU0AU8Dd2tX2nelVJZSyqqUynIfrJQ6D5gH/H3EecKBHwLNOPfr+iJwvda62PtVEEIIIYQQQoQ6w+2zpbVuA64f47EaIHbEfTuBmFGObQY2eKGIQgghhBBCCDEhI45sCSGEEEIIIUTAk2BLCCGEEEIIIbxAgi0hhBBCCCGE8AIJtoQQQgghhBDCCyTYEkIIIYQQQggvkGBLCCGEEEIIIbxAgi0hhBBCCCGE8AIJtoQQQgghhBDCCyTYEkIIIYQQQggvkGBLCCGEEEIIIbxAgi0hhBBCCCGE8AIJtoQQQgghhBDCCyTYEkIIIYQQQggvkGBLCCGEEEIIIbxAgi0hhBBCCCGE8AIJtoQQQgghhBDCCyTYEkIIIYQQQggvkGBLCCGEEEIIIbzAcMGWUuoLSqm9Sql+pdRjkzj+P5RSDUqpDqXUH5VSkcMeS1ZKPaeU6lZKVSulbvZq4YUQQgghhBDCxXDBFlAP/BD440QHKqWuAr4BXAbkALnA94Ydcj8wAGQAtwAPKqVWeLi8QgghhBBCCHEWwwVbWutntdb/AloncfjtwKNa66NaawvwA+DTAEqpGOBG4D6ttVVrvR14AbjVKwUXQgghhBBCiGHC/F2AGVoBPD/s54NAhlIqBcgC7FrrkhGPXzLaiZRSdwF3uX7sV0od8UJ5fS0VaPF3IWZI6mAMwVqH7Mk+OQjbiGB4TyE46iF1MAZpI84UrO9pIAqGegRrHSZsIwI92IoFOob97P5/3CiPuR+PG+1EWuuHgYcBlFJ7tdbrPVtU3wuGekgdjEHqEHxtRDDUAYKjHlIHY5A24kxSB+MIhnqEch18Oo1QKbVNKaXHuG2fximtQPywn93/7xrlMffjXdP4PUIIIYQQQggxJT4NtrTWm7TWaozbhdM45VFg9bCfVwONWutWoAQIU0otHvH40enXQAghhBBCCCEmx3AJMpRSYUqpKMAMmJVSUUqpsaY7/hn4nFJquVIqCfhv4DEArXU38CzwfaVUjFLqAuA64IlJFOPhmdbDIIKhHlIHY5A6eO9c/hIMdYDgqIfUwRikjTiT1ME4gqEeIVsHpbX2dEFmRCn1XeA7I+7+ntb6u0qpLOAYsFxrXeM6/ivAvcAs4J/Av2mt+12PJeNMIX8FzuyG39BaP+WTigghhBBCCCFCmuGCLSGEEEIIIYQIBoabRiiEEEIIIYQQwUCCLSGEEEIIIYTwAgm2hlFKJSulnlNKdSulqpVSN/u7TBNRSn1BKbVXKdWvlHpsxGOXKaVOKKV6lFJblVKT3pzRl5RSkUqpR12veZdSar9S6uphjwdKPf6ilDqllOpUSpUope4Y9lhA1MFNKbVYKdWnlPrLsPsCog6uLSb6lFJW16142GMzqoO0Ef4hbYTxSBsx5rmljfADaSOMR9qI0yTYOtP9wACQAdwCPKiUWuHfIk2oHvghzkQgQ5RSqTizMd4HJAN7gb/5vHSTEwacBC4BEnCW+RmlVE6A1ePHQI7WOh74MPBDpVR+gNXB7X5gj/uHAKzDF7TWsa5bHnisDtJG+Ie0EcYjbcTopI3wD2kjjEfaCDettdycSUJicDaQS4bd9wTwE3+XbZLl/yHw2LCf7wJ2jKhfL7DU32WdZH0OATcGaj2APOAU8PFAqwPwSeAZ4LvAXwLt8wRsA+4Y5f4Z1UHaCGPdpI3wa9mljRj9vNJGGOgmbYRfyy5txLCbjGydtgSwa61Lht13EDB6j9RYVuAsPzC071g5AVAfpVQGzvfjKAFWD6XUA0qpHuAEzkbyZQKoDkqpeOD7wFdHPBQwdXD5sVKqRSn1vlJqk+u+mdZB2giDkDbCf6SNGJe0EQYhbYT/SBtxNgm2TosFOkbc1wHE+aEsnhCQ9VFKhQNPAo9rrU8QYPXQWt+Ds2wX4Rxq7iew6vAD4FGt9ckR9wdSHe4FcoFMnBsQvqiUWsjM6xBIr8FkBGR9pI3wO2kjxhZIr8FkBGR9pI3wO2kjRpBg6zQrED/ivnigyw9l8YSAq49SyoRzysUA8AXX3QFXD621XWu9HZgH3E2A1EEptQa4HPi/UR4OiDoAaK13aa27tNb9WuvHgfeBa5h5HQLmNZikgKuPtBH+JW3EhALmNZikgKuPtBH+JW3E6CTYOq0ECFNKLR5232qcQ9CB6CjO8gOglIoBFmLQ+iilFPAozkXFN2qtB10PBVQ9RgjjdFkDoQ6bgBygRinVAHwNuFEptY/AqcNoNKCYeR2kjfAjaSMMYRPSRoxH2gg/kjbCEDYhbcQozzbAQjSj3IC/Ak/jXPB2Ac6hwRX+LtcEZQ4DonBmsHnC9f8wIM1V/htd9/0UKPR3ecepx0NAIRA74v6AqAeQjnNBaCxgBq4CuoHrAqgO0cDsYbefA/9wlT9Q6pDoeu3dfwe3uN6HPE/UQdoIv9ZD2gj/10HaiInPL22E/+ohbYT/6yBtxGjn9HeljHTDmcbxX64XtQa42d9lmkSZv4sz4h5++67rsctxLrDsxZlZJcff5R2jDtmucvfhHKJ1324JlHq4/gDfAdqBTuAwcOewxw1fhzE+W38JpDq43oc9OIf0211fvFd4qg7SRvitDtJGGPAmbcSo55c2wj91kDbCgDdpI5w35XqiEEIIIYQQQggPkjVbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlhBCCCGEEEJ4gQRbQgghhBBCCOEFEmwJIYQQQgghhBdIsCWEEEIIIYQQXiDBlgh5Sql4pdR3lVLL/F0WIYTxSBshhBiPtBFiPBJsCQHrge8A4f4uiBDCkKSNEEKMR9oIMSYJtoSAtUA/cMzfBRFCGJK0EUKI8UgbIcaktNb+LoMQfqOUOg4sHXH3P7XWH/VHeYQQxiJthBBiPNJGiIlIsCVCmlJqA/BX4CjwI9fdp7TW1f4rlRDCKKSNEEKMR9oIMZEwfxdACD87CMwDfqu1LvR3YYQQhiNthBBiPNJGiHHJmi0R6lYAEcA+fxdECGFI0kYIIcYjbYQYlwRbItStAzRwwM/lEEIYk7QRQojxSBshxiXBlgh1a4FyrXWnvwsihDAkaSOEEOORNkKMS4ItEeqWI6lahRBjkzZCCDEeaSPEuCRBhgh17cA6pdRVQAdQqrVu9W+RhBAG0o60EUKIsbUjbYQYh6R+FyFNKbUSeBRYBUQBF2mtt/u3VEIIo5A2QggxHmkjxEQk2BJCCCGEEEIIL5A1W0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeIEEW0IIIYQQQgjhBRJsCSGEEEIIIYQXSLAlhBBCCCGEEF4gwZYQQgghhBBCeMH/B8g2n7To6U4JAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# samples of the valid sets:\n",
    "\n",
    "def plot_series(series, y=None, y_pred=None, x_label=\"$t$\", y_label=\"$x(t)$\"):\n",
    "    plt.plot(series, \".-\")\n",
    "    if y is not None:\n",
    "        plt.plot(n_steps, y, \"bx\", markersize=10)\n",
    "    if y_pred is not None:\n",
    "        plt.plot(n_steps, y_pred, \"ro\")\n",
    "    plt.grid(True)\n",
    "    if x_label:\n",
    "        plt.xlabel(x_label, fontsize=16)\n",
    "    if y_label:\n",
    "        plt.ylabel(y_label, fontsize=16, rotation=0)\n",
    "    plt.hlines(0, 0, 100, linewidth=1)\n",
    "    plt.axis([0, n_steps + 1, -1, 1])\n",
    "\n",
    "fig, axes = plt.subplots(nrows=1, ncols=3, sharey=True, figsize=(12, 4))\n",
    "for col in range(3):\n",
    "    plt.sca(axes[col])\n",
    "    plot_series(X_valid[col, :, 0], y_valid[col, 0],\n",
    "                y_label=(\"$x(t)$\" if col==0 else None))\n",
    "save_fig(\"time_series_plot\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2FaWwFESFpaX",
    "outputId": "50755831-783f-4d98-8d00-56c4a19a4d88"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.020211367"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Naive predictions (just predict by the last observed value):\n",
    "\n",
    "y_pred = X_valid[:, -1]\n",
    "np.mean(keras.losses.mean_squared_error(y_valid, y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oUBy3jOEFpaY",
    "outputId": "7d697978-6dbb-4f15-bccd-0c96015f809c"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA++ElEQVR4nO3deXiU1dn48e89mSyQkJCNsCYQloRFAQM1iEpwgaq12qp1QautSuvSX99WW+1ia62trW93X19bXOpSd+tuXwWVKCBbooAkkACBhC0bCWQj28z5/TGTGEL2zPJM5v5c11wyz3LmzHEm9zzPOec+YoxBKaWU8jabvyuglFIqOGjAUUop5RMacJRSSvmEBhyllFI+oQFHKaWUT2jAUUop5RMacJRSSvmEJQOOiNwuIjki0iQiT/Zy7A9EpFREjonIEyIS3mFfnIi8JiL1IlIsItd4vfJKKaW6ZMmAAxwC7gee6OkgEVkK3A2cC0wEUoFfdTjkYaAZSAKWAY+IyEwv1FcppVQvxMqZBkTkfmC8MeaGbvY/B+wzxvzU/fxc4FljzGgRiQSqgVnGmEL3/meAg8aYu33yBpRSSrWz+7sCgzQTeKPD861AkojEA8mAoy3YdNi/qKuCRGQ5sBwgIiIiIzk52Ts1DiBOpxObzaoXwb6j7aBt0EbbwaW7digsLKw0xiR2d16gB5wo4FiH523/HtHFvrb9I7oqyBizAlgBkJaWZgoKCjxb0wCUnZ1NVlaWv6vhd9oO2gZttB1cumsHESnu6bxAD9V1QHSH523/ru1iX9v+Wh/USymlVCeBHnDygNkdns8GyowxR4BCwC4iUzvtz/Nh/ZRSSrlZMuCIiF1EIoAQIEREIkSkq9t/TwM3isgMEYkFfg48CWCMqQdeBe4TkUgRWQhcAjzjkzehlFLqBJYMOLgCx3FcQ56vdf/75yKSLCJ1IpIMYIx5F3gQWA0Uux+/7FDOrcAwoBx4HrjFGKNXOEop5QeWHDRgjLkXuLeb3VGdjv0T8KduyqkCLvVg1ZRSSg2QVa9wlFJKDTEacJRSSvmEBhyllFI+oQFHKaWUT2jAUUop5RMacJRSSvmEBhyllFI+oQFHKaWUT2jAUUop5RMacJRSSvmEBhyllFI+oQFHKaWUT2jAUUop5RMacJRSSvmEBhyllFI+oQFHKaWUT2jAUUop5ROWDTgiEicir4lIvYgUi8g13Rz3d/ey022PJhGp7bA/W0QaO+wv8N27UEop1caSS0y7PQw0A0nAHOAdEdlqjMnreJAx5rvAd9uei8iTgLNTWbcbYx7zam2VUkr1yJJXOCISCVwG3GOMqTPGrAXeBK7r43lPeb+WSiml+sOSAQeYBjiMMYUdtm0FZvZy3mVABfBxp+0PiEiliKwTkSyP1VIppVSfWfWWWhRwrNO2Y8CIXs67HnjaGGM6bLsLyMd1e+4q4C0RmWOM2dPxRBFZDiwHSExMJDs7e+C1HyLq6uq0Hei6HXZXO8g/4mBGfAhTYkP8UzEf0s+Ci7aDy0DbQU7822wNIjIXWGeMGd5h2x1AljHm4m7OmQDsBaYZY4p6KPtd4B1jzEPdHZOWlmYKCnRsQXZ2NllZWf6uht91bofc4mquWrGeFoch3G7juZszyUiJ9V8FfUA/Cy7aDi7dtYOI5Bpj5nV3nlVvqRUCdhGZ2mHbbCCvm+MBvgl80lOwcTOADLJ+Koh9XFhBi8P1Q62p1cmGoiN+rpFSgcGSAccYUw+8CtwnIpEishC4BHimh9O+CTzZcYOIjBSRpSISISJ2EVkGnA2856WqqyBQXtN4wvOIUEt+jZSyHCt/U24FhgHlwPPALcaYPBFJds+nSW47UEQWAOOBlzuVEQrcj2sgQSXwPeBSY4zeL1MDcrShmbe3HWb+xFh+eP5URkeH8/ym/bQ4Oo/EV0p1ZtVBAxhjqoBLu9hegmtQQcdt64HILo6tAOZ7qYoqCP39oyLqmlv59aWzSB8dzYwxMdz0dA7PrC/m22dO8nf1lLI0K1/hKGUp5TWNPPnJXi6ZPZb00dEAnDt9FGdPS+TP7xdypK7JzzVUyto04CjVR/+zejetDsN/nTetfZuI8IuvTOd4s4M/rCzs4WyllAYcpfpgf1UDz28q4RvzJzAx4cS7t1NGjeCbCybywuYS8g51nj6mlGqjAUepPvjL+7sQEf7fOVO73P/986YSOzyMX72ZjxXntillBRpwlOrFwTonr312gOsXpDA6JqLLY2KGhXLnkjQ27avi7W2HfVxDpQKDBhylevHarmaGhYZwS9aUHo+7cv4EZo6N5ldv5vHX9wvJLa72UQ2VCgwacJTqwUubS8gpc3DRqWOIiwzr8dgQm3D1/GQq65v58/u7WPbYBg06SnWgAUepbuQWV3P3q58D8OaWQ30KHscaW9rzJjW2OFm7q8KLNVQqsGjAUaobq/JLcbr7/1scfcuZlpkaT3iorT3ovLX10EmpcJQKVhpwlOqOO9gIEGq3kZka3+spGSmxPHtTJncuTePuC9I5dKyRSx9eR/6hGu/WVakB2LT3CA+v3u2zW7+WTW2jlL/trqgnISqMRWMM15w3v89LEGSkxLYfe9bUBG58MofL//4Jf7tqLufNSPJmlZU6SW5xNRuKjpCZGs+UUVFs2lvFJ3sq+SC/nJLqBgDCQmw8f/PpZEyM82pdNOAo1YXGFgfrdldyecZ4zh1ZOeD1bmaOjeGN2xdy01M53PR0DovTErn9nKlDfv0cZQ25xdVc8+gGmludrkt147pwjwi1MTo6om0TzQ4ntz73Kb/4yky+PGs0IbZuVnB59ln42c9YVFICycnwm9/AsmV9ro8GHKW6sHFvFcdbHJyTPgpKKwdVVlJ0BD+5IJ1rH9/I6oIK1u05wvNBsGib8r8NRUdoanVnMjewYHI83z93KnOTR7L9YA3LHttAS6sTm00Q4LbnPmVi/HCWzhzN8LAQzpya+MXn9NlnYflyaGhw9VEWF7ueQ5+DjvbhKNWF1TvLiQi1sWBy7/02ffHZ/qPt/27RRduUj0xOdKVhElxXNXcuSXMNbLGHtPc3/nBJGi8sX8C6u8/lf5edRohN+MfHRScP7f/Zz6Ch4cQXaGhwbe8jvcJRqhNjDB/uLOeMyQlEhIZ4pMzM1HjC7DYaW5yI0KcBCEoNVlmNK4P5jWdO4oJTxpx0Vd2xvxHgwlPGsLeyjj+uLMRpvvhxlJESCyUlXb9Id9u7oFc4SnWyp6KekqoGFqeP8liZbb8m00aPIC4yTG+nKZ9YlV9GamIkP//KjD5/5jJTEwiz2wiRTqMzk5O7PqG77V3QgKNUJ6t3lgO4+m88KCMllisyxlNZ16xzc5TX1TS2sKHoCOf3c2Rkx1ttz97Uoa/xN7+B4cNPPHj4cNf2PtKAo1QnqwvKSUsawbiRwzxe9tzkkcCJfTpKecNHBRW0Og3nT+//UPyMlFhuWzzlxKuiZctgxQpIScGIQEqK63k/RqlZNuCISJyIvCYi9SJSLCLXdHPcDSLiEJG6Do+s/pajFEBtYwub9laRlZ7olfJnjo0hNETYogFHedn7O8qIjwxjbrIHb98uWwb79vHRhx/Cvn39CjZg7UEDDwPNQBIwB3hHRLYaY/K6OHa9MeZMD5SjgtzaXZW0Og3npHn2dlqbiNAQpo+J5rMSTeqpvKfF4WT1znKWzuxhTo0fWPIKR0QigcuAe4wxdcaYtcCbwHX+KEcFjw93lhMdYfdqp/7cCSPZduAYDqcu1Ka8Y/PeKmoaWy2X2cKqVzjTAIcxpuMi8VuBRd0cP1dEKoEq4BngAWNMa3/KEZHlwHKAxMREsrOzB/0mAl1dXV1QtYPTGN77/DjpcTbWrvm4fbun2yG8vpWGZgfPvbOaCSMs+ZvvJMH2WehOoLTDszuasNuAwzvIrtjp8fIH2g5WDThRQOfF4Y8BI7o49mNgFlAMzAReBFqBB/pTjjFmBbACIC0tzWRlZQ289kNEdnY2wdQO2w4cpea9dVx19iyyThvfvt3T7TCxsp4V27IJGTWFrNP7PqTUn4Lts9CdQGgHYwz3bFrN2dPiWHrefK+8xkDbwao/r+qA6E7booHazgcaY4qMMXuNMU5jzOfAfcDl/S1HqQ93liMCi6Z5Z8BAm5T44cQOD2XLfu3HUZ5XUFbL/qrj/R4O7QtWDTiFgF1EpnbYNhvoS0e/gfblSAZTjgoyq3eWM2fCSOKjwr36OiLCnAkj+azkqFdfRwWn9/PLADjXw/PIPMGSAccYUw+8CtwnIpEishC4BFf/zAlE5AIRSXL/Ox24B3ijv+Wo4FZR28TWA8e8Njqts7nJseyuqKOmscUnr6eCx6od5cyeMJJR0RH+rspJLBlw3G4FhgHlwPPALcaYPBFJds+1abv5fS6wTUTqgf/gCjC/7a0cX70JFRiyC1zZBTyZzqYncyaMxBjYtr9zF6NSA1de08jW/Uc5f7r1rm7AuoMGMMZUAZd2sb0E12CAtud3Anf2txylOsouqGDUiHBmju3c5ecdsyeMBOCzkmrOnJrgk9dUQ9/7O1w/nM6fMdpjZT74IMyfD4sXd3/M6tWweXPvZVn5Ckcpn2hxOPm4sILFaaMQ8c0kuZhhoUxOjNSMA8qj3t9RxoS4YUxLiur94D6aPx++8Q1XUOnK6tWu/fP7MCBOA44Kes9tLKG2qZWU+OG9H+xBc5Nj+Wz/UYzRCaBq8BqaW1m7u5Lzpid59IfT4sXw0ktdB522YPPSSz1fAbXRgKOCWm5xNfe9nQ/A3z7Y9cViUz4wN3kkVfXN7K867rPXVEPXml2VNLc6vTIcuqug099gAxpwVJBbt7uyPcVMi8O3K3HOaevH0fk4ygNW5ZcRHWFn/sQ4r5TfMeg88cTEfgcb0ICjgpzd/Q2wdV5sygfSkkYwLDRE5+OoQdu8r4p3th1m9oSRhIZ478/64sVwyy3wzDMTueWW/gUb0ICjglxO8VHihofxw/OnnbjYlA/YQ2ycMj5G18ZRg5JbXM2yxzZyvMXBhqIjXr0tvHo1PPIIXHfdPh55pPuBBN3RgKOCVumxRrILyrn69Ancfs5Uvyz7PDd5JPmHjtHY4vD5a6uhYUPREVpanQA4ncZrt4U79tl8+9v7uh1I0BMNOCpo/fvTAzgNXJExwW91mDthJC0OQ/7hGr/VQQW2zNR42galeeu2cFcDBHoavdYdDTgqKBljeDlnP1+aFMfEhEi/1aNtNUbtx1EDNXfCSCJCQzh1XIxXbgv3NBqtv0FHA44KSpv3VbPvSANXzvPf1Q1AUnQEY2MidAVQNWC7K+poaHZw3YIUr9wW3ry559FobUFHMw0o1Y0XN+8nKtzOBad4LgXIQM1JHqkZB9SA5exz/ViZ56Xh0D/+ce+j0RYvdh3XGw04KujUNrbwn88Pc/HsMQwP8386wbkTYjlQfZyK2iZ/V0UFoJziKuIjw5jo40wZA6EBRwWdd7Yd5niLgyv8fDutzZzkkQB6laMGJLe4moyUWJ/lARwMDTgq6LyYs58po6KY657p72+zxsYQIvDYmiKfptZRga+itoniIw3Mm+j7If0DoQFHBZVdZbV8VnKUK+dNsMwvwvzDNTiBjXurWPbYBg06qs9yi6sAyEjxTv+Np2nAUUHl5dwD2G3CpXPH+bsq7TYUHXEtjA60tPo2n5sKbDn7qgmz25g1zjfrOA2WBhwVNFocTl799ADnpI8icUS4v6vTLjM1nlB3UjebTXyaz00FtpziauaMH0m4PcTfVekTywYcEYkTkddEpF5EikXkmm6Ou15EckWkRkQOiMiDImLvsD9bRBrdy1LXiUiB796FspIPd5ZTWdfMNywyWKBNRkosz910OpFhIcxLifVLih0VeI43O9h+8BgZAdJ/AxYOOMDDQDOQBCwDHhGRmV0cNxz4LyABOB04l5OXnL7dGBPlfqR5r8rKk3KLq3l49W6P9Wk8+nERkWEhRA/z/1DozuZNjOOSuePYekDzqqm+2XrgKK1Ow7wA+oFiyYAjIpHAZcA9xpg6Y8xa4E3gus7HGmMeMcasMcY0G2MOAs8CC31bY+VpucXVXLViPX9cWTDojvS6plZ+9PIWcoqraWh28M0nNlmyY37JjCQamh18sqfS31VRAaDtMxxIV8TW+6nnMg1wGGMKO2zbCizqw7lnA3mdtj0gIr8DCoCfGWOyO58kIsuB5QCJiYlkZ590SNCpq6vzWDvsrnaws8pBelwIU2J7v9/8h5zjtDhcPelNLU6ef38ztZPD+vWarU5D9v5W3tjTTG2za5sBmvtZnifboSctTkNECDz5/hZspdbpYwLftYHVWakd3vu0kbGRwpZNn/j8tQfaDlYNOFHAsU7bjgEjejpJRL4FzANu6rD5LiAf1+25q4C3RGSOMWZPx3ONMSuAFQBpaWkmKytrMPUfErKzs/FEO+QWV/P7VetxOA1hdkevCQZXfLyH7ZU7sQm4F+PksqzTWDA5oU+vtaGoEocTXv30APuONHP6pDi+dto47n0zj5ZWJ6F2G1efN7/Pvww91Q59cV7pp2woOsJZZy8ixGaNYdvg2zawMqu0g9Np+H/ZK7nwlPFkZZ3q89cfaDtYNeDUAZ3H+UUDtd2dICKXAr8DzjPGtN+TMMZs7HDYUyJyNXAh8JDHaqt69M7nh0+4Wlm3u7LbP/aPrSnit//ZyUWnjuH6BSm8nHOAl3MP8Pa2w70GnNziaq5+dAPN7rVBkmOH888b5pOVloiIMHXUCDYUHSEzNd6ytyGWzBzN29sO81lJtddyY6nAt7uijprGVst+jrtj1YBTCNhFZKoxZpd722xOvlUGgIh8GXgUuMgY83kvZRvAOj8dg0BxZX37vw2uK4+lM0eTNvrEC9bH1+7l/nd2cNEpY/jrlXOwh9j40qR44iLD+MfHRZw5JYELThnT7es8s35fe7AR4Bvzx7M4fVT7/owAGAGWlZZIaIiwMr9MA47qlrcTdnqLJQcNGGPqgVeB+0QkUkQWApcAz3Q+VkTOwTVQ4DJjzKZO+0aKyFIRiRARu4gsw9XH857334UCKKtpZM2uSpbOTOJHS9O4+4J06ppaufh/1vLYmiKc7ntm/1y3l1+/nc8Fs0bzl6tcwabNHUvSmD0+hrv+vY0D1Q1dvs5Tn+zj9S2HEIEQgfBQW59uwVlNdEQoCyYnsDKvFGOMv6ujLCqQEnZ2ZNUrHIBbgSeAcuAIcIsxJk9EknH1ycwwxpQA9wAxwH86pCpZY4y5AAgF7gfSAQewE7jUGKNzcXzkiXV7aXU6+emF00mJdy10dnnGeH7y6ufc/84OXvvsALHDw1m72xWU/nb1XEJDTvwdFGa38dDVp3Hh39bw/Re28OLyzPaA5HQafv/eTv7xURFLZiRxwxkT+Wz/UUvfNuvNkhlJ/Pz17ewur2NqUo/dlipI5RZXM29iYCTs7MiyAccYUwVc2sX2ElyDCtqed7tSgzGmApjvjfqp3tU0tvDchhIuOGVMe7ABSIgKZ8V1GfxhZQEPr94D1GIT+PbCSScFmzbJ8cP5zddm8f0XtvCX93dx59I0mlud/PiVrby+5RDXZaZw71dnEmITzpgSeFc2HZ3vDjgr88s04KiTlNc2UnykgWtPT/F3VfrNsgFHBb7nNpZQ29TKLYsmn7RPRBgeZm8fiSa40nSc3kNal0vmjGPd7koezt7N8PAQXty0n+KqBn60NI1bsyYH3K+97iRFRzB7wkhW5pVy2+Ip/q6Osphcd/9NIGUYaGPJPhwV+JpaHTyxdi9nTklg1riYLo/JTI0nzG4jRCDUbutTDrF7vzqTsTERPPhuAcVVDYSGuHKPDZVg02bJjCS2HjjG4WPH/V0VZTE5xdWE223MGtv198rKNOAor3j9s4OU1zbxnUWp3R6TkRLLszdl8sMlab3OzWkzPMzO+TO+WBba6TRDMrvy0plJALyfX+bnmiirySmuZvb4kYTZA+/Pd+DVWFme02n4x8dFzBwbzZm99KdkpMRy2+Ip/ergv3j2WCJC+3dlFGgmJ0aRmhDJSg04qoPjzQ7yAixhZ0fah6M8btWOMooq6vnb1XO9cqur7crI6pM4B0NEOH9mEo+v2cux4y3EDAv1d5WUBQRiws6OPH6FIyIPichbXWyPFpF7RWR6h20/EJFtIqJXWkOEMYa/f7SHCXHDuHDW6N5PGKCBXBkFmiUzRrvywRWU+7sqyiLe3HoIALuF0h71h0f/0IvIZOA7wK+62D0P+CWuuTFt/g6MAq73ZD2U/2zeV81nJUe5+azUEyZvqv6bO2EkCVHheltNAa65Ny9sKgHgO//KtWTG8954+i/CfwFbjTE5XeybCzThmrQJgDHmOPA0J69fowJQbnE1P3l1GyMi7FyRYa1FzgKRzSacPyOJD/LL+NsHhQH5B0Z5zrrdFe3JbAN1KfI+BRwRmSIiLSLyq07bHxGRWhGZJyLhwLXAc12cvwP4AxAOtIiIEZFX3LtfAGaIyBmDeifKr3KLq7nm0Q3sqajneLOD/MM1/q7SkJCaGEljq5M/r9o16HWBVGCLDHd1udsCeLBMnwKOMWY38BjwAxFJABCRXwDfBr7mvqLJBEYCa7oo4ptAEfAWsMD9uMO9bwtQA3x5oG9C+d+GoiPtiTONGZpDlf2hbfVPAzQH6K9a5Rl7K+sJC7Hx/XOn9nkagdX055bar4AQ4C4RuRFXf8x1xpj33fszcX0vtnVx7lZgPPChMWaD+1EMYIxxus/JHOB7UBbQ8ddWoP76sqIzJicQ4Z5v4TQQO7x/i9CpocEYw/v55SxOT+T7500LyGAD/Qg4xphS4C/A94B/AP/PGPNSh0PGAjXGmOYuTp8JhAGfdlN8hft8FaDGjRyGARZNSwzYX19WlJESy7M3Z3Jr1mRGR4fz+3d3UljW7bJQaojafrCG0prGEyY9B6L+DhrYhasfZr0x5uFO+yJwDQroymm4rn62dLP/ODCsn3VRFrJ2t2vNu7u+nK7BxsMyUmL58ZfTefm7ZxBut/HNxzd1u0yDGppW7SjDJnBOh/WdAlGfA4573Zl/AOuBhSIyu9MhR4Du/tLMBfYYY7rrSY4DKrvZpwLA2l0VJESFkT5asxt7y4S44Tx945doaG7lm49v4khdd7/v1FCzKr+MeSlxxEUG9i3Vvo5SOw14HdfAgSygBPhtp8N2AqEiMr6LImbQYTh0FyYBukZNgHI6DWt3V3LmlARsATohLVCkj47miRvmc/Doca74+3r+vEqHSw91+6sa2HG4hvNnJPm7KoPWa8ARkSnA/wErge+5+2h+BVwoImd3OPRj93+/1EUxR4HZ7tU3M0WkvUdZREYC0zqcrwLMztJaKuuaOXNqor+rEhTmTYzjjiXTKKqs568f7OLKf6zn8bVF1Da2tB+TW1zNw6t3azAaAj7Y4Zr4e94QCDg95lITkdG4As0OYJl7RBm4Jmv+GPgdcAaAMWafiGwCLsa1PHRHvwAex3WVFAGcBax177sIaAZeG+R7UX6yZlcFQK+JOpXntDgMgqtjtNVp+PXbO/jNOzuYPiaalPjhrMovw+E02G02fvGVGUwfOwIRYVdZLXmHapgzYSSnjo/BbrMRYhNCQ2zkHTpG3qEaFk5JOKEfLre4mrf3NDNiUrX2z/nBqh1lTBkVxaSEyN4PtrgeA457ZNpJ+eWNMQ5g+sln8AjwVxG5zRjT0OH47cDp3bzMtcDLxpgTJhiISByuILUEV//OT4wxJ00qdR/7A+AuXAMP/o1rOeqm/pajBmbt7kqmJUUxOibC31UJGpmp8YSH2mhpdRIaYuPuC9Kpbmghp7iKVflltDhcU9KbHU5+/sb2k85/en1xt2X/aVUhKfHDmZQQSYgIHxVW4HAa3t63QUcg+tix4y1sLKri5rO7X+YjkHg6W/QzuK58bsWVWaBHIjIHWAzM6mL3w7iufJKAOcA7IrLVGJPXqYylwN3AOcAhXFdKv3Jv63M5amAaWxxs3FsVkMvdBrKeMmZv2nuEax/fRKvDid1m4ycXpDN5VBRvbD3Iq7kHMbhmq194yhjOSR9Fq9Pwfn4Zq/LLMLhWX7XbhCN1zRRV1NHqzqfSNvFUA47vZBeU0+o0nDc98G+ngYcDjjHGISLfxjUMui9GA99yZzJoJyKRwGXALGNMHbBWRN4EruOLQNLmeuDxtgAiIr8GngXu7mc5agA276uiudXJWdP0dpqvZaTEdvnH/0uT4nn+5pODUWS4nXe2HXZdFdltfGvhpPZ9kxOj+HhXRfu+By+fTUZKLLnF1Sx7dAON7iwSOqHXt1bll5EQFcbcCSP9XRWPEGOMv+twEhGZC3xijBnWYdudwCJjzMWdjt0K/NYY86L7eQKuiaQJQHI/ylkOLAcYMWpCRty3HvHKe1NKqaGq+PdfyTXGzOtuv1UXYIsCjnXadgzoapJH52Pb/j2iP+UYY1YAKwDS0tJMwe8u6n+th5js7GyysrJ6POaCv64hZpidF5Yv8E2l/KAv7TDUPfnGB9y7vpFfXzqL6zKD9/apLz8La3ZVcN3jm3jsm/MsN0Ktu3aQ3/d8nlUXLKkDojttiwa6yunR+di2f9f2sxzVTxW1Tew4XMNZOhx6yEuJtpE+egSv5Oz3d1WCxqr8MiJCbZw5dejcrrZqwCkE7CIytcO22UBXHf157n0djytzj3rrTzmqn9a509mcNYS+EKprIsIV8yaw9cAxzeXmA65knWWcNTWRiNAQf1fHYywZcIwx9bjm8twnIpEishC4BNcouM6eBm4UkRkiEgv8HHhyAOWoflqzq5LY4aHMHBvj76ooH7h0zljsNuFlvcrxurxDNRw61jgksgt0ZMmA43Yrrnk15cDzuObW5IlIsojUiUgygDHmXeBBYDVQ7H78srdyfPc2hiZjDGt2VXDGlARCNJ1NUIiPCuec9FG89tlBWhzO3k9QA/b+jjJE4NwAT9bZmVUHDWCMqQIu7WJ7Ca7BAB23/Qn4U3/KUYOzq7yO8tomztLsAkHlinkTWJlfRnZBxZD79W0lq/LLyEiOJT4q3N9V8SgrX+EoC/u40J3ORvtvgkpWWiIJUWF6W82L3ssrJe9QDdPHdB7vFPg04KgBWbu7ktSESMbHDvd3VZQPhYbY+NrccXy4s1yXR/CC3OJqbnvWtU7lSzn7h1zyVQ04qt+aWh1sLKrS0WlB6op5E2h1Gl7fcsjfVRlyNhQdaU8l1OpwpRIaSjTgqH7LLa7meItDlyMIUtOSRjB7fAwv5+zHiplKAtkp7hGfAoTabUMulZAGHNVva3dVEmITMlPj/F0V5SeXZ4xnZ6lrqQPlOUcaXLcpl2WmDMnM3BpwVL+9u72U0dHhFJbV+bsqyk++OnscYXabDh7wsJV5ZSRFh3PfV2cOuWADGnBUP31UUE5RZT2Hjjay7LENQ65TU/VNzPBQlsxI4o2th2hqdfi7Oh7T00qpbQvReesz39ji4KNC13DzobpUu2Xn4ShremOrq6PYAC26PkpQu2LeBN7edpgfvbyN68+YGPCfg9ziapY9toHmVichNuH7505lUoJryl9RRR1/+3AXrQ7vLUS3dlclDc0Ols4c7dFyrUQDjuoXp3uCeYgMzU5N1XfDw1w5vt7ceoiV+aUB3+ewoegITS1ODOB0GP6wsrDL47y1EN3K/FJGRNg5fdLQ/U5pwFH9cqC6gWlJUVwyZ9xJK02q4LJpb1X7v4fC1e6YmAjaxtyF22387uunMmNsNCKQf6iGH7+yjWZ3Sp/TJ3l2wIzDaXh/RznnpI8izD50ezo04Kg+a2xxsO3AMW5YOJHbFk/xd3WUn2WmxhMWIjQ7DCG2wL7abW518uiavcQMs3NtZgrnpCedEDynJY1gQtxwHnh1IzllDnKKq5k30XNBJ7e4mqr6ZpbMGLq300AHDah+2H7wGM0OZ0D/ilWek5ESy5Pf+hIhNmHJjKSA/lz89YNCdhyu4Q9XzOFHS9O7fC8ZKbHcNiecC08ZzR9XFvD5gc5rOw7ce3mlhNltLEob2nPbNOCoPtu8zzU6Z14A/2FRnnXGlATOmBzPjtLAnY/zaUk1j2Tv4YqM8b0mJBURfvu1U4iPDOf7L35GQ3ProF/fGMPK/FLOnJJAVPjQvumkAUf1Wc6+KlITI4dcBls1OFlpo9hTUc/+qgZ/V6XfGppbueOlrYyJGcYvLp7Rp3NGDg/jT1fOZm9lPfe/s2PQddhZWsv+quMsCYLs2xpwVJ84nYac4mrmp2h2AXWiLPdtoGx3BvFA8vv/28neynr++4pTGRER2ufzzpicwPKzU3luYwkr80oHVYeVee61b6ZrwFEKgN0VdRw73sK8iXo7TZ0oNSGSCXHD+Kig3N9V6Zd1uyt5an0xN5wxkTMm9z8R7R3npzFrXDR3/Xsb5TWNA67He3mlZCTHkjhi6N850ICj+iTH3X8z34Mjc9TQICJkTRvFJ3uOBEzWgTW7KvjuM7mMjYngri+nD6iMMLuNv1w5l+MtDm5+Oof/+XBXv7MQ7K9qIP9wDUtmDv2rG7BgwBGROBF5TUTqRaRYRK7p4djrRSRXRGpE5ICIPCgi9g77s0Wk0b0kdZ2IFPjmXQw9OfuqSIgKIyVe179RJ1s0LZGGZkf7DxMryy2u5oYnNlPb1EplfTP5hwc+4GHKqCiuXzCRrQeO8YeVhVzzaP/SPa3KLwMY8sOh21gu4AAPA81AErAMeEREZnZz7HDgv4AE4HTgXODOTsfcboyJcj/SvFPloW9zcRXzUuIQGZo5ntTgnDElnrAQG9kBcFttza4KHO5lFRweWHMmepidtm9FU6uTf67b2+dlG1bml5KWNIKJCZGDqkOgsFTAEZFI4DLgHmNMnTFmLfAmcF1XxxtjHjHGrDHGNBtjDgLPAgt9V+PgUHqskf1Vx7X/RnVreJidL02KI7vA+gMH2mby2zyUnikzNYHwUBs2ARF4e9thbn46t9d+nar6ZjbtrQqa22kAYqUFlERkLvCJMWZYh213AouMMRf34fzXgZ3GmLvdz7OBmbjWMyoAfmaMye7m3OXAcoDExMSMl156aVDvpS92VzvYWeUgPS6EKbEhXn+9/qqrqyMqKopNh1v5361N/GJBBKkx1qunt7W1QzDrSxu8u7eFFwqa+eOiYcQPs9Rv2RM89nkTm0pb+cqkUGbE9++71107tH2Xp8XZ2HPU8OquZkJtcF6yndAQYXoX3/E1B1p4fHsz9y6IYGKAfa+6a4fFixfnGmPmdXee1WYZRQGdp+8eA0b0dqKIfAuYB9zUYfNdQD6uW3RXAW+JyBxjzJ7O5xtjVgArANLS0kxWVtZA6n+STXuP8MaWQ4yJiWBERCjltY2U1zSxu6KOLfvrwUB4qMOSiQ+zs7PJysoi+808hoXu57qvLCY0xLp/SLylrR2CWV/aYPyMWl4o+JimuClknZ7sm4r1k8Np+OGa9/nyrLH86eq5/T6/u3bovOU7FXXc+uynvFlUC4BNWrjolDHMHBdDYlQ4CSPC2fx5ASPCnZwyey4ZATYYZ6DfCZ8GHPcVx6Judq8DvgdEd9oeDdT2Uu6lwO+A84wxlW3bjTEbOxz2lIhcDVwIPNSvig/Q+j2VLHtsI84OF5EhNiEhKsxdP9e2phYnG4oqLRdw2uQUVzE3eWRQBhvVd5MToxg3chjZBeVcY9GA82mJK2dZbxkFBis1MYqvnDqGgtJaV/ZpA/+3vZS3th0+6dhlj2+05A9Ob/BpwDHGZPW0392HYxeRqcaYXe7Ns4G8Hs75MvAocJEx5vPeqgD4pNfb4TT88s289mBjE7glazJ3nJ+GzSbta280utOhF5bVYYyxXKd8XVMr+YdquP2cqf6uirI4EWFRWiJvfHaQ5lanJbMer8ovIzRE2ieretOCyQmEh+6mpdVJqN3GszeeTtqYaCprm1jxcRHPbyoJunWlLPWJMMbUA68C94lIpIgsBC4BnunqeBE5B9dAgcuMMZs67RspIktFJEJE7CKyDDgbeM+778KVG+nXb+dTWFaH3SaEiKuj8pz0L1byy0iJ5dmbMrlzyTQumDWaN7Yc4sevbKPVnf7cKj4rqcZpNH+a6pusaYnUNzvIKa7q/WAfM8awKr+MzNT4fmUVGKi27/gPl6S5rmAmxhEVbmdiQiSXZYwnPNQWdOtKWa0PB+BW4AmgHDgC3GKMyQMQkWRcfTIzjDElwD1ADPCfDlcGa4wxFwChwP1AOuAAdgKXGmO8Phfn0TVFPPnJPm48cxIXnjKGDUVHulw7JiMlloyUWIwx/OX9Xfz1g10cPd7CQ1fPJSLUGp2Im/dVYxOYmzzS31VRAeCMKQmEhggfFVYMaPa+N+2pqGNvZT3fXjjRZ6/Z9h3vavuzN2V2+7dhqLJcwDHGVAGXdrOvBNfAgrbni3sopwKY7+n69eaNLQf57X92ctGpY/jZhdOx2aTXD5OI8IPzpxEXGcYv38zj6/+7jvNnJHH2tFF+/yDm7Kti+phon/wiVIEvKtzO/IlxfFRQwU8umO7v6pxgpXuSpVVylnUXjIYyS91SC3Trdldy58tbyUyN40/fmN1++6yvrj9jIv913lTyD9fy1w92s+yx/s1a9rRWp2HL/qOazkb1y6JpiewsreXwseP+rsoJ3s8vY9a4aMaOHNb7wcorNOB4yL9zD/Ctf25mTPQw/nHdPMLtA7slFhpiax/V0LZ2ur/sr3XS0OzQCZ+qX7LSRgHwkYUmgVbUNvHZ/qOcPz04UshYlQYcD3h/Rxl3vLyVZoeTstpGdpfXDbiszNR4wkNtJzz3l8Jq1wCGebokgeqHaUlRjImJ4CMLLVfwwY4yjMHrw6FVzzTgDJLTaU5YhKl1kLmZ2joTz5qSgNNA7HD/9Z3sqnYwPnYYo2Mi/FYHFXhEXMOOPyqo4KEBZFD2hlX5ZYwbOYzpY3qdQ668SAPOID21fh/7Kuvbhz97YohjRkosf7xyNnab8OzGEg/VtH+MMRRWO7X/Rg3IhNjhNLQ4+POqQr/3RTY0t7J2dyXnz0iy3Dy3YKMBpwvHmkyfviAFpbU88H87OSd9FC8u7zDe3gMjT0aNiGDprNG8knuAxhbfrzFSUtVATbPR/hs1IC3u+WRO88XERn9Zs6uSplan3k6zAA04XahuMlzdy7oWjS0Ovv/CZ0RH2Hnw8lPJmBjHbYuneHSY47Wnp3DseAtvbT3ksTL76uXcAwBEhllu5LwKAGdOTcTuHqVpD/HvxMZV+WVER7iyWSv/0oDTjeZWJ4+tKep2/4PvFrCztJb/vmI2CVHeWRo2MzWOKaOi+JePb6vlFlfzyGpXftO7X91miXvwKrBkpMTy9+syCBHhnHT/zSdzOA0f7ixncfoozQVoAfp/oBs2cSXb+8Ub22luPTHdzEeFFTyxbi83nDGRxe4hoN4gIiw7PZmt+4+y/WDnJNres35PZfsCVf6+HaIC13nTk/j6aePILqjgaEOzX+qQW+ybZJ2qbzTgdCE2XHhheSbLz07l6fXFXPPohvbFlI7UNXHny1uZlhTF3RcMbC30/vj6aeMZFhrCvzYUe/212rRlFRCCK8+T8rybzkrleIvDb4Nf3t/hSta5aJr3k3Wq3mnA6UJMuPClSfH89MLpPHT1XPIO1fCVh9by3MZirlyxger6Zv56lW/yncUMC+Wrs8fyxpZD1DS2eP31APZW1hMaIlwyOTRo0qYr70gbPYKzpibw5Cf7aGr17eCX3H1VvJyznxmamskyNOD04uLZY3nttjOwCfz0te3tkzobmn335bk2M4XjLQ5edXfke5PTaXh3eymL00Zx6dQwDTZq0G4+K5WK2ibe2nryWjDekltczTWPbaS6oYW8QzXaD2kRGnD6IH10NN+YN6H9uTHGp/0ap4yPYfb4GP61sQRvLwm+7eAxSmsa+fIsTQGiPOOsqQmkJY3gsTVFXv/8ttlQdKS979Xp4++r6p4GnD5alDaKCD+uX7EsM4Xd5XVs3OvddUbe3V6K3Sacm66drMozRIQbz5rEztJa1u32zR/+zNT49qUWw7Qf0jI04PTRSYsp+fhW08WnjiU6wu7VwQPGGN7dfpgFk+OJ8WNKHTX0XDJnLAlR4Tzaw1QDT0pNiAQDC1LjtB/SQjTg9ENGSqzHJ3f21bCwEC7PmMD/bT/Mg+/u9Mo96YKyWvYdaeCCWWM8XrYKbuH2EK5fkMJHhRUUltV6/fWyC8sxwF0XTNdgYyEacALInOQYHE54JHuPV/JTvbu9FBHNqKu8Y1lmChGhNh5fs9frr/XBjnISosI5dVyM119L9Z3lAo6IxInIayJSLyLFInJND8feICIOEanr8MgaSFmBYH+Va0Erg3cmZL67vZT5KXEkjvBO5gQV3OIiw7jstPH8+9MDXrtKB1cet48KKzgnPbHfiyAq77JcwAEeBpqBJGAZ8IiIzOzh+PXGmKgOj+xBlGVpmanxhNvd/8tEPNoRuq+ynp2ltSzV0WnKizJT42h1Gq9dpQNs3ldFbWOrZZaSVl+wVMARkUjgMuAeY0ydMWYt8CZwnT/LsoqMlFieuzmTU8fHgDHERYZ5rOz38koBWDpTv6TKe0q8fJUO8OGOcsJCbJw5JcHjZavBsVoq4GmAwxhT2GHbVmBRD+fMFZFKoAp4BnjAGNPa37JEZDmwHCAxMZHs7OwBvwlv+/ZUJ3cdhh89s4bb53pmcbQX1x9nYrSN3Vs3sdu9ra6uztLt4CvaDp5rg/CjDuwCrcY1ajn8aDHZ2Z6d0PzWpw1Mi7Wxef1aj5YL+lloM9B2sFrAiQI6Z6k8BnS3TN/HwCygGJgJvAi0Ag/0tyxjzApgBUBaWprJysrqf+19qMS+iz+tKiRq4qnMG+QiaYePHafo3Q/50dI0srKmtG/Pzs7G6u3gC9oOnmuDLGD23Cq+9c/NTE2K4qavLRx0mR0VVdRR9u5H3HZ+OlkLJnq0bNDPQpuBtoNPb6mJSLaImG4ea4E6ILrTadFAl+MojTFFxpi9xhinMeZz4D7gcvfufpUVaG46axJJ0eHc/86OQc/eXplXBqDZBZRPzJ8Yx7cWTmTr/qMcOnrco2V/sKMcgHPSvZfFXQ2cTwOOMSbLGCPdPM4ECgG7iEztcNpsIK+vL0H7/OJBl2Vpw8Ps3LEkjS37j/LO54PLUfXu9lKmjopicmKUh2qnVM+uyJiA08ArHs4P+MHOMtJHj2B87HCPlqs8w1KDBowx9cCrwH0iEikiC4FLcPXNnERELhCRJPe/04F7gDcGUlYguuy08aSPHsHv39054Ey8VfXNbNx7RK9ulE8lxw9n4ZR4XsrZj9Ppmfxqx463sHlftV7dWJilAo7brcAwoBx4HrjFGJMHICLJ7rk2ye5jzwW2iUg98B9cAea3fSlrKAixCT+7aDr7q47zzPqBpbxZlV+K08DSmRpwlG9dOT+ZA9XH+WSPZ0aqfVRYgcNpOHe6BhyrstqgAYwxVcCl3ewrwTUYoO35ncCdAylrqDhraiKLpiXytw92cXnGeEYO799Q6Rc37ydmmJ2mFt+uVaLUkhlJxAwL5YXNJZw5dfBDmD/cUUZcZBhzJmgqG6uy4hWO6qefXjid2sZWbnoqp18T6d7ccpBPS45Sc7yVZY9v1DVDlE9FhIbwtbnjWJlXRnX94JagbnU4WV1QQVZaIiGaXcCyNOAMAXVNrdhEyCmu5hv/WM8q96iz7lTUNvHz1z/n+y9uAbw7CU+pnlw5fwLNDievfXZwUOV8WnKUY8dbdFkNi9OAMwRsKDqCwdXx6nAavvOvHH744hbyD9WccNzxZgcPfbCLrP9ezfOb9rN0xmjC7f5b40ep6WOimT0+hhc37x/U8P4PdpZhtwlnT9PsAlZmuT4c1X+ZqfGE2W20tDqxh9g4b/oo3s0r5dXPDrJwSjxZaaPI3VfNxr1HqG5oYenMJH785XQmJ0aRW1zNhqIjZKbGaxp35RdXzk/mp699ztYDx5gzYeSAyvhgRzmnp8YxIkLXcbIyDThDQNvicB0Dx7HjLTy/qYQVHxW1r7IoAvdfOotrM1NOOFcDjfKni2eP4ddv5/Pi5pIBBZziI/XsLq/jmi8l936w8iu9pTZEdF4cLmZYKN9dNJkbFk5snwlrwzVXQSkrGRERykWnjuHNLYeob2rt9/lPfrIPgKRoXVbD6jTgDHELpyQQHqr9NMrarpw/gfpmR7+zZuQWV/OUO+Dc8fJWHWlpcXpLbYjr6nabUlYzLyWW1MRIHl+7l4rapj5/Vp9Zv4+2RAVtIy31M25dGnCCgPbTKKsTERZOTuCZDcX8sayAMLuNZ2/K7PFzu/3gMf7zuWtZdBt6BR8INOAopSxhRITrz5HTQHMvVyvlNY3c/HQOCVFh3HfpLApKa/UKPgBowFFKWcK505N4bO1emludOA0kRnWdpqmxxcHNz+RytKGFV25ZwMyxMZyny0kHBB00oJSyhIyUWJ6/OZMbz5xE4ogw7n0rn9UF5SccY4zhR69sY9uBo/zlqjnMHBvjp9qqgdCAo5SyjIyUWO75ygze+d5ZTIyP5Kancng5Z3/7/oc+3M1bWw/x46XpmuE8AOktNaWU5YyKjuDF72Ryy78+5UevbOOz/Uepa2zlza2H+Ppp4/juolR/V1ENgAYcpZQljYgI5Ykb5nPjU5t5bmMJ4MqW8Y154xHRjNCBSG+pKaUsK8xu4/RJcSdky8gtPurHGqnB0ICjlLK0BZM1W8ZQYbmAIyJxIvKaiNSLSLGIXNPDsX93Lznd9mgSkdoO+7NFpLHD/gLfvAullKe0Zcv44ZK0XieDKmuzYh/Ow0AzkATMAd4Rka3GmLzOBxpjvgt8t+25iDwJODsddrsx5jGv1VYp5XWaLWNosNQVjohEApcB9xhj6owxa4E3gev6ce5T3q2lUkqpgbBUwAGmAQ5jTGGHbVuBmX049zKgAvi40/YHRKRSRNaJSJZHaqmUUqrfrHZLLQo41mnbMWBEH869HnjanLhO7V1APq5bdFcBb4nIHGPMns4ni8hyYDlAYmIi2dnZ/a/9EFNXV6ftgLYDaBu00XZwGXA7GGN89gCyAdPNYy0wF2jodM4dwFu9lDsBaAVSeznuXeB7vdVz2rRpRhmzevVqf1fBErQdtA3aaDu4dNcOQI7p4W+rT69wjDFZPe1398PYRWSqMWaXe/Ns4KQBA518E/jEGFPUWxUAnTGmlFJ+YKk+HGNMPfAqcJ+IRIrIQuAS4JleTv0m8GTHDSIyUkSWikiEiNhFZBlwNvCeF6qulFKqF5YKOG63AsOAcuB54BbjHhItIsnu+TTJbQeLyAJgPPByp3JCgftxDSSoBL4HXGqM0bk4SinlB1YbNIAxpgq4tJt9JbgGFnTcth6I7OLYCmC+F6qolFJqAKx4haOUUmoI0oCjlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJywXcETkdhHJEZEmEXmyD8f/QERKReSYiDwhIuEd9sWJyGsiUi8ixSJyjVcrr5RSqluWCzjAIeB+4IneDhSRpcDdwLnARCAV+FWHQx4GmoEkYBnwiIjM9HB9lVJK9YHlAo4x5lVjzOvAkT4cfj3wuDEmzxhTDfwauAFARCKBy4B7jDF1xpi1wJvAdV6puFJKqR7Z/V2BQZoJvNHh+VYgSUTigWTAYYwp7LR/UVcFichyYLn7aZOIbPdCfQNNAlDp70pYgLaDtkEbbQeX7tohpaeTAj3gRAHHOjxv+/eILva17R/RVUHGmBXACgARyTHGzPNsVQOPtoOLtoO2QRttB5eBtoNPb6mJSLaImG4eawdQZB0Q3eF5279ru9jXtr92AK+jlFJqkHwacIwxWcYY6eZx5gCKzANmd3g+GygzxhwBCgG7iEzttD9v4O9AKaXUQFlu0ICI2EUkAggBQkQkQkS6u/X3NHCjiMwQkVjg58CTAMaYeuBV4D4RiRSRhcAlwDN9qMaKwb6PIULbwUXbQdugjbaDy4DaQYwxnq7IoIjIvcAvO23+lTHmXhFJBvKBGcaYEvfxPwTuAoYB/wa+a4xpcu+LwzW8+nxco97uNsY855M3opRS6gSWCzhKKaWGJsvdUlNKKTU0acBRSinlExpwOgjW3Gs95a8TkXNFZKeINIjIahHpcWJXoBKRcBF53P3/vVZEPhORCzrsD4p2ABCRf4nIYRGpEZFCEbmpw76gaQcAEZkqIo0i8q8O24KmDdxTWRpFpM79KOiwr9/toAHnRMGae63L/HUikoBrpN89QByQA7zo89r5hh3YjysTRQyu9/ySiEwMsnYAeACYaIyJBr4K3C8iGUHYDuD6m7C57UmQtsHtxpgo9yMNBt4OOmjAzZ17rRqY1ZYOR0SeAQ4aY+72a+V8RETuB8YbY25wP18O3GCMOcP9PBJXOou5xpidfquoj4jINlzJYOMJ0nYQkTQgG/g+MJIgagcRuQr4Oq6RsVOMMdcG23dCRLKBfxljHuu0fUDtoFc4X5hG17nXguEKpzszcbUB0D63aQ9B0CYikoTrM5FHELaDiPyviDQAO4HDwH8IonYQkWjgPuCOTruCpg06eEBEKkVknYhkubcNqB004HyhX7nXgkRQtomIhALPAk+5f60FXTsYY27F9f7OwnXrpIngaodf48pEv7/T9mBqA3DNcUwFxuGa7PmWiExmgO2gAecLmnvtZEHXJiJiw5WNohm43b056NoBwBjjcC/rMR64hSBpBxGZA5wH/LmL3UHRBm2MMRuNMbXGmCZjzFPAOuBCBtgOGnC+oLnXTnZCrjr3fdrJDNE2EREBHsc1aOQyY0yLe1dQtUMX7HzxfoOhHbJwLehYIiKlwJ3AZSLyKcHTBt0xgDDQdjDG6MP9AF4AngcigYW4LhFn+rtePnjfdiAC1+ikZ9z/tgOJ7ja4zL3t98AGf9fXi+3wd2ADENVpe9C0AzAKuArXLZMQYClQjysPYVC0AzAcGN3h8QfgFff7D4o2cLfDSPf//7a/B8vcn4W0gbaD39+UlR64hve97m7UEuAaf9fJR+/7Xvcvl46Pe937zsPVcXwc12ilif6ur5faIMX9vhtx3S5oeywLsnZIBD4CjgI1wOfAzR32B0U7dGqTe3GN1AqqNnB/Fjbjuk121P1j7PzBtIMOi1ZKKeUT2oejlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJzTgKKWU8gkNOEoppXxCA45SAUJEokXkXhGZ7u+6KDUQGnCUChzzgF8Cof6uiFIDoQFHqcAxF9cyAfn+rohSA6GpbZQKACKyA0jvtPnfxpjL/VEfpQZCA45SAUBE5uPKZp4H/Na9+bAxpth/tVKqf+z+roBSqk+24loI7SFjzAZ/V0apgdA+HKUCw0wgDPjU3xVRaqA04CgVGE7DtV7PFj/XQ6kB04CjVGCYC+wxxtT4uyJKDZQGHKUCwwx0OLQKcDpoQKnAcBQ4TUSW4lpLfpcx5oh/q6RU/+gVjlKB4RdAGfA6sB7Q9DYq4Og8HKWUUj6hVzhKKaV8QgOOUkopn9CAo5RSyic04CillPIJDThKKaV8QgOOUkopn9CAo5RSyic04CillPKJ/w/sw4IAACSQYAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Zu5G0eLAFpaY",
    "outputId": "4a6d35e5-ea3d-4a09-9938-eb7bf27e9993"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "219/219 [==============================] - 1s 1ms/step - loss: 0.1001 - val_loss: 0.0545\n",
      "Epoch 2/20\n",
      "219/219 [==============================] - 0s 928us/step - loss: 0.0379 - val_loss: 0.0266\n",
      "Epoch 3/20\n",
      "219/219 [==============================] - 0s 847us/step - loss: 0.0202 - val_loss: 0.0157\n",
      "Epoch 4/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0131 - val_loss: 0.0116\n",
      "Epoch 5/20\n",
      "219/219 [==============================] - 0s 917us/step - loss: 0.0103 - val_loss: 0.0098\n",
      "Epoch 6/20\n",
      "219/219 [==============================] - 0s 846us/step - loss: 0.0089 - val_loss: 0.0087\n",
      "Epoch 7/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0080 - val_loss: 0.0079\n",
      "Epoch 8/20\n",
      "219/219 [==============================] - 0s 880us/step - loss: 0.0073 - val_loss: 0.0071\n",
      "Epoch 9/20\n",
      "219/219 [==============================] - 0s 846us/step - loss: 0.0066 - val_loss: 0.0066\n",
      "Epoch 10/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0061 - val_loss: 0.0062\n",
      "Epoch 11/20\n",
      "219/219 [==============================] - 0s 880us/step - loss: 0.0057 - val_loss: 0.0057\n",
      "Epoch 12/20\n",
      "219/219 [==============================] - 0s 807us/step - loss: 0.0054 - val_loss: 0.0055\n",
      "Epoch 13/20\n",
      "219/219 [==============================] - 0s 846us/step - loss: 0.0052 - val_loss: 0.0052\n",
      "Epoch 14/20\n",
      "219/219 [==============================] - 0s 848us/step - loss: 0.0049 - val_loss: 0.0049\n",
      "Epoch 15/20\n",
      "219/219 [==============================] - 0s 845us/step - loss: 0.0048 - val_loss: 0.0048\n",
      "Epoch 16/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0046 - val_loss: 0.0048\n",
      "Epoch 17/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0045 - val_loss: 0.0045\n",
      "Epoch 18/20\n",
      "219/219 [==============================] - 0s 807us/step - loss: 0.0044 - val_loss: 0.0044\n",
      "Epoch 19/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0043 - val_loss: 0.0043\n",
      "Epoch 20/20\n",
      "219/219 [==============================] - 0s 843us/step - loss: 0.0042 - val_loss: 0.0042\n"
     ]
    }
   ],
   "source": [
    "# Linear predictions:\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[50, 1]),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\")\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oKI4_MVtFpaZ",
    "outputId": "0443d4f8-54bf-4682-c7f5-3cacc093365e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 516us/step - loss: 0.0042\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.004168087150901556"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E9OZe5otFpaZ",
    "outputId": "f7016e8b-387a-479f-f89d-134acd6f08f3"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABAUUlEQVR4nO3dd3hUZfr/8fednhASEgJBICGAE6SD6EJAioKg66KroquggqJYfqyiYtuvLoqu7iq6q2tDpSmI6CLKrl2KiKACCgqCQaRJCS1AAulz//44QwxxEtKmAPfrus6VmXOeOeczKXPnOeU5oqoYY4wxdS0k0AGMMcacmKzAGGOM8QkrMMYYY3zCCowxxhifsAJjjDHGJ6zAGGOM8QkrMMYYY3zCrwVGRBJFZI6IHBKRzSIytJK2t4vIThE5ICKTRSSyzLKFIpIvIrme6Uf/vANjjDFV5e8ezHNAIZAMDANeEJH25RuJyCDgXqA/kAa0Ah4q12y0qsZ6pjY+TW2MMaba/FZgRKQecCnwgKrmqupiYC5wtZfmw4FJqrpGVbOBh4ER/spqjDGm9sL8uK10oERVM8vMWwX09dK2PfBuuXbJItJQVfd65j0mIn8HfgT+T1UXetuoiIwCRgFER0d3S23alNgNGyho1IhtxU3Yvz8Clyunym/C7XYTEhI8h66CKY9l8S6YskBw5bEs3gVTFoDMzMw9qtqo2i9UVb9MQG9gZ7l5NwALvbTdAJxX5nk4oECa53l3oD4QidPbyQFaHytDt27dVN1u1fr1VUeP1kcfVQXVvDytsgULFlS9sR8EUx7L4l0wZVENrjyWxbtgyqKqCizXGnzu+7NE5gJx5ebFeYrDsdoeeZwDoKpfqWqOqhao6jTgC+D3VUohAi4XrF9PQoIzKzu7qm/BGGNMVfmzwGQCYSLiKjOvM7DGS9s1nmVl22Xpr7vHylNAqpwkPR3Wr6dBA+fp/v1VfqUxxpgq8luBUdVDwNvAeBGpJyK9gIuA17w0fxUYKSLtRCQBuB+YCiAiDURkkIhEiUiYiAwD+gAfVTmMywWbNpEYWwhYD8YYY3zB30eRbgGigV3ATOBmVV0jIqme61lSAVT1Q+BxYAGw2TON86wjHHgE2A3sAf4M/FFVq34tjMsFbjdNDv8MWIExxhhf8OdZZKjqPuCPXuZvAWLLzXsKeMpL293AmbUK4nL20jXctx44zXaRGWOMD/i1wASN9HQA4netB6wHY44fBw8eZNeuXRQVFdV4HfHx8axdu7YOU9WcZfHOn1nCw8Np3LgxcXHlz8GqvZOzwCQmQmIi0b84l+RYgTHHg4MHD5KVlUWzZs2Ijo5GpOrntZSVk5ND/fr16zhdzVgW7/yVRVXJy8tj27ZtAHVeZILnSh5/c7kI3bCeevXsLDJzfNi1axfNmjUjJiamxsXFmLJEhJiYGJo1a8auXbvqfP0ndYE5ci2M9WDM8aCoqIjo6OhAxzAnoOjo6Frtdq3IyVtg0tNh61aaxOdZgTHHDeu5GF/w1e/VyVtgPGeStY/8yXaRGWOMD5z0BaZNyHrrwRhjjA+c9AWmtdsKjDHHmxEjRvCHP/yhWq/p168fo0eP9lGiXz344IN06NDB59s5HpycpykDxMVBcjKp+Zm2i8wYHznWvv2hQ4cyY8aMaq/36aefPjLSepW9/fbbhIeHV3tbpuZO3gID4HJxypb1HDwIJSUQGhroQMacWHbs2FH6+H//+x833HDDUfOKi4uPal9UVFSlIhAfH1/tLImJidV+jamdk3cXGYDLRdJ+52p+68WYk8nSpfDYY85XX2rSpEnp1MAzfPmR5/n5+aSkpDBz5kzOOeccoqOjmThxInv37uXKK6+kefPmREdH0759e6ZMmXLUesvvIuvXrx+33HILf/nLX0hKSqJx48aMHTsWt9t9VJuyu8jS0tJ45JFHuPHGG4mLi+O0007jiSeeOGo7mZmZ9O3bl6ioKNq0acP7779PbGwsU6dOrfL3wO128/DDD5OSkkJkZCQdO3bk3XffParN+PHjadGiBZGRkTRp0oRRo0aVLlu0aBE9evQgNjaW+Ph4unfvzurVq6u8/UA6uXsw6enUOziFWHLYv78+DRsGOpAx1TNmDKxcWb3XZGdHs3o1uN0QEgKdOkF1OgRdusC//lW9bVbmvvvuY8KECUyaNInw8HDy8/M5/fTTueeee4iLi+PTTz/lxhtvJDU1lf79+1e4nhkzZnDbbbexZMkSVq5cydChQ+nWrRtXXnllha/55z//yUMPPcRdd93FnDlzuPvuuznrrLPIyMjA7XZz8cUX06RJE7788kvy8vIYM2YMBQUF1Xp/Tz/9NE888QQvvvgiZ5xxBtOnT+eSSy5hxYoVdOnShdmzZzNhwgRmzpxJx44d2bVrFwsXLgScHt5FF13EyJEjmTFjBkVFRXzzzTeEHie7W07uAuM50O9iPdnZpwc4jDH+ceCAcOQfe7cbDhyoXoGpa3/+858ZMmTIUfPuuuuu0sejRo1i/vz5zJw5s9IC065dO8aPHw9Aeno6L7/8MvPmzau0wAwcOLC0V3PTTTfx0ksvMW/ePDIyMvjkk0/48ccf+fjjj2nWrBngFKRevXpV6/1NmDCBsWPHMnToUMDprSxatIgJEyYwffp0Nm/ezCmnnMLAgQMJDw8nNTWVNm3aAM7wQPv372fw4MG0bt0agNNOO61a2w8kKzBYgTHHr5r0JD79NJ8LL6xHYSFERMCMGZCRUefRquyMM8446nlJSQl///vfmTVrFtu2baOgoIDCwkL69etX6Xo6dep01POmTZsec/iTyl6zbt06mjZtWlpcAM4880xCQqp+ZOHgwYNs3779N0XprLPO4v333wfgsssu4+mnn6Zly5YMGjSI8847j7PPPpv69euTmJjIiBEjGDRoEP3796d///5cdtllpKSkVDlDIJ3cx2BOPRVwCowdgzEni+7d3cybBw8/DPPmBba4ANSrV++o5xMmTODJJ5/krrvuYt68eaxcuZI//vGPFBYWVrqe8icHiMhRx2Cq+xpVrbMr3L2t58i8lJQUfvzxRyZOnEhcXBx33nknffr04dChQwBMmTKFr776ij59+jB37lzS09P56KOq318xkE7uAhMTQ/EpzT09mECHMcZ/MjLgvvsCX1y8Wbx4MYMHD+bqq6+mS5cutG7dmszMTL/naNu2Ldu2bWP79u2l85YvX37MolVWXFwcTZs2ZfHixUfNX7x4Me3atSt9HhUVxQUXXMA///lPli1bxtq1a/niiy9Kl3fu3Jl77rmHhQsX0q9fP6ZNm1aLd+Y/J/cuMkBcLtJ3ZLLICowxQSE9PZ1Zs2axePFikpKS+Pe//83GjRvp2rWrX3Oce+65tGnThuHDhzNhwgTy8vK44447CAsLq1bP5q677uKvf/0rLpeLbt26MX36dD7//HNWrFgBwNSpUykuLqZ79+7ExsYya9YswsPDcblcbNy4kYkTJ3LhhRfSrFkzfv75Z7777jtuvvlmX73tOnXSF5iQNi5ci2Yzd3+gkxhjAO6//342btzI+eefT3R0NCNGjGDYsGH88MMPfs0REhLCnDlzuP766/nd735HWloaTz75JJdccglRUVFVXs+tt95KTk4Od999N1lZWbRp04bZs2fTpUsXABo0aMA//vEPxo4dS1FREe3atWP69Om0bNmSrKwsMjMzueyyy9izZw/JyckMGzaMe+65x0fvuo6p6kkzdevWTX9jwgRV0NuH7/3tMi8WLFhQpXb+Ekx5LIt3dZXlhx9+qJP1HDx4sE7WUxeOtywrV65UQJcvXx7wLHWtst8vYLnW4DP3pO/BHDmTLPqX9UD3wGYxxgSVOXPmUK9ePVwuF5s2beKOO+6gc+fOnH66nXVaFSf3QX4oLTBxWesDHMQYE2xycnIYPXo07dq1Y9iwYbRt25aPPvrI7stTRdaDadWKEkJI3GcFxhhztGuuuYZrrrkm0DGOW9aDiYxkb70WNDloBcYYY+qSFRhgT6KLlHz/n2dvjDEnMiswwMHGLloWr0fd1bu/hDHGmIpZgQEON3MRz0EObdod6CjGGHPCsAIDFKalA3BopR2HMcaYumIFBkpPVS5cbcdhjDGmrliBASLbpFFEGJppPRhjgtGDDz5Ihw4dKnzuzejRo485xH9Ntu0r5e/SeSKwAgPENwzjZ1oRutEKjDF1afDgwQwYMMDrsrVr1xIXF8cnn3xS7fWOHTuWzz77rLbxjrJ582ZEhOXLl/t8WycLKzBAQgKsx0X0VttFZkxduv7665k/fz6bNm36zbJJkyYd8zbIFYmNjaWhn+5x7s9tnWiswPBrgamf9ROonapsTgJLl8JjjzlffeiCCy4gOTmZKVOmHDW/qKiI1157jauuugpVZeTIkbRs2ZLo6GhcLhePP/54pfddKb/bqqSkhLFjx5KQkEBCQgJjxoyhpKTkqNd8+OGH9O7dm4SEBBITExk0aBBr164tXd6xY0fAuWuliJTuXiu/LbfbzcMPP0xKSgqRkZF07NiRd999t3T5pk2bEBFmz57NueeeS0xMDO3atat2T62goIAxY8aQnJxMVFQUPXr0OOq+MkVFRdx66600bdqUyMhIUlJSuPfee0uXv/3223Tq1Ino6GgSExPp27cvWVlZ1cpQWzZUDBAXBz/hIrzwMGzfDmVukWpMUBszBlaurNZLorOzYfVqcLshJAQ6dYL4+KqvoEuXKt+rOSwsjOHDhzN16lTGjRtXervh//73v+zZs4errroKt9tNs2bNePPNN2nUqBFff/01o0aNomHDhowcObJK23nyySd5+eWXefnll+nUqRPPPfccM2bMOGpQykOHDjFmzBg6depEXl4ejzzyCIMHD+aHH34gIiKCBQsWcPbZZ/Phhx/SuXNnIiIivG7r6aef5oknnuDFF1/kjDPOYPr06VxyySWsWLGidAh+gP/7v//jiSee4Pnnn+eRRx7hiiuuYPPmzcTGxlbpPd199928+eabTJ48mVatWvHUU09x3nnnsX79ek455RSeeeYZ5syZwxtvvEFaWhq//PILP/74IwA7d+7kiiuu4LHHHuPSSy8lNzeXL7/8skrbrUvWg8H5G9tWzzlVmfV2HMac2OTAAae4gPP1wAGfbm/kyJFs2bKFTz/9tHTepEmTGDhwIM2bNyc8PJzx48dz5plnkpaWxuWXX85NN93EzJkzq7yNf/3rX9x9991cfvnlnHbaaTz99NM0adLkqDaXXnopl156KS6Xi06dOjFlyhQ2btzI119/DUBSUhIADRs2pEmTJiQmJnrd1oQJExg7dixDhw4lPT2d8ePH07t3byZMmHBUu9tvv53Bgwfjcrl49NFH2bdvHyur+M/AoUOHeOGFF/jHP/7BBRdcQNu2bXnxxRdJTk7mueeeA5xjRunp6fTu3ZvU1FR69uzJtddeC8D27dspKipiyJAhpKWl0aFDB66//nqSk5Or/D2tC37twYhIIjAJGAjsAe5T1dcraHs7cA8QDcwGblbVgnJtXMD3wH9U9araZNuT4IJDQGYm1MGZJ8b4RRV7EmXlf/op9S68EAoLISICZszw6b2TXS4Xffr0YfLkyQwcOJDt27fz0UcfMWvWrNI2L774Iq+88gqbN28mLy+PoqIiWrRoUaX1HzhwgB07dpBR5j2EhITQvXt3tm7dWjpvw4YNPPDAA3z11Vfs3r0bt9uN2+1my5YtVX4vBw8eZPv27fTq1euo+WeddRbvv//+UfM6depU+rhp06YA7Nq1q0rb2bhxI0VFRUdtJzQ0lIyMjNIbr40YMYJzzz2X9PR0Bg4cyO9//3vOP/98QkJC6Ny5MwMGDKBDhw4MHDiQAQMGMGTIEBo1alTl91oX/N2DeQ4oBJKBYcALItK+fCMRGQTcC/QH0oBWwEMVrG9ZXQTLS0qhMCTSejDmhOfu3h3mzYOHH3a++rC4HHH99dfzzjvvsG/fPqZOnUpiYiIXXnghALNmzWLMmDGMGDGCjz76iJUrV3LLLbdQWFhYpxkGDx7M7t27mThxIl999RXffvstYWFhNdqOt+H6y88LDw//zbLKjiuVpZ5jwZVt5/TTT2fTpk08+uijuN1uhg8fzrnnnovb7SY0NJSPP/6Yjz/+mE6dOjFp0iRcLherVq2q2husI34rMCJSD7gUeEBVc1V1MTAXuNpL8+HAJFVdo6rZwMPAiHLruwLYD8yri3wNEkPYFtnaCow5OWRkwH33+aW4AAwZMoSoqCimT5/O5MmTueaaa0o/gBcvXkz37t0ZPXo0p59+OqeeeiobNmyo8rrj4+M55ZRTjjrGoKqlu74A9u7dy9q1a/nLX/7CgAEDaNu2LTk5ORQXF5e2OXLMpfzJAWXFxcXRtGnTow62H3kP7dq1q3LmY2nVqhURERFHbaekpISlS5cetZ369etz2WWX8cILL/Dee+8xf/58fvrpJ8ApRBkZGYwbN45ly5bRtGnTo3qN/uDPXWTpQImqlj0XeBXQ10vb9sC75doli0hDVd0rInHAeJweTqVHAUVkFDAKIDk5mYULF3ptV1zcnkxa0XjlSpZV0AYgNze3wnUEQjDlsSze1VWW+Ph4cnJyar2ekpKSOllPdQ0ZMoRx48axf/9+rrjiCnJycigpKSE1NZWpU6cye/ZsWrVqxezZs/nss89o0KBBac6CggLcbneFz2+66SYef/xxUlJSaN++PS+//DI7duygcePG5OTkEBYWRsOGDXn++edJSEhgx44d3H///YSFhZGfn09OTg6JiYlER0czd+5ckpKSiIyMJD4+/jfb+vOf/8yjjz5K8+bN6dKlC7NmzeLzzz9n0aJF5OTkkJubCzjHUcp/n/Py8ir83hcVFVFcXExOTg5RUVGMHDmSe+65h5iYGNLS0njuuefIysrimmuuIScnh2effZbk5GQ6depEWFgYU6dOJS4ujvj4eObNm8fChQvp378/jRs35rvvvmPr1q20bNmywu3n5+fX/d9MTe6zXJMJ6A3sLDfvBmChl7YbgPPKPA8HFEjzPH8auMfz+EFgelUydOvWrcJ7To8cqfpcvbtUIyJUi4srbBdM93pXDa48lsW7uspS2T3TqyMQ93tXVV2xYoUC2rNnz6OyFBQU6HXXXacNGjTQ+Ph4ve666/Shhx7SFi1alLYbN26ctm/fvsLnRUVFOmbMGI2Pj9f4+HgdPXq03nTTTdq3b9/SNvPmzdP27dtrZGSktm/fXj/88EOtV6+eTpkypTTLyy+/rCkpKRoSElL62vLbKikp0fHjx2vz5s01PDxcO3TooHPmzCldvnHjRgV02bJlR71/QN96660Kvz/Dhw/XCy64oDRLfn6+3nbbbdq4cWONiIjQ7t276+eff17a/qWXXtKuXbtqbGys1q9fX/v06aNffPGFqjq/K+edd17pa1u3bq3/+Mc/Ktz2kddUBFiuNfncr8mLarQh6AocLjfvTuC/XtquAi4v87yhp8A0BLoAa4AIrcMCM3as6i3hLznfko0bK2wXTB9cqsGVx7J4ZwWmYpbFu0Bk8UWB8edB/kwgzHPm1xGdPcWivDWeZWXbZanqXqAfzoH/LSKyExgLXCoi39QmXEICrC6yU5WNMaau+K3AqOoh4G1gvIjUE5FewEXAa16avwqMFJF2IpIA3A9M9Sx7CWiN05PpArwIvAcMqk2+Bg2cq/kB51RlY4wxteLv05RvwbmuZRcwE+faljUikioiuSKSCqCqHwKPAwuAzZ5pnGfZYVXdeWQCcoF8Va3V3cISEmAHp+COqWc9GGOMqQN+vdBSVfcBf/QyfwsQW27eU8BTVVjng3WRLSEBQMhrdir1rMAYY0yt2VAxHg0aOF/XFKaT970VGBOc1AZjNT7gq98rKzAeR0YT/3Szi/CtP/Pl50UBzWNMeeHh4eTl5QU6hjkB5eXlHTXyQF2xAuPhGYSUTFyEUcLKdzYFNI8x5TVu3Jht27Zx+PBh68mYOqGqHD58mG3bttG4ceM6X78N1+8xcCA89BCsV+dMsr5N1wOuyl9kjB/FxcUBv46UW1P5+flERUXVVaxasSze+TNLeHg4ycnJpb9fdckKjEdGBgwZAovnpEMxtA2z4zAm+MTFxdX6g2DhwoV07dq1jhLVjmXxLpiy1IbtIitjyBDYUZxEcWy8XQtjjDG1ZAWmjJ49AYS9CS67FsYYY2rJCkwZzZtDaiqsl3QrMMYYU0tWYMrp2RO+znbB5s2Qnx/oOMYYc9yyAlNOr16wIscFqvDzz4GOY4wxxy0rMOX07Flm0EvbTWaMMTVmBaacTp1ge4wVGGOMqS0rMOWEhUGbHglkhyXZqcrGGFMLVmC86NUL1hW7KF5nPRhjjKkpKzBe9OzpjElWvNYKjDHG1JQVGC969ID1pBO1ZxscOhToOMYYc1yyAuNFgwaQn+I50P/TTwHNYowxxysrMBVI+J1TYNw/2m4yY4ypCSswFWg50Ckwu5dYgTHGmJqwAlOB7v1j2c4p7P/aTlU2xpiasAJTgVatYFO4C9lgPRhjjKkJKzAVEIFDTV0k7rUCY4wxNWEFphIR7dNJKtnF7p8OBDqKMcYcd6zAVCL5LOdA/5p3rBdjjDHVZQWmEq0GOQVm+2dWYIwxprqswFQiom1rAA6vsgJjjDHVZQWmMtHRZMelEr1tPQUFgQ5jjDHHFyswx1CU5uJUdyYrVgQ6iTHGHF+swBxDbFcXLtazZEmgkxhjzPHFCswxxHROJ5FsvluwN9BRjDHmuGIF5lhczplke5dmohrgLMYYcxyxAnMsngLTMHs9GzYEOIsxxhxHrMAcS8uWaEiIHYcxxphq8muBEZFEEZkjIodEZLOIDK2k7e0islNEDojIZBGJLLNsuojsEJGDIpIpItf7LHREBLRsSfvw9Xzxhc+2YowxJxx/92CeAwqBZGAY8IKItC/fSEQGAfcC/YE0oBXwUJkmjwFpqhoHXAg8IiLdfBVaXC46RWdaD8YYY6rBbwVGROoBlwIPqGquqi4G5gJXe2k+HJikqmtUNRt4GBhxZKFn/pFLH9UztfZZeJeLlPz1rFmt5OaG+WwzxhhzIvHnp2U6UKKqZe/gtQro66Vte+Ddcu2SRaShqu4FEJHncYpONPAt8L63jYrIKGAUQHJyMgsXLqx28GaAqzCXxmSxYkU4sbHVX4ev5Obm1ug9+YJl8S6YskBw5bEs3gVTllpRVb9MQG9gZ7l5NwALvbTdAJxX5nk4Ti8lrVy7UOAs4H4g/FgZunXrpjXy4YeqoH1DFunVV2+s2Tp8ZMGCBYGOUMqyeBdMWVSDK49l8S6YsqiqAsu1Bp/7/jwGkwvElZsXB+RUoe2Rx0e1VdUSdXa1NQdurqOcv+U5Vbl/SiarV8f7bDPGGHMi8WeByQTCRMRVZl5nYI2Xtms8y8q2y1LP7jEvwvDlMZjUVAgPp0fD9axdG0dxsc+2ZIwxJwy/FRhVPQS8DYwXkXoi0gu4CHjNS/NXgZEi0k5EEnB2gU0FEJHGInKFiMSKSKjnjLMrgfk+Cx8WBq1bc1rYevLzQ/nuO59tyRhjThj+Pk35FpyD8ruAmcDNqrpGRFJFJFdEUgFU9UPgcWABsNkzjfOsQ3F2h/0CZAMTgDGq+i6+5HKRfNC5L4ydrmyMMcfm13NuVXUf8Ecv87cAseXmPQU85aXtbryfeeZbLhcRn3xC46TDfPFFDKNH+z2BMcYcV2yomKpyuSA/n36uH60HY4wxVWAFpqrS0wE4K/l7tmyBX34JcB5jjAlyVmCqynOqcueYtYAdhzHGmGOpdYERkfC6CBL0mjWDqChaFPxMTAw28KUxxhxDtQqMiNwqIpeWeT4JyBORH0WkTZ2nCyYhIXDqqdTb8Qvdu1sPxhhjjqW6PZhbgd0AItIHuBwYCqwEnqzTZMEoPZ3oX36hZ0/49ls4dCjQgYwxJnhVt8A0AzZ5Hg8G3lLVN4EHgR51FytIxcQQ/csv/KHB55SUwLJlgQ5kjDHBq7oF5iDQyPP4XGCe53EREFVXoYLS0qUwaxYhbjfd7x9ID5bacRhjjKlEdQvMx8DLnmMvpwIfeOa3BzbWZbCgs3AhlJQAIEWF/KnxQjsOY4wxlahugfl/wBdAEjDEc2U+wOk4Q7+cuPr1g8hIFECV/B79WLIE3O4A5zLGmCBVrQKjqgdV9c+qepFnvLAj88ep6qN1Hy+IZGTAvHnszcgAVdqdEcP+/bBuXaCDGWNMcKruacrtyp6OLCLnish0EblPRELrPl6Qychg3T33QHg4vX+eBtjpysYYU5Hq7iKbBHQFEJHmOLc1TsTZdfZI3UYLTsXx8fCHP9Dg/RmcklRkB/qNMaYC1S0wbYFvPI8vA75S1d8DV+Pck+XkMHw4smsXt7T+yHowxhhTgeoWmFCg0PO4P/C+5/EGILmuQgW93/8eGjXissPTyMyE3bsDHcgYY4JPdQvMauBmEemNU2COHOhvBuypy2BBLTwchg7FtW4uCexj6dJABzLGmOBT3QJzD3ADsBCYqarfe+ZfCHxdh7mC3/DhhBQVclXoG3YcxhhjvKjWHS1VdZGINALiVDW7zKKJwOE6TRbsunSBjh25ceM0blpyS6DTGGNM0Kn2cP2qWoIzgnIHEWkvIlGquklVd/kgX/ASgeHDaZ/7NQe+WkdBQaADGWNMcKnudTBhIvIEkA2sAr4HskXk8ZPmvjBlDRuGOySUK4um8e23gQ5jjDHBpbo9mMeBq4CbgHTABdyMc5ryY3Ub7TjQpAlF55zH1bzGks9LAp3GGGOCSnULzFBgpKpOU9UNnmkqcD0wrM7THQciRw2nOdvIeXd+oKMYY0xQqW6Bice55qW8DUCDWqc5Hg0ezKGIBnRYMRXVQIcxxpjgUd0Cswrnrpbl3eZZdvKJimJTjys4P38Om747GOg0xhgTNKpbYO4GhotIpohME5GpIvIjznGZsXUf7/gQecNwYshj57/fCnQUY4wJGtUdrn8RzsH9t4BYIM7zeBDeezYnhVZXdiczpA1J708LdBRjjAka1brQEkBVtwP/V3aeiHQGLq2rUMebkFDhi9bDuXb9X2DDBmjdOtCRjDEm4Kp9oaXx7sCFV+NGyH/p1UBHMcaYoGAFpo50PL858+iPe9qrdh9lY4zBCkyd6d4dXpPhxGRtgs8/D3QcY4wJuCodgxGRucdoElcHWY5rsbHwU8eLObwmlphp06Bv30BHMsaYgKpqD2bvMaaNwEl/8KFbn3r8Ry5H33oLDh0KdBxjjAmoKvVgVPVaXwc5EfTsCS88O5xrcifDnDlw1VWBjmSMMQFjx2DqUK9esJizONCwJUydGug4xhgTUH4tMCKSKCJzROSQiGwWkaGVtL1dRHaKyAERmSwikZ75kSIyyfP6HBH5VkTO99+7qFhKCjRMCmFm2DXo/PmwdWugIxljTMD4uwfzHFAIJOOMvvyCiLQv30hEBgH3Av2BNKAV8JBncRiwFeiLM/jmA8CbIpLm4+zH9OWXkJ0Nj2ddg6iy5W+vBTqSMcYEjN8KjIjUw7na/wFVzVXVxcBcnHvJlDccmKSqazy3Zn4YGAGgqodU9UHPXTTdqvo/nJMMuvnljVRi4ULnEpiNtGIRvak3exo2xLIx5mQl6qcPQBHpCixR1egy88YCfVV1cLm2q4BHVXWW53kSsBtIUtW95domA5uBLqq6zst2RwGjAJKTk7u98cYbtXofubm5xMbGel22Zk0cd97ZmYKCEK5lMpO5nm+ee46D7drVaps1zeNvlsW7YMoCwZXHsngXTFkAzj777BWqeka1X6iqfpmA3sDOcvNuABZ6absBOK/M83BAgbRy7cKBT4GJVcnQrVs3ra0FCxZUunzJEtXrrlOtzwEtDI9WvemmWm+zNnn8ybJ4F0xZVIMrj2XxLpiyqKoCy7UGn/v+PAaTy28vyIwDcqrQ9sjj0rYiEgK8hnNMZ3TdxaydjAyYNAkGXBzHbPcluGe+Afn5gY5ljDF+588CkwmEiYirzLzOwBovbdd4lpVtl6We3WMiIsAknJMFLlXVIt9ErrkJE+C1kOGEHNgPc481EIIxxpx4/FZgVPUQ8DYwXkTqiUgv4CKcXkh5rwIjRaSdiCQA9wNTyyx/AWgLDFbVPN8mr5lWraDrnefwC83IftruE2OMOfn4+zTlW4BoYBcwE7hZVdeISKqI5IpIKoCqfgg8DizAOYC/GRgHICItgBuBLsBOz+tyRWSYn9/LMd37f6HMqXc19Zd8hHv7zkDHMcYYv/JrgVHVfar6R1Wtp6qpqvq6Z/4WVY1V1S1l2j6lqsmqGqeq16pqgWf+ZlUVVY3yvObINMOf76UqYmMh9YHhhFHCN3cGXTxjjPEpGyrGxwbfdRqr63Wn3n+mknPQrokxxpw8rMD4WEgI1LtlOG2LVzNtzLeBjmOMMX5jBcYPWt77J4pCImDaNH7+OdBpjDHGP6zA+ENiIsXnX8gV7te55/bCQKcxxhi/sALjJ9E3DSeJPRTO/YBPPw10GmOM8T0rMP4yaBCanMwtMdMYMwaKiwMdyBhjfMsKjL+EhyPDhnFu4f/YuWYPEycGOpAxxviWFRh/Gj6ckOIixrlm8sADsHfvsV9ijDHHKysw/tSpE3TpwsjwaRw4AA8+GOhAxhjjO1Zg/G34cGJ+WMEnHcbw7fNLWb060IGMMcY3rMD4m8sZTPrs75/hY3d/Jl671G56aYw5IVmB8bfvvgNAVImSAmKXL7TR/I0xJyQrMP7Wrx9ERQEg6ia6STx33AEFBYGNZYwxdc0KjL9lZMD8+fCXvyBpafxl/900+fkL/vWvQAczxpi6ZQUmEDIy4G9/g6VLCUttxidh5/PBQ1+zY0eggxljTN2xAhNITZrA/PmENW3MO3kDmXjjN4FOZIwxdcYKTKA1a0bE5/NxxzXgz/89l9WvfxfoRMYYUyeswASD1FTCF80nPySGZsP7o2t+CHQiY4ypNSswQaJ+51Z8+bf55BWHk9+rP2RmBjqSMcbUihWYIHLx3S5ubTeP3Bw37rPPgQ0bAh3JGGNqzApMEAkJgTtebss57k/Jy86Hc86BzZsDHcsYY2rECkyQ6dkTOg3tyNlFn3A46yD5Pc+GX34JdCxjjKk2KzBB6PLLYVlxV/oWfEzB9r3k9TwHu0jGGHO8sQIThH74wdldtpwzOZ8P0O070P79YdeuQEczxpgqswIThPr1g8hICA2Fr0N7cl7Je+T/uJnCvgNgz55AxzPGmCqxAhOEMjJg3jx4+GFYtAhGTu3D5ZFzKVm3nn3dzkX3ZQc6ojHGHFNYoAMY7zIynAmcA/99+vTn/sHv8OiaC9lw6iASV3xCYsv4wIY0xphKWA/mONGyJTy+ahBzr5lNavZKfm5zPsv+/impM2bA0qWBjmeMMb9hPZjjSGgoXDbtD/zcaRZdxg5B7hsEgM6Ygcyb92uXxxhjgoD1YI5Dre68GC67nBDchOJG8/LZ+eI7gY5ljDFHsQJznAq7/VYkMhIFBCXp1Qms6TyUkq+WBzqaMcYAVmCOXxkZsGABG6+/ntxX3+aD9Ntp/t17hPY4k7zufWHuXHC7A53SGHMSswJzPMvIYMuwYdS/+mL+sG4CH760lb9EPcXuZZvgoovQ006DF16Aw4cDndQYcxLya4ERkUQRmSMih0Rks4gMraTt7SKyU0QOiMhkEYkss2y0iCwXkQIRmeqX8EFOBP50Qxw3/Xg71/bewJ94g3U7G8Att1B0Sgrcfz/s3BnomMaYk4i/ezDPAYVAMjAMeEFE2pdvJCKDgHuB/kAa0Ap4qEyT7cAjwGQf5z3upKbCJwvCaPT//kS7nK84i895L6cP+uij0KIFXHcdrF4d6JjGmJOA3wqMiNQDLgUeUNVcVV0MzAWu9tJ8ODBJVdeoajbwMDDiyEJVfVtV3wH2+jz4cSgkBJo1g5AQ4QvO4mKdQ7r+yBv1r6do+hvQsSMMGgQffwxLlsBjj9m1NMaYOieq6p8NiXQFlqhqdJl5Y4G+qjq4XNtVwKOqOsvzPAnYDSSp6t4y7R4BmqvqiEq2OwoYBZCcnNztjTfeqNX7yM3NJTY2tlbrqEsV5VmzJo477+xMUZEQFqZcdNF2fvghjh1rirmRidwe9gyNirNQEQDc4eGseuopDrb/TYey1lkCwbJULJjyWBbvgikLwNlnn71CVc+o9gtV1S8T0BvYWW7eDcBCL203AOeVeR4OKJBWrt0jwNSqZujWrZvW1oIFC2q9jrpUWZ4lS1QffdT5esTGjaqPPabarUO+vskQdYOqZypp1Fj1tttU//tf1YMH6zSLv1mWigVTHsviXTBlUVUFlmsNPvf9eQwmF4grNy8OyKlC2yOPvbU1FcjIgPvuO/oC/7Q0uPdeWP59JN2m30FRWDTFhFJIOMt3p1Hw74kweDCamAi9e8P48c5utOLigL0PY8zxyZ8FJhMIExFXmXmdgTVe2q7xLCvbLkvL7B4ztddqWAYRi+YR+reHWf/yZ7x+21e0aZTNOczjKRnLz2vz0QcfhF69yI9tyL4+f4Rnn4Uff3T6PMYYUwm/jUWmqodE5G1gvIhcD3QBLgJ6emn+KjBVRGYAO4D7galHFopIGE72UCBURKKAYlW1f7OrKyMDycigPfAv4Mkno1i48BxmzjyHR2Y/Roju5WwWMLDgE/p//imJn7/rvC4lBQYM+HXasMEZeDMy0sZEM8YA/j9N+RYgGtgFzARuVtU1IpIqIrkikgqgqh8CjwMLgM2eaVyZ9dwP5OGcynyV5/H9fnsXJ7DQUOjfH155xbls5vdXNWQ2Q7iRiZzKBlqxgTtjJ/JFSXfy3ngHhg2D5GTo1YuWr7wCffvCpEl2cacxxr8FRlX3qeofVbWeqqaq6uue+VtUNVZVt5Rp+5SqJqtqnKpeq6oFZZY9qKpSbnrQn+/lZBAZCbfcAtHRTuGJioLL7m7FnktGMTz6LWLzdnMGy/hYBqGqCEBREVx/PdSvD23bwhVXOKdBf/ABbN9uu9aMOYnYcP2mUkfurrlwoXMr57J7v3buDGXJkjNY99Y4+ry5iFB3IcWE8xDjaJ6QT8+8VbjmfUX9WbNKX3M4JonC9l1o0KczdO4MXbrAaadBeLjTYOlS7xszxhx3rMCYYyp7d82ymjSBSy4BLsmAW+eR+dIUDp15LfEHMvjwC/jrEsjOhnj20yv2O1ofWkWXwyvpvHwVXVY+S2iRp1MaEQHt2kHTpvDJJ1BS4nSf7B43xhzXrMCYupGRwfaCAvr1y6CrZ5bbDevWweLFDXjxxT68/20fZ4FCaFExXaJ+5Pymq+hZbyXt8lbRbP5nhBUVOW3y8uCCC5wDQl27/jo1aRKQt2eMqT4rMMZnQkKcjkm7ds7oNP37Q2EhhIXB7beHkZfXniXft2fi90PZvRt6sJT5nEMEhaiEsjGmM8mffUPcf/5Tus7Chk3Y2rArkRmn03ywp+i0bOmM9mmMCSpWYIxfVHYsByArC77/PoN3/zufsC8W8v6hfryxOYO8PIjjAF1ZSd/4bzl13zd02fstLTI/hmklzovj451jOaef7hQcEVIXLLBTpo0JMCswxm8qOpYDzpnOycnAgAwggyHAyyWwcSN8/30833/fl7fe6ls6EHQUeXRgNV35lt4l33LGym9pvfgFIkryAWgJ6JQpSI8ecOaZcOqp4HI5X1u0+PWkAmOMz1iBMUErNNSpB6eeChdfDOee++tuNsKjufiBM1E9k4/XwTPrYP3aYsYdGsut/JtQ3LhV2bN8E7HLvie6OLd0vRoaSkGTNLbHnEp0ZxennHXqrxtq2dI56QDsjDZjaskKjDluHGs3m2oYe/77J7j8JUoKCykJjeDhzrOZs6MHhdt24WI9p/ITrpKfaL3tJ+fx+qXwn4O/riMkBFJTkUaN4NtvnTPawsKca3nOPtvpZjVubD0gY6rACow5rlS2m00EGl2YAQvm8fPkybS67jqeycjgGSAnJ5nMzGTWrTuLSZPg/gVHXqU0ZK9TbFjPqe6fOG3rT/T5ZRFNiot/vXh07NijN5aUBE2asD+6Cb8UJZPYtglNT2/inOWWnOx8bdIEMjNtCB1z0rICY048GRlsKSigVZkP9Pr1oVs3Z2rV6tddbRERwuvvJJGcnMSmTT3YuBGWboJvvlnKg4v7E6bOxaOj+Td7aEQTdtIiYiet3Vk02b6TiH07SWYJDVbuhJl5XuO0BJg82el2deni3HY0JcX5mpoKjRrZWXDmhGQFxpx0KtrV1rnzUa1gqdMo7/R+jG6cwaZNlE5fbYKvvoKs0vZKfXJIJosm7OQUdnJdxKsMLHyPEBR1uzm05DsiP19KeNHRhcgdEcn++imEtkghvmPq0QXoyNfvv7fjQea4YwXGnJQq29VWvlEc0BXnDOiyli79tScUHi5MeDqORo3i2LnTRVYWrFrZnLP/O49QdyFFEsEf3HP5rKgHDdlLCltJZYvztXALqXu3kLJ3K62/n0fj4u2EqPuobZWO4CYhuM/8HaGtW0KDBkdNmbsa8M2GeNr3akDH3mWWRUYeFdp22Rl/sQJjTA0d66SDI72gI8eDFvTI4OBByMpKYufOJHbu7Mprr8Hz7/06BmiLplAvspiwXdupv38rKWzhGl5lEB8RguJWN9u/3kr+sr0kyH4auLMJw7lLRbpn+vXGFg53ZBQa3wCJikR+2UpLtxumTIHLL4dOnZzjSZ7p261JfLYmie7nJZBxVqj3N25n15kqsgJjTC0csydU5niQ4FwTGh8P6enO4pQUp0g5x4Ng5kzIyAgDUikpSSU7uxe5n6Sh136Gu6gQd2gEC0a+xXf1MtizB/bsVg5m5bHnp/1wYD8N8DIV7KfBrv30YCkdcBMCaEkJxW/8h/CZM4+K29Uzuf8m5MUkoklJaKJTfEKTkwjTAsLemuXc4TQ8HJ56Cs46y+kpJSQ4B7tEqlaDrDd1wrMCY0wAVdYLCg31dCyuzIA0p1FIv35cc9SHsQAxLF0aQ//+TT276+DVV51LerKznWn/fli1cimnvdyfkGLnFO57us1jRXFn2LuX8AN7CNu/h4bsIenIdHgPSVs8ExtJYhnJ7AQ8u+8KC2H06KPej1tCOBTegMaFDTiXBA5IA1a3SSCmaQM0IYGQhAaEJCUQk7ubpJcepWVRIfraa/DKK0j37s49IcpMS5eFsfAzOWahqlWPyu2Gzz8n9bXXrNjVMSswxgRYdY4HVba48t11ABlw9a+77P5V2igGSGHJEufmpEeK1DPPQGIaHMiBrQchJwfiVi/hyskDCC0ppETCmNJ+Atu1KWG5+4k4lE1k3n4iD2dTn/0kkE0D3U/Iuu1ErXOeR5P/21gFBXD11V7f1+8IoTOR5BPF3sgoNDKK4rAo3OFRuCOiCHfn02j7KkTdICHkpLZHo6IIdRcT6i4ixF1E0eFiivKKiA4rIjKkCIqLkaIi5/Tz4mKnwFDmbL+0NKc6HxlewnPt09rsZL7amEzHAcl0O7+xUwTLq4vdhydQz84KjDEniKoWqvKncB/Rs2dVilRPGOE0Cu3Xjxu9NDr65Ad47TVo0wa2HIK87HwKd+2n3rIFZEy8Fikuwh0axrxe48iKaoH7cD7uvHzIyyd7Rz6Hs/OJpIAo8olz5xOdn09YST7hJflEkY+LHQhuBHCrm12bD7OeZhQTRhHhFBHu9XGJhKNhYWhEOGe6v+SswvnOMS6ETTtjOLSvgMSiZSQUZhFT4owC0dYz8bzzPvMj48iLS6YgPpmChGTCQtycsvy/iLsEDQ1j7+W3QOvWhEeFEh4dRkR0KGGRofy0MZQffgylbYdQ0tuGOl3VI1NmJtx3Hy0LC51v3AsvOEMdRUY6BS0yEiIj+fLbSBYsiaTf2eK7nl0dsAJjjCnl+95UFNAEhl8Jw9JKe1ODjlGoIiKOvj2QqjO/8LOl6EX9oagQwiMofv416rucQVI1D96aCbNmOe1DQpzhhnr2hPx8p+NUUAAFm5fS/YMlTq8sJIJnOr3CyugMCgqcdpJ3mJIdu4g6mEUyWTRmF8lkkVyQRfJuZ2rMD7RgEyGeEy6kuIhGrz/t9fvj8ky8WfG3WMAJd911Xpf38EyFhJMfHok7IoqSsEjc4c4U4i4mbtcG8PTs9nbsS2FyCkTHQEwMUi8G6sWwPTuGrXtjaH9mDK7OzrLfTGvW0AxqdJ8MKzDGmDpX297UkXVUVKhEPP/MD8yA+U4j6deP08qtq1EjePfdX4vUuHHHPtvvX79pEMPSpWn075/GCs96PvnEGbz78GHn1kV5ebB96VJajeqPFBWiYRF8cdccdjQ5nYLDJaXTks9LWPZlCSGUEEYJHdqW0DqthOKCEkoKS0jZt5Jb1t5KqBZRIuE8mzyeLSEtCSkqIKSogNDiAjQvn9CSAiLxTEWeiV+nDqym/pETOtRNyXdrKGAjMRwunUJxcwrQDeD9yn9UTaBZ5S28swJjjAlate1RVe3YFDUudtHRZRq5MsD1a6M+XtbVrVyv7JVJ5TP9DpZ2LC12d1ShZ/fBB06xKypy5hUVQchXS+Hq/qin2O3+9ztkn5ZRuryoUPnP64X8783DROlh6occ5q7/d5iRVx52quaR6a234O23fz2PvpqswBhjTmhVKlJ1tZ66OBmjFj27UikZ0OzXnl2H3zQSkhpF8tbcSA4UJpAdAe2uBMo3S02F999H8/JqVGGswBhjjB/VRcHzZ7Fj3jyyevbcXpOcVmCMMeYkVdVCtQ121mT9ITV5kTHGGHMsVmCMMcb4hBUYY4wxPmEFxhhjjE9YgTHGGOMTVmCMMcb4hBUYY4wxPmEFxhhjjE9YgTHGGOMTVmCMMcb4hBUYY4wxPuHXAiMiiSIyR0QOichmERlaSdvbRWSniBwQkckiElmT9RhjjAkMf/dgngMKgWRgGPCCiLQv30hEBgH3Av2BNKAV8FB112OMMSZw/FZgRKQecCnwgKrmqupiYC5wtZfmw4FJqrpGVbOBh4ERNViPMcaYAPHncP3pQImqZpaZtwro66Vte+Ddcu2SRaQhkFqN9SAio4BRnqe5IvJjDfMfkQTsqeU66lIw5bEs3gVTFgiuPJbFu2DKAtCmJi/yZ4GJBQ6Um3cAqF+Ftkce16/melDVl4CXqhu2IiKyXFXPqKv11VYw5bEs3gVTFgiuPJbFu2DKAk6emrzOn8dgcoG4cvPigJwqtD3yOKea6zHGGBMg/iwwmUCYiLjKzOsMrPHSdo1nWdl2Waq6t5rrMcYYEyB+KzCqegh4GxgvIvVEpBdwEfCal+avAiNFpJ2IJAD3A1NrsB5fqLPdbXUkmPJYFu+CKQsEVx7L4l0wZYEa5hFVresgFW9MJBGYDJwL7AXuVdXXRSQV+AFop6pbPG3vAO4BooHZwE2qWlDZevz2RowxxhyTXwuMMcaYk4cNFWOMMcYnrMAYY4zxCSswVSQikSIyyTP2WY6IfCsi5wc4k0tE8kVkeiBzeLJcISJrPePDbRCR3gHKkSYi74tItmcsu2dFxC/Xe4nIaBFZLiIFIjK13LL+IrJORA6LyAIRaRGILCLSQ0Q+EZF9IrJbRN4SkVN8maWyPOXajBMRFZEBgcoiIjEi8ryI7PGMg7gogFku9/xN5YjIDyLyRx9nqfQzria/w1Zgqi4M2IozYkA88ADwpoikBTDTc8CyAG4fABE5F/gHcC3OBa99gJ8DFOd5YBdwCtAF5+d1i5+2vR14BOcElFIikoRz5uMDQCKwHJgViCxAAs4ZQWlAC5zrx6b4OEtleQAQkdbAEGBHgLO8hPMzauv5ensgsohIM2A6cAfOdX53Aa+LSGMfZqnwM66mv8P+vJL/uOY5PfrBMrP+JyIbgW7AJn/nEZErgP3AEuBUf2+/nIeA8ar6pef5tgBmaQk8q6r5wE4R+RBn6CGfU9W3AUTkDKB5mUWXAGtU9S3P8geBPSJymqqu82cWVf2gbDsReRb4zBcZqpKnjGdxzhp9PlBZRKQNcCHQXFUPemavCEQWz+P9ZX5e74nIIaA1zj9QvshS2WdcQ2rwO2w9mBoSkWSc8dX8foGniMQB44E7/b1tL1lCgTOARiLyk4j84tktFR2gSE8DV3h2dTQDzgc+DFCWI9rjjJcHlP4hb8BPhe8Y+hDgi5RF5DKgUFXfD2QOoDuwGXjIs4vsexG5NEBZlgNrReRCEQn17B4rAL7zV4Byn3E1+h22AlMDIhIOzACm+eo/0GN4GGe06a0B2HZ5yUA4zu6N3ji7pbriXBwbCJ/h/NIfBH7B+UN9J0BZjqjW+Hn+IiKdgL/i7H4JVIZY4FFgTKAylNEc6IDzs2kKjAamiUhbfwdR1RKcC85fxyksrwM3ej7Yfc7LZ1yNfoetwFSTiITgjBpQiPML6O/tdwEGAP/097YrkOf5+m9V3aGqe4CngN/7O4jnZ/MRzr7iejgj0ibgHB8KpKAbP09ETgU+AG5T1c8DlQNn9+prqroxgBmOyAOKgEdUtVBVPwMWAAP9HcRzosPjQD8gAue4yCuev39fb9vbZ1yNfoetwFSDiAgwCee/9ktVtSgAMfrhHKDdIiI7gbHApSLyTQCy4Llfzy9AMFyxmwik4ByDKfCMXTeFABS7co4aW0+cexq1JkC7pjxn/3wKPKyq/hpiqSL9gVs9Z/ztxPn5vSki9wQgi992P1VBF2CRqi5XVbeqLgO+wvnn0mcq+Yyr0e+wFZjqeQHn7JLBqpp3rMY+8hLOD7aLZ3oReA8YFKA84HyI/1lEGoszdtwY4H/+DuHpPW0EbhaRMBFpgHPzulWVvrCOeLYZBYQCoSISJc4p0nOADiJyqWf5X4HvfLl7taIsnuNS84HnVPVFX22/qnlwCkwHfv193g7ciHOGpL+zLAK2APd52vTC+YfuowBkWQb0PtJjEZGuOLugfV0EK/qMq9nvsKraVIUJ55ROBfJxuotHpmEBzvUgMD3AGcJxzv7ZD+wEngGiApSlC7AQyMa5YdNbQGM//iy03PSgZ9kAYB3ObpiFQFogsgDjPI/L/g7nBvJ7U67dJmBAAH9O7YGlwCGc8REvDmCW0cBPOLuhfgbu9HGWSj/javI7bGORGWOM8QnbRWaMMcYnrMAYY4zxCSswxhhjfMIKjDHGGJ+wAmOMMcYnrMAYY4zxCSswxhxHxLlfypBA5zCmKqzAGFNFIjLV8wFffvry2K825uRj94Mxpno+Ba4uN68wEEGMCXbWgzGmegpUdWe5aR+U7r4aLSLveW4ru1lErir7YhHpKCKfikieOLctnioi8eXaDPfci6RARLLkt7cYThTnVseHRORnL9v4q2fbBZ5BJF/1xTfCmGOxAmNM3XoImIszJtpLwKueuxUiIjE4Nz/LBX4HXAz0pMztckXkRmAizgCinXBGgi4/Yu1fgXdxRredBUz2jJCM5wZZY3FuE+0C/gB8Xfdv05hjs7HIjKkiT0/iKpzBAMt6TlXvEREFXlHVG8q85lNgp6peJSI3ABNwbsmb41neD+eeIy5V/UlEfsEZvPTeCjIo8HdVvc/zPAzn5mqjVHW6iNyBMxpxBw3M7SSMKWXHYIypnkXAqHLz9pd5vLTcsqXABZ7HbXGGOC97k6YlgBtoJyIHgWbAvGNkKB2yXVWLRWQ30Ngz6y3gNmCjiHyE02Oaq6oFx1inMXXOdpEZUz2HVfWnctOeKr5WqPjGbOpZXhXleyaK529Zndtot8HpxRwEngRWeG4QZYxfWYExpm718PJ8refxD0BnESl7H/OeOH+Ha1U1C9iGcxOuGlPVfFV9T1VvB87EucdJr9qs05iasF1kxlRPpIg0KTevRFV3ex5fIiLLcG7INASnWHT3LJuBcxLAqyLyVyAB54D+26r6k6fN34B/ikgWzp1KY4D+qvpkVcKJyAicv+uvcE4m+BNOj2d9Nd+nMbVmBcaY6hkA7Cg3bxvQ3PP4QeBSnLt67gauVed+6qjqYREZBPwL58yufJyzwW47siJVfUFECoE7gX8A+4D3q5FvP3APzskE4Ti9pktUdWM11mFMnbCzyIypI54zvC5T1f8EOosxwcCOwRhjjPEJKzDGGGN8wnaRGWOM8QnrwRhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPGJ/w9iJ2/B+AXG0AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_learning_curves(loss, val_loss):\n",
    "    plt.plot(np.arange(len(loss)) + 0.5, loss, \"b.-\", label=\"Training loss\")\n",
    "    plt.plot(np.arange(len(val_loss)) + 1, val_loss, \"r.-\", label=\"Validation loss\")\n",
    "    plt.gca().xaxis.set_major_locator(mpl.ticker.MaxNLocator(integer=True))\n",
    "    plt.axis([1, 20, 0, 0.05])\n",
    "    plt.legend(fontsize=14)\n",
    "    plt.xlabel(\"Epochs\")\n",
    "    plt.ylabel(\"Loss\")\n",
    "    plt.grid(True)\n",
    "\n",
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QxuDWH50Fpaa",
    "outputId": "d031e1df-4c55-47be-a303-94a4ece089ad"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA+5ElEQVR4nO3deXiU1dn48e89mSyQkJCNsCYhLAmLAgZqFJXgAlVrtVXrglZbldalv76tttrF1lrfLr7dfX1tFa1Kca+7rQJKFJAtUUASSIBAAkI2EiAL2WbO74+ZpCFkzyzPZO7Pdc0l8yxnzhxncs/znHPuI8YYlFJKKW+z+bsCSimlgoMGHKWUUj6hAUcppZRPaMBRSinlExpwlFJK+YQGHKWUUj6hAUcppZRPWDLgiMhdIpIrIk0i8nQvx35PRMpE5JiIPCUi4R32xYnIayJSLyIlInK91yuvlFKqS5YMOMAh4CHgqZ4OEpHFwH3ABUAqkAb8osMhjwLNQBKwBHhMRGZ4ob5KKaV6IVbONCAiDwHjjTE3d7P/OWC/MebH7ucXACuMMaNFJBKoAWYaY4rc+5cDnxtj7vPJG1BKKdXO7u8KDNIM4I0Oz7cBSSISDyQDjrZg02H/gq4KEpGlwFKAiIiIzOTkZO/UOIA4nU5sNqteBPuOtoO2QRttB5fu2qGoqKjKGJPY3XmBHnCigGMdnrf9e0QX+9r2j+iqIGPM48DjAOnp6aawsNCzNQ1AOTk5ZGdn+7safqftoG3QRtvBpbt2EJGSns4L9FBdB0R3eN7279ou9rXtr/VBvZRSSnUS6AEnH5jV4fksoNwYcwQoAuwiMqXT/nwf1k8ppZSbJQOOiNhFJAIIAUJEJEJEurr99yxwi4hMF5FY4KfA0wDGmHrgVeBBEYkUkfnA5cByn7wJpZRSJ7FkwMEVOE7gGvJ8g/vfPxWRZBGpE5FkAGPMu8DDwBqgxP34eYdy7gCGARXA88Dtxhi9wlFKKT+w5KABY8wDwAPd7I7qdOwfgD90U041cIUHq6aUUmqArHqFo5RSaojRgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ+wbMARkTgReU1E6kWkRESu7+a4v7qXnW57NIlIbYf9OSLS2GF/oe/ehVJKqTaWXGLa7VGgGUgCZgPviMg2Y0x+x4OMMd8Gvt32XESeBpydyrrLGLPMq7VVSinVI0te4YhIJHAlcL8xps4Ysw54E7ixj+c94/1aKqWU6g9LBhxgKuAwxhR12LYNmNHLeVcClcBHnbb/WkSqRGS9iGR7rJZKKaX6zKq31KKAY522HQNG9HLeTcCzxhjTYdu9QAGu23PXAm+JyGxjzN6OJ4rIUmApQGJiIjk5OQOv/RBRV1en7UDX7bCnxkHBEQfT40OYHBvin4r5kH4WXLQdXAbaDnLy32ZrEJE5wHpjzPAO2+4Gso0xl3VzzgRgHzDVGFPcQ9nvAu8YYx7p7pj09HRTWKhjC3JycsjOzvZ3NfyuczvkldRw7eMbaHEYwu02nrsti8yUWP9V0Af0s+Ci7eDSXTuISJ4xZm5351n1lloRYBeRKR22zQLyuzke4OvAxz0FGzcDyCDrp4LYR0WVtDhcP9SaWp1sLD7i5xopFRgsGXCMMfXAq8CDIhIpIvOBy4HlPZz2deDpjhtEZKSILBaRCBGxi8gS4DzgPS9VXQWBiuONJz2PCLXk10gpy7HyN+UOYBhQATwP3G6MyReRZPd8muS2A0XkLGA88HKnMkKBh3ANJKgCvgNcYYzR+2VqQI42NPP29sPMS43l+xdNYXR0OM9vPkCLo/NIfKVUZ1YdNIAxphq4oovtpbgGFXTctgGI7OLYSmCel6qogtBfPyymrrmVX14xk4zR0UwfE8Otz+ayfEMJ3zxnor+rp5SlWfkKRylLqTjeyNMf7+PyWWPJGB0NwAXTRnHe1ET+uLqII3VNfq6hUtamAUepPvrfNXtodRj+68Kp7dtEhJ99aRonmh38bmVRD2crpTTgKNUHB6obeH5zKV+bN4HUhJPv3k4eNYKvn5XKC1tKyT/UefqYUqqNBhyl+uBPq3cjIvy/86d0uf+7F04hdngYv3izACvObVPKCjTgKNWLz+ucvPbpQW46K4XRMRFdHhMzLJR7FqWzeX81b28/7OMaKhUYNOAo1YvXdjczLDSE27Mn93jcNfMmMGNsNL94M58/ry4ir6TGRzVUKjBowFGqBy9tKSW33MGlp48hLjKsx2NDbMJ185Kpqm/mj6t3s2TZRg06SnWgAUepbuSV1HDfq58B8ObWQ30KHscaW9rzJjW2OFm3u9KLNVQqsGjAUaobqwrKcLr7/1scfcuZlpUWT3iorT3ovLXt0CmpcJQKVhpwlOqOO9gIEGq3kZUW3+spmSmxrLg1i3sWp3PfxRkcOtbIFY+up+DQce/WVakB2LzvCI+u2eOzW7+WTW2jlL/tqawnISqMBWMM1184r89LEGSmxLYfe+6UBG55Oper/voxf7l2DhdOT/JmlZU6RV5JDRuLj5CVFs/kUVFs3lfNx3ureL+ggtKaBgDCQmw8f9uZZKbGebUuGnCU6kJji4P1e6q4KnM8F4ysGvB6NzPGxvDGXfO59Zlcbn02l4Xpidx1/pQhv36Osoa8khquf2Ijza1O16W6cV24R4TaGB0d0baJZoeTO577hJ99aQZfnDmaEJt3VnDRgKNUFzbtq+ZEi4PzM0ZBWdWgykqKjuBHF2dww5ObWFNYyfq9R3g+CBZtU/63sfgITa3uTOYGzpoUz3cvmMKc5JHs+Pw4S5ZtpKXVic0mCHDnc5+QGj+cxTNGMzwshHOmJHr0c6oBR6kurNlVQUSojbMmxbOxbPDlfXrgaPu/W9yLtmnAUd42KdGVhkmA8FAb9yxKb//ctfU3tt1umz1hJO/ll/H7lYX87SPXOpaPfbiXFbd67seRBhylOjHG8MGuCs6elEBEaIhHysxKiyfMbqOxxYkIfRqAoNRglR93ZTC/5ZyJXHzamFMCR8f+RoBLThvDvqo6fr+yCKfx/I8jHaWmVCd7K+sprW5gYcYoj5XZ9msyffQI4iLD9OpG+cSqgnLSEiP56Zem9/kzl5WWQJjdRoh0MTpzxQpITWXB+edDaqrreT9owFGqkzW7KgBc/TcelJkSy9WZ46mqa9a5Ocrrjje2sLH4CBf1c2Rk24+j7y9KP/l22ooVsHQplJQgxkBJiet5P4KOBhylOllTWEF60gjGjRzm8bLnJI8ETu7TUcobPiyspNVpuGha/4fiZ6bEcufCySdfFf3kJ9DQcPKBDQ2u7X1k2YAjInEi8pqI1ItIiYhc381xN4uIQ0TqOjyy+1uOUgC1jS1s3ldNdkaiV8qfMTaG0BBhqwYc5WWrd5YTHxnGnGQP3b4tLe3f9i5YedDAo0AzkATMBt4RkW3GmPwujt1gjDnHA+WoILdudxWtTsP56Z69ndYmIjSEaWOi+bRUk3oq72lxOFmzq4LFMzw4pyY52XUbravtfWTJKxwRiQSuBO43xtQZY9YBbwI3+qMcFTw+2FVBdITdq536cyaMZPvBYziculCb8o4t+6o53tjq2cwW//3fMHz4yduGD3dt7yOrXuFMBRzGmI6LxG8DFnRz/BwRqQKqgeXAr40xrf0pR0SWAksBEhMTycnJGfSbCHR1dXVB1Q5OY3jvsxNkxNlYt/aj9u2ebofw+lYamh08984aJoyw5G++UwTbZ6E7gdIOK3Y2YbcBh3eSU7nLM4WOG8eo732PtGXLCK+ooGnUKIpvvZWKceOgr21ijLHcAzgXKOu07TYgp4tj04CJuK7WTgMKgB/1t5yOj6lTpxplzJo1a/xdBZ/adqDGpNz7tvln3oGTtnu6HfZV1pmUe982KzaWeLRcbwq2z0J3AqEdnE6nOee375tv/H2z116ju3YAck0Pf1ut+vOqDojutC0aqO18oDGm2BizzxjjNMZ8BjwIXNXfcpT6YFcFIrBgqncGDLRJiR9O7PBQth7QfhzleYXltRyoPtHv4dC+YNWAUwTYRWRKh22zgL509BtoX45kMOWoILNmVwWzJ4wkPircq68jIsyeMJJPS4969XVUcFpdUA7ABR6eR+YJlgw4xph64FXgQRGJFJH5wOW4+mdOIiIXi0iS+98ZwP3AG/0tRwW3ytomth085rXRaZ3NSY5lT2UdxxtbfPJ6Knis2lnBrAkjGRUd4e+qnMKSAcftDmAYUAE8D9xujMkXkWT3XJu2sXgXANtFpB74F64A86veyvHVm1CBIafQlV3Ak+lsejJ7wkiMge0Hjvnk9VRwqDjeyLYDR7lomvWubsC6o9QwxlQDV3SxvRSI6vD8HuCe/pajVEc5hZWMGhHOjLGdu/y8Y9aEkQB8WlrDOVMSfPKaauhbvdP1w+mi6aM9VubDD8O8ebBwYffHrFkDW7b0XpaVr3CU8okWh5OPiipZmD4KEe8sPNVZzLBQJiVGasYB5VGrd5YzIW4YU5Oiej+4j+bNg699zRVUurJmjWv/vHm9l6UBRwW95zaVUtvUSkr88N4P9qA5ybF8euBo23B9pQalobmVdXuquHBakkd/OC1cCC+91HXQaQs2L73U8xVQGw04KqjlldTw4NsFAPzl/d3klfhuqPKc5JFU1zdzoPqEz15TDV1rd1fR3Or0ynDoroJOf4MNaMBRQW79nqr2FDMtDtdiU74yu60fR+fjKA9YVVBOdISdealxXim/Y9B56qnUfgcb0ICjgpzd/Q2wdbXYlJelJ41gWGiIzsdRg7ZlfzXvbD/MrAkjCQ3x3p/1hQvh9tth+fJUbr+9f8EGNOCoIJdbcpS44WF8/6KpHl27vS/sITZOGx+ja+OoQckrqWHJsk2caHGwsfiIV28Lr1kDjz0GN964n8ce634gQXc04KigVXaskZzCCq47cwJ3nT/FL8s+z0keScGhYzS2OHz+2mpo2Fh8hJZWJwBOp/HabeGOfTbf/Ob+bgcS9EQDjgpa//zkIE4DV2dO8Fsd5kwYSYvDUHD4uN/qoAJbVlo8bYPSvHVbuKsBAj2NXuuOBhwVlIwxvJx7gC9MjCM1IdJv9WhbjVH7cdRAzZkwkojQEE4fF+OV28I9jUbrb9DRgKOC0pb9New/0sA1c/13dQOQFB3B2JgIXQFUDdieyjoamh3ceFaKV24Lb9nS82i0tqCjmQaU6saLWw4QFW7n4tM8lwJkoGYnj9SMA2rAcve7fqzM9dJw6B/+sPfRaAsXuo7rjQYcFXRqG1v412eHuWzWGIaH+T+d4JwJsRysOUFlbZO/q6ICUG5JNfGRYaT6OFPGQGjAUUHnne2HOdHi4Go/305rMzt5JIBe5agBySupITMl1md5AAdDA44KOi/mHmDyqCjmuGf6+9vMsTGECCxbW+zT1Doq8FXWNlFypIG5qb4f0j8QGnBUUNldXsunpUe5Zu4Ey/wiLDh8HCewaV81S5Zt1KCj+iyvpBqAzBTv9N94mgYcFVRezjuI3SZcMWecv6vSbmPxEdfC6EBLq2/zuanAlru/hjC7jZnjfLOO02BpwFFBo8Xh5NVPDnJ+xigSR4T7uzrtstLiCXUndbPZxKf53FRgyy2pYfb4kYTbQ/xdlT6xbMARkTgReU1E6kWkRESu7+a4m0QkT0SOi8hBEXlYROwd9ueISKN7Weo6ESn03btQVvLBrgqq6pr5mkUGC7TJTInluVvPJDIshLkpsX5JsaMCz4lmBzs+P0ZmgPTfgIUDDvAo0AwkAUuAx0RkRhfHDQf+C0gAzgQu4NQlp+8yxkS5H+neq7LypLySGh5ds8djfRpPfFRMZFgI0cP8PxS6s7mpcVw+ZxzbDmpeNdU32w4epdVpmBtAP1AsGXBEJBK4ErjfGFNnjFkHvAnc2PlYY8xjxpi1xphmY8znwApgvm9rrDwtr6SGax/fwO9XFg66I72uqZUfvLyV3JIaGpodfP2pzZbsmF80PYmGZgcf763yd1VUAGj7DAfSFbH1fuq5TAUcxpiiDtu2AQv6cO55QH6nbb8Wkd8AhcBPjDE5nU8SkaXAUoDExERyck45JOjU1dV5rB321DjYVe0gIy6EybG932/+Xe4JWhyunvSmFifPr95C7aSwfr1mq9OQc6CVN/Y2U9vs2maA5n6W58l26EmL0xARAk+v3oqtzDp9TOC7NrA6K7XDe580MjZS2Lr5Y5+/9kDbwaoBJwo41mnbMWBETyeJyDeAucCtHTbfCxTguj13LfCWiMw2xuzteK4x5nHgcYD09HSTnZ09mPoPCTk5OXiiHfJKavjtqg04nIYwu6PXBIOPf7SXHVW7sAm4F+PkyuwzOGtSQp9ea2NxFQ4nvPrJQfYfaebMiXF85YxxPPBmPi2tTkLtNq67cF6ffxl6qh364sKyT9hYfIRzz1tAiM0aw7bBt21gZVZpB6fT8P9yVnLJaePJzj7d568/0HawasCpAzqP84sGars7QUSuAH4DXGiMab8nYYzZ1OGwZ0TkOuAS4BGP1Vb16J3PDp90tbJ+T1W3f+yXrS3mV//axaWnj+Gms1J4OfcgL+cd5O3th3sNOHklNVz3xEaa3WuDJMcO5+83zyM7PRERYcqoEWwsPkJWWrxlb0MsmjGat7cf5tPSGq/lxlKBb09lHccbWy37Oe6OVQNOEWAXkSnGmN3ubbM49VYZACLyReAJ4FJjzGe9lG0A6/x0DAIlVfXt/za4rjwWzxhN+uiTL1ifXLePh97ZyaWnjeHP18zGHmLjCxPjiYsM428fFXPO5AQuPm1Mt6+zfMP+9mAjwNfmjWdhxqj2/ZkBMAIsOz2R0BBhZUG5BhzVLW8n7PQWSw4aMMbUA68CD4pIpIjMBy4Hlnc+VkTOxzVQ4EpjzOZO+0aKyGIRiRARu4gswdXH857334UCKD/eyNrdVSyekcQPFqdz38UZ1DW1ctn/rmPZ2mKc7ntmf1+/j1++XcDFM0fzp2tdwabN3YvSmTU+hnv/uZ2DNQ1dvs4zH+/n9a2HEIEQgfBQW59uwVlNdEQoZ01KYGV+GcYYf1dHWVQgJezsyKpXOAB3AE8BFcAR4HZjTL6IJOPqk5lujCkF7gdigH91SFWy1hhzMRAKPARkAA5gF3CFMUbn4vjIU+v30ep08uNLppES71ro7KrM8fzo1c946J2dvPbpQWKHh7Nujyso/eW6OYSGnPw7KMxu45HrzuCSv6zluy9s5cWlWe0Byek0/Pa9Xfztw2IWTU/i5rNT+fTAUUvfNuvNoulJ/PT1HeypqGNKUo/dlipI5ZXUMDc1MBJ2dmTZgGOMqQau6GJ7Ka5BBW3Pu12pwRhTCczzRv1U7443tvDcxlIuPm1Me7ABSIgK5/EbM/ndykIeXbMXqMUm8M35E08JNm2S44fz31+ZyXdf2MqfVu/mnsXpNLc6+eEr23h96yFuzErhgS/PIMQmnD058K5sOrrIHXBWFpRrwFGnqKhtpORIAzecmeLvqvSbZQOOCnzPbSqltqmV2xdMOmWfiDA8zN4+Ek1wpek4s4e0LpfPHsf6PVU8mrOH4eEhvLj5ACXVDfxgcTp3ZE8KuF973UmKjmDWhJGszC/jzoWT/V0dZTF57v6bQMow0MaSfTgq8DW1Onhq3T7OmZzAzHExXR6TlRZPmN1GiECo3danHGIPfHkGY2MiePjdQkqqGwgNceUeGyrBps2i6UlsO3iMw8dO+LsqymJyS2oIt9uYObbr75WVacBRXvH6p59TUdvEtxakdXtMZkosK27N4vuL0nudm9NmeJidi6b/Z1lop9MMyezKi2ckAbC6oNzPNVFWk1tSw6zxIwmzB96f78CrsbI8p9Pwt4+KmTE2mnN66U/JTInlzoWT+9XBf9mssUSE9u/KKNBMSowiLSGSlRpwVAcnmh3kB1jCzo60D0d53Kqd5RRX1vOX6+Z45VZX25WR1SdxDoaIcNGMJJ5cu49jJ1qIGRbq7yopCwjEhJ0defwKR0QeEZG3utgeLSIPiMi0Dtu+JyLbRUSvtIYIYwx//XAvE+KGccnM0b2fMEADuTIKNIumj3blgyus8HdVlEW8ue0QAHYLpT3qD4/+oReRScC3gF90sXsu8HNcc2Pa/BUYBdzkyXoo/9myv4ZPS49y27lpJ03eVP03Z8JIEqLC9baaAlxzb17YXArAt/6RZ8mM573x9F+E/wK2GWNyu9g3B2jCNWkTAGPMCeBZTl2/RgWgvJIafvTqdkZE2Lk601qLnAUim024aHoS7xeU85f3iwLyD4zynPV7KtuT2QbqUuR9CjgiMllEWkTkF522PyYitSIyV0TCgRuA57o4fyfwOyAcaBERIyKvuHe/AEwXkbMH9U6UX+WV1HD9ExvZW1nPiWYHBYeP+7tKQ0JaYiSNrU7+uGr3oNcFUoEtMtzV5W4L4MEyfQo4xpg9wDLgeyKSACAiPwO+CXzFfUWTBYwE1nZRxNeBYuAt4Cz34273vq3AceCLA30Tyv82Fh9pT5xpzNAcquwPbat/GqA5QH/VKs/YV1VPWIiN714wpc/TCKymP7fUfgGEAPeKyC24+mNuNMasdu/PwvW92N7FuduA8cAHxpiN7kcJgDHG6T4na4DvQVlAx19bgfrry4rOnpRAhHu+hdNA7PD+LUKnhgZjDKsLKliYkch3L5wakMEG+hFwjDFlwJ+A7wB/A/6fMealDoeMBY4bY5q7OH0GEAZ80k3xle7zVYAaN3IYBlgwNTFgf31ZUWZKLCtuy+KO7EmMjg7nt+/uoqi822Wh1BC14/PjlB1vPGnScyDq76CB3bj6YTYYYx7ttC8C16CArpyB6+pnazf7TwDD+lkXZSHr9rjWvLv3ixkabDwsMyWWH34xg5e/fTbhdhtff3Jzt8s0qKFp1c5ybALnd1jfKRD1OeC41535G7ABmC8iszodcgTo7i/NHGCvMaa7nuQ4oKqbfSoArNtdSUJUGBmjNbuxt0yIG86zt3yBhuZWvv7kZo7Udff7Tg01qwrKmZsSR1xkYN9S7esotTOA13ENHMgGSoFfdTpsFxAqIuO7KGI6HYZDd2EioGvUBCin07BuTxXnTE7AFqAT0gJFxuhonrp5Hp8fPcHVf93AH1fpcOmh7kB1AzsPH+ei6Un+rsqg9RpwRGQy8G9gJfAddx/NL4BLROS8Dod+5P7vF7oo5igwy736ZpaItPcoi8hIYGqH81WA2VVWS1VdM+dMSfR3VYLC3NQ47l40leKqev78/m6u+dsGnlxXTG1jS/sxeSU1PLpmjwajIeD9na6JvxcOgYDTYy41ERmNK9DsBJa4R5SBa7LmD4HfAGcDGGP2i8hm4DJcy0N39DPgSVxXSRHAucA6975LgWbgtUG+F+Una3dXAvSaqFN5TovDILg6Rludhl++vZP/fmcn08ZEkxI/nFUF5TicBrvNxs++NJ1pY0cgIuwuryX/0HFmTxjJ6eNjsNtshNiE0BAb+YeOkX/oOPMnJ5zUD5dXUsPbe5sZMbFG++f8YNXOciaPimJiQmTvB1tcjwHHPTLtlPzyxhgHMO3UM3gM+LOI3GmMaehw/A7gzG5e5gbgZWPMSRMMRCQOV5BahKt/50fGmFMmlbqP/R5wL66BB//EtRx1U3/LUQOzbk8VU5OiGB0T4e+qBI2stHjCQ220tDoJDbFx38UZ1DS0kFtSzaqCclocrinpzQ4nP31jxynnP7uhpNuy/7CqiJT44UxMiCREhA+LKnE4DW/v36gjEH3s2IkWNhVXc9t53S/zEUg8nS16Oa4rnztwZRbokYjMBhYCM7vY/SiuK58kYDbwjohsM8bkdypjMXAfcD5wCNeV0i/c2/pcjhqYxhYHm/ZVB+Ryt4Gsp4zZm/cd4YYnN9PqcGK32fjRxRlMGhXFG9s+59W8zzG4ZqtfctoYzs8YRavTsLqgnFUF5Rhcq6/abcKRumaKK+todedTaZt4qgHHd3IKK2h1Gi6cFvi308DDAccY4xCRb+IaBt0Xo4FvuDMZtBORSOBKYKYxpg5YJyJvAjfyn0DS5ibgybYAIiK/BFYA9/WzHDUAW/ZX09zq5NypejvN1zJTYrv84/+FifE8f9upwSgy3M472w+7rorsNr4xf2L7vkmJUXy0u7J938NXzSIzJZa8khqWPLGRRncWCZ3Q61urCspJiApjzoSR/q6KR4gxxt91OIWIzAE+NsYM67DtHmCBMeayTsduA35ljHnR/TwB10TSBCC5H+UsBZYCjBg1ITPuG4955b0ppdRQVfLbL+UZY+Z2t9+qC7BFAcc6bTsGdDXJo/Oxbf8e0Z9yjDGPA48DpKenm8LfXNr/Wg8xOTk5ZGdn93jMxX9eS8wwOy8sPcs3lfKDvrTDUPf0G+/zwIZGfnnFTG7MCt7bp778LKzdXcmNT25m2dfnWm6EWnftIL/t+TyrLlhSB0R32hYNdJXTo/Oxbf+u7Wc5qp8qa5vYefg45+pw6CEvJdpGxugRvJJ7wN9VCRqrCsqJCLVxzpShc7vaqgGnCLCLyJQO22YBXXX057v3dTyu3D3qrT/lqH5a705nc+4Q+kKorokIV8+dwLaDxzSXmw+4knWWc+6URCJCQ/xdHY+xZMAxxtTjmsvzoIhEish84HJco+A6exa4RUSmi0gs8FPg6QGUo/pp7e4qYoeHMmNsjL+ronzgitljsduEl/Uqx+vyDx3n0LHGIZFdoCNLBhy3O3DNq6kAnsc1tyZfRJJFpE5EkgGMMe8CDwNrgBL34+e9leO7tzE0GWNYu7uSsycnEKLpbIJCfFQ452eM4rVPP6fF4ez9BDVgq3eWIwIXBHiyzs6sOmgAY0w1cEUX20txDQbouO0PwB/6U44anN0VdVTUNnGuZhcIKlfPncDKgnJyCiuH3K9vK1lVUE5mcizxUeH+ropHWfkKR1nYR0XudDbafxNUstMTSYgK09tqXvRefhn5h44zbUzn8U6BTwOOGpB1e6pIS4hkfOxwf1dF+VBoiI2vzBnHB7sqdHkEL8grqeHOFa51Kl/KPTDkkq9qwFH91tTqYFNxtY5OC1JXz51Aq9Pw+tZD/q7KkLOx+Eh7KqFWhyuV0FCiAUf1W15JDSdaHLocQZCamjSCWeNjeDn3AFbMVBLITnOP+BQg1G4bcqmENOCoflu3u4oQm5CVFufvqig/uSpzPLvKXEsdKM850uC6TbkkK2VIZubWgKP67d0dZYyODqeovM7fVVF+8uVZ4wiz23TwgIetzC8nKTqcB788Y8gFG9CAo/rpw8IKiqvqOXS0kSXLNg65Tk3VNzHDQ1k0PYk3th2iqdXh7+p4TE8rpbYtROetz3xji4MPi1zDzYfqUu2WnYejrOmNba6OYgO06PooQe3quRN4e/thfvDydm46OzXgPwd5JTUsWbaR5lYnITbhuxdMYWKCa8pfcWUdf/lgN60O7y1Et253FQ3NDhbPGO3Rcq1EA47qF6d7gnmIDM1OTdV3w8NcOb7e3HaIlQVlAd/nsLH4CE0tTgzgdBh+t7Koy+O8tRDdyoIyRkTYOXPi0P1OacBR/XKwpoGpSVFcPnvcKStNquCyeV91+7+HwtXumJgI2sbchdtt/OarpzN9bDQiUHDoOD98ZTvN7pQ+Z0707IAZh9OwemcF52eMIsw+dHs6NOCoPmtscbD94DFunp/KnQsn+7s6ys+y0uIJCxGaHYYQW2Bf7Ta3Onli7T5ihtm5ISuF8zOSTgqeU5NGMCFuOL9+dRO55Q5yS2qYm+q5oJNXUkN1fTOLpg/d22mggwZUP+z4/BjNDmdA/4pVnpOZEsvT3/gCITZh0fSkgP5c/Pn9InYePs7vrp7NDxZndPleMlNiuXN2OJecNprfryzks4Od13YcuPfyywiz21iQPrTntmnAUX22Zb9rdM7cAP7Dojzr7MkJnD0pnp1lgTsf55PSGh7L2cvVmeN7TUgqIvzqK6cRHxnOd1/8lIbm1kG/vjGGlQVlnDM5gajwoX3TSQOO6rPc/dWkJUYOuQy2anCy00ext7KeA9UN/q5KvzU0t3L3S9sYEzOMn102vU/njBwexh+umcW+qnoeemfnoOuwq6yWA9UnWBQE2bc14Kg+cToNuSU1zEvR7ALqZNnu20A57gzigeS3/97Fvqp6/ufq0xkREdrn886elMDS89J4blMpK/PLBlWHlfnutW+macBRCoA9lXUcO9HC3FS9naZOlpYQyYS4YXxYWOHvqvTL+j1VPLOhhJvPTuXsSf1PRHv3RenMHBfNvf/cTsXxxgHX4738MjKTY0kcMfTvHGjAUX2S6+6/mefBkTlqaBARsqeO4uO9RwIm68Da3ZV8e3keY2MiuPeLGQMqI8xu40/XzOFEi4Pbns3lfz/Y3e8sBAeqGyg4fJxFM4b+1Q1YMOCISJyIvCYi9SJSIiLX93DsTSKSJyLHReSgiDwsIvYO+3NEpNG9JHWdiBT65l0MPbn7q0mICiMlXte/UadaMDWRhmZH+w8TK8srqeHmp7ZQ29RKVX0zBYcHPuBh8qgobjorlW0Hj/G7lUVc/0T/0j2tKigHGPLDodtYLuAAjwLNQBKwBHhMRGZ0c+xw4L+ABOBM4ALgnk7H3GWMiXI/0r1T5aFvS0k1c1PiEBmaOZ7U4Jw9OZ6wEBs5AXBbbe3uShzuZRUcHlhzJnqYnbZvRVOrk7+v39fnZRtWFpSRnjSC1ITIQdUhUFgq4IhIJHAlcL8xps4Ysw54E7ixq+ONMY8ZY9YaY5qNMZ8DK4D5vqtxcCg71siB6hPaf6O6NTzMzhcmxpFTaP2BA20z+W0eSs+UlZZAeKgNm4AIvL39MLc9m9drv051fTOb91UHze00ALHSAkoiMgf42BgzrMO2e4AFxpjL+nD+68AuY8x97uc5wAxc6xkVAj8xxuR0c+5SYClAYmJi5ksvvTSo99IXe2oc7Kp2kBEXwuTYEK+/Xn/V1dURFRXF5sOt/N+2Jn52VgRpMdarp7e1tUMw60sbvLuvhRcKm/n9gmHED7PUb9mTLPusic1lrXxpYijT4/v33euuHdq+y1PjbOw9anh1dzOhNrgw2U5oiDCti+/42oMtPLmjmQfOiiA1wL5X3bXDwoUL84wxc7s7z2qzjKKAztN3jwEjejtRRL4BzAVu7bD5XqAA1y26a4G3RGS2MWZv5/ONMY8DjwOkp6eb7OzsgdT/FJv3HeGNrYcYExPBiIhQKmobqTjexJ7KOrYeqAcD4aEOSyY+zMnJITs7m5w38xkWeoAbv7SQ0BDr/iHxlrZ2CGZ9aYPx02t5ofAjmuImk31msm8q1k8Op+H7a1fzxZlj+cN1c/p9fnft0HnLtyrruGPFJ7xZXAuATVq49LQxzBgXQ2JUOAkjwtnyWSEjwp2cNmsOmQE2GGeg3wmfBhz3FceCbnavB74DRHfaHg3U9lLuFcBvgAuNMVVt240xmzoc9oyIXAdcAjzSr4oP0Ia9VSxZtglnh4vIEJuQEBXmrp9rW1OLk43FVZYLOG1yS6qZkzwyKION6rtJiVGMGzmMnMIKrrdowPmk1JWzrLeMAoOVlhjFl04fQ2FZrSv7tIF/7yjjre2HTzl2yZObLPmD0xt8GnCMMdk97Xf34dhFZIoxZrd78ywgv4dzvgg8AVxqjPmstyoAPun1djgNP38zvz3Y2ARuz57E3RelY7NJ+9obje506EXldRhjLNcpX9fUSsGh49x1/hR/V0VZnIiwID2RNz79nOZWpyWzHq8qKCc0RNonq3rTWZMSCA/dQ0urk1C7jRW3nEn6mGiqapt4/KNint9cGnTrSlnqE2GMqQdeBR4UkUgRmQ9cDizv6ngROR/XQIErjTGbO+0bKSKLRSRCROwisgQ4D3jPu+/ClRvpl28XUFReh90mhIiro/L8jP+s5JeZEsuKW7O4Z9FULp45mje2HuKHr2yn1Z3+3Co+La3BaTR/muqb7KmJ1Dc7yC2p7v1gHzPGsKqgnKy0+H5lFRiotu/49xelu65gUuOICreTmhDJlZnjCQ+1Bd26UlbrwwG4A3gKqACOALcbY/IBRCQZV5/MdGNMKXA/EAP8q8OVwVpjzMVAKPAQkAE4gF3AFcYYr8/FeWJtMU9/vJ9bzpnIJaeNYWPxkS7XjslMiSUzJRZjDH9avZs/v7+boydaeOS6OUSEWqMTccv+GmwCc5JH+rsqKgCcPTmB0BDhw6LKAc3e96a9lXXsq6rnm/NTffaabd/xrravuDWr278NQ5XlAo4xphq4opt9pbgGFrQ9X9hDOZXAPE/XrzdvbP2cX/1rF5eePoafXDINm016/TCJCN+7aCpxkWH8/M18vvp/67loehLnTR3l9w9i7v5qpo2J9skvQhX4osLtzEuN48PCSn508TR/V+ckK92TLK2Ss6y7YDSUWeqWWqBbv6eKe17eRlZaHH/42qz222d9ddPZqfzXhVMoOFzLn9/fw5Jl/Zu17GmtTsPWA0c1nY3qlwVTE9lVVsvhYyf8XZWTrC4oZ+a4aMaOHNb7wcorNOB4yD/zDvKNv29hTPQw/nbjXMLtA7slFhpiax/V0LZ2ur8cqHXS0OzQCZ+qX7LTRwHwoYUmgVbWNvHpgaNcNC04UshYlQYcD1i9s5y7X95Gs8NJeW0jeyrqBlxWVlo84aG2k577S1GNawDDXF2SQPXD1KQoxsRE8KGFlit4f2c5xuD14dCqZxpwBsnpNCctwtQ6yNxMbZ2J505OwGkgdrj/+k521zgYHzuM0TERfquDCjwirmHHHxZW8sgAMih7w6qCcsaNHMa0Mb3OIVdepAFnkJ7ZsJ/9VfXtw589McQxMyWW318zC7tNWLGp1EM17R9jDEU1Tu2/UQMyIXY4DS0O/riqyO99kQ3NrazbU8VF05MsN88t2GjA6cKxJtOnL0hhWS2//vcuzs8YxYtLO4y398DIk1EjIlg8czSv5B2kscX3a4yUVjdwvNlo/40akBb3fDKn+c/ERn9Zu7uKplan3k6zAA04XahpMlzXy7oWjS0OvvvCp0RH2Hn4qtPJTI3jzoWTPTrM8YYzUzh2ooW3th3yWJl99XLeQQAiwyw3cl4FgHOmJGJ3j9K0h/h3YuOqgnKiI1zZrJV/acDpRnOrk2Vri7vd//C7hewqq+V/rp5FQpR3lobNSotj8qgo/uHj22p5JTU8tsaV3/S+V7db4h68CiyZKbH89cZMQkQ4P8N/88kcTsMHuypYmDFKcwFagP4f6IZNXMn2fvbGDppbT04382FRJU+t38fNZ6ey0D0E1BtEhCVnJrPtwFF2fN45ibb3bNhb1b5Alb9vh6jAdeG0JL56xjhyCis52tDslzrklfgmWafqGw04XYgNF15YmsXS89J4dkMJ1z+xsX0xpSN1Tdzz8jamJkVx38UDWwu9P756xniGhYbwj40lXn+tNm1ZBYTgyvOkPO/Wc9M40eLw2+CX1TtdyToXTPV+sk7VOw04XYgJF74wMZ4fXzKNR66bQ/6h43zpkXU8t6mEax7fSE19M3++1jf5zmKGhfLlWWN5Y+shjje2eP31APZV1RMaIlw+KTRo0qYr70gfPYJzpyTw9Mf7aWr17eCXvP3VvJx7gOmamskyNOD04rJZY3ntzrOxCfz4tR3tkzobmn335bkhK4UTLQ5edXfke5PTaXh3RxkL00dxxZQwDTZq0G47N43K2ibe2nbqWjDekldSw/XLNlHT0EL+oePaD2kRGnD6IGN0NF+bO6H9uTHGp/0ap42PYdb4GP6xqRRvLwm+/fNjlB1v5IszNQWI8oxzpySQnjSCZWuLvf75bbOx+Eh736vTx99X1T0NOH20IH0UEX5cv2JJVgp7KurYtM+764y8u6MMu024IEM7WZVniAi3nDuRXWW1rN/jmz/8WWnx7Usthmk/pGVowOmjUxZT8vGtpstOH0t0hN2rgweMMby74zBnTYonxo8pddTQc/nssSREhfNED1MNPCktIRIMnJUWp/2QFqIBpx8yU2I9Prmzr4aFhXBV5gT+veMwD7+7yyv3pAvLa9l/pIGLZ47xeNkquIXbQ7jprBQ+LKqkqLzW66+XU1SBAe69eJoGGwvRgBNAZifH4HDCYzl7vZKf6t0dZYhoRl3lHUuyUogItfHk2n1ef633d1aQEBXO6eNivP5aqu8sF3BEJE5EXhORehEpEZHrezj2ZhFxiEhdh0f2QMoKBAeqXQtaGbwzIfPdHWXMS4kjcYR3Mieo4BYXGcaVZ4znn58c9NpVOrjyuH1YVMn5GYn9XgRReZflAg7wKNAMJAFLgMdEZEYPx28wxkR1eOQMoixLy0qLJ9zu/l8m4tGO0P1V9ewqq2Wxjk5TXpSVFker03jtKh1gy/5qahtbLbOUtPoPSwUcEYkErgTuN8bUGWPWAW8CN/qzLKvITInluduyOH18DBhDXGSYx8p+L78MgMUz9EuqvKfUy1fpAB/srCAsxMY5kxM8XrYaHKulAp4KOIwxRR22bQMW9HDOHBGpAqqB5cCvjTGt/S1LRJYCSwESExPJyckZ8Jvwtm9OcXLvYfjB8rXcNcczi6O9uOEEqdE29mzbzB73trq6Oku3g69oO3iuDcKPOrALtBrXqOXwoyXk5Hh2QvNbnzQwNdbGlg3rPFou6GehzUDbwWoBJwronKXyGNDdMn0fATOBEmAG8CLQCvy6v2UZYx4HHgdIT0832dnZ/a+9D5Xad/OHVUVEpZ7O3EEuknb42AmK3/2AHyxOJzt7cvv2nJwcrN4OvqDt4Lk2yAZmzanmG3/fwpSkKG79yvxBl9lRcWUd5e9+yJ0XZZB9VqpHywb9LLQZaDv49JaaiOSIiOnmsQ6oA6I7nRYNdDmO0hhTbIzZZ4xxGmM+Ax4ErnLv7ldZgebWcyeSFB3OQ+/sHPTs7ZX55QCaXUD5xLzUOL4xP5VtB45y6OgJj5b9/s4KAM7P8F4WdzVwPg04xphsY4x08zgHKALsIjKlw2mzgPy+vgTt84sHXZalDQ+zc/eidLYeOMo7nw0uR9W7O8qYMiqKSYlRHqqdUj27OnMCTgOveDg/4Pu7yskYPYLxscM9Wq7yDEsNGjDG1AOvAg+KSKSIzAcux9U3cwoRuVhEktz/zgDuB94YSFmB6MozxpMxegS/fXfXgDPxVtc3s2nfEb26UT6VHD+c+ZPjeSn3AE6nZ/KrHTvRwpb9NXp1Y2GWCjhudwDDgArgeeB2Y0w+gIgku+faJLuPvQDYLiL1wL9wBZhf9aWsoSDEJvzk0mkcqD7B8g0DS3mzqqAMp4HFMzTgKN+6Zl4yB2tO8PFez4xU+7CoEofTcME0DThWZbVBAxhjqoErutlXimswQNvze4B7BlLWUHHulEQWTE3kL+/v5qrM8Ywc3r+h0i9uOUDMMDtNLb5dq0SpRdOTiBkWygtbSjlnyuCHMH+ws5y4yDBmT9BUNlZlxSsc1U8/vmQatY2t3PpMbr8m0r259XM+KT3K8ROtLHlyk64ZonwqIjSEr8wZx8r8cmrqB7cEdavDyZrCSrLTEwnR7AKWpQFnCKhrasUmQm5JDV/72wZWuUeddaeytomfvv4Z331xK+DdSXhK9eSaeRNodjh57dPPB1XOJ6VHOXaiRZfVsDgNOEPAxuIjGFwdrw6n4Vv/yOX7L26l4NDxk4470ezgkfd3k/0/a3h+8wEWTx9NuN1/a/woNW1MNLPGx/DilgODGt7//q5y7DbhvKmaXcDKLNeHo/ovKy2eMLuNllYn9hAbF04bxbv5Zbz66efMnxxPdvoo8vbXsGnfEWoaWlg8I4kffjGDSYlR5JXUsLH4CFlp8ZrGXfnFNfOS+fFrn7Ht4DFmTxg5oDLe31nBmWlxjIjQdZysTAPOENC2OFzHwHHsRAvPby7l8Q+L21dZFIGHrpjJDVkpJ52rgUb502WzxvDLtwt4cUvpgAJOyZF69lTUcf0Xkns/WPmV3lIbIjovDhczLJRvL5jEzfNT22fC2nDNVVDKSkZEhHLp6WN4c+sh6pta+33+0x/vByApWpfVsDoNOEPc/MkJhIdqP42ytmvmTaC+2dHvrBl5JTU84w44d7+8TUdaWpzeUhviurrdppTVzE2JJS0xkifX7aOytqnPn9XlG/bTlqigbaSlfsatSwNOENB+GmV1IsL8SQks31jC78sLCbPbWHFrVo+f2x2fH+Nfn7mWRbehV/CBQAOOUsoSRkS4/hw5DTT3crVScbyR257NJSEqjAevmElhWa1ewQcADThKKUu4YFoSy9bto7nVidNAYlTXaZoaWxzctjyPow0tvHL7WcwYG8OFupx0QNBBA0opS8hMieX527K45ZyJJI4I44G3ClhTWHHSMcYYfvDKdrYfPMqfrp3NjLExfqqtGggNOEopy8hMieX+L03nne+cS2p8JLc+k8vLuQfa9z/ywR7e2naIHy7O0AznAUhvqSmlLGdUdAQvfiuL2//xCT94ZTufHjhKXWMrb247xFfPGMe3F6T5u4pqADTgKKUsaUREKE/dPI9bntnCc5tKAVe2jK/NHY+IZoQORHpLTSllWWF2G2dOjDspW0ZeyVE/1kgNhgYcpZSlnTVJs2UMFZYLOCISJyKviUi9iJSIyPU9HPtX95LTbY8mEantsD9HRBo77C/0zbtQSnlKW7aM7y9K73UyqLI2K/bhPAo0A0nAbOAdEdlmjMnvfKAx5tvAt9uei8jTgLPTYXcZY5Z5rbZKKa/TbBlDg6WucEQkErgSuN8YU2eMWQe8CdzYj3Of8W4tlVJKDYSlAg4wFXAYY4o6bNsGzOjDuVcClcBHnbb/WkSqRGS9iGR7pJZKKaX6zWq31KKAY522HQNG9OHcm4Bnzcnr1N4LFOC6RXct8JaIzDbG7O18sogsBZYCJCYmkpOT0//aDzF1dXXaDmg7gLZBG20HlwG3gzHGZw8gBzDdPNYBc4CGTufcDbzVS7kTgFYgrZfj3gW+01s9p06dapQxa9as8XcVLEHbQdugjbaDS3ftAOSaHv62+vQKxxiT3dN+dz+MXUSmGGN2uzfPAk4ZMNDJ14GPjTHFvVUB0BljSinlB5bqwzHG1AOvAg+KSKSIzAcuB5b3curXgac7bhCRkSKyWEQiRMQuIkuA84D3vFB1pZRSvbBUwHG7AxgGVADPA7cb95BoEUl2z6dJbjtYRM4CxgMvdyonFHgI10CCKuA7wBXGGJ2Lo5RSfmC1QQMYY6qBK7rZV4prYEHHbRuAyC6OrQTmeaGKSimlBsCKVzhKKaWGIA04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfMJyAUdE7hKRXBFpEpGn+3D890SkTESOichTIhLeYV+ciLwmIvUiUiIi13u18koppbpluYADHAIeAp7q7UARWQzcB1wApAJpwC86HPIo0AwkAUuAx0Rkhofrq5RSqg8sF3CMMa8aY14HjvTh8JuAJ40x+caYGuCXwM0AIhIJXAncb4ypM8asA94EbvRKxZVSSvXI7u8KDNIM4I0Oz7cBSSISDyQDDmNMUaf9C7oqSESWAkvdT5tEZIcX6htoEoAqf1fCArQdtA3aaDu4dNcOKT2dFOgBJwo41uF5279HdLGvbf+IrgoyxjwOPA4gIrnGmLmerWrg0XZw0XbQNmij7eAy0Hbw6S01EckREdPNY90AiqwDojs8b/t3bRf72vbXDuB1lFJKDZJPA44xJtsYI908zhlAkfnArA7PZwHlxpgjQBFgF5EpnfbnD/wdKKWUGijLDRoQEbuIRAAhQIiIRIhId7f+ngVuEZHpIhIL/BR4GsAYUw+8CjwoIpEiMh+4HFjeh2o8Ptj3MURoO7hoO2gbtNF2cBlQO4gxxtMVGRQReQD4eafNvzDGPCAiyUABMN0YU+o+/vvAvcAw4J/At40xTe59cbiGV1+Ea9TbfcaY53zyRpRSSp3EcgFHKaXU0GS5W2pKKaWGJg04SimlfEIDTgfBmnutp/x1InKBiOwSkQYRWSMiPU7sClQiEi4iT7r/v9eKyKcicnGH/UHRDgAi8g8ROSwix0WkSERu7bAvaNoBQESmiEijiPyjw7agaQP3VJZGEalzPwo77Ot3O2jAOVmw5l7rMn+diCTgGul3PxAH5AIv+rx2vmEHDuDKRBGD6z2/JCKpQdYOAL8GUo0x0cCXgYdEJDMI2wFcfxO2tD0J0ja4yxgT5X6kw8DbQQcNuLlzr9UAM9vS4YjIcuBzY8x9fq2cj4jIQ8B4Y8zN7udLgZuNMWe7n0fiSmcxxxizy28V9RER2Y4rGWw8QdoOIpIO5ADfBUYSRO0gItcCX8U1MnayMeaGYPtOiEgO8A9jzLJO2wfUDnqF8x9T6Tr3WjBc4XRnBq42ANrnNu0lCNpERJJwfSbyCcJ2EJH/E5EGYBdwGPgXQdQOIhINPAjc3WlX0LRBB78WkSoRWS8i2e5tA2oHDTj/0a/ca0EiKNtEREKBFcAz7l9rQdcOxpg7cL2/c3HdOmkiuNrhl7gy0R/otD2Y2gBccxzTgHG4Jnu+JSKTGGA7aMD5D829dqqgaxMRseHKRtEM3OXeHHTtAGCMcbiX9RgP3E6QtIOIzAYuBP7Yxe6gaIM2xphNxphaY0yTMeYZYD1wCQNsBw04/6G51051Uq46933aSQzRNhERAZ7ENWjkSmNMi3tXULVDF+z85/0GQztk41rQsVREyoB7gCtF5BOCpw26YwBhoO1gjNGH+wG8ADwPRALzcV0izvB3vXzwvu1ABK7RScvd/7YDie42uNK97bfARn/X14vt8FdgIxDVaXvQtAMwCrgW1y2TEGAxUI8rD2FQtAMwHBjd4fE74BX3+w+KNnC3w0j3//+2vwdL3J+F9IG2g9/flJUeuIb3ve5u1FLgen/XyUfv+wH3L5eOjwfc+y7E1XF8AtdopVR/19dLbZDift+NuG4XtD2WBFk7JAIfAkeB48BnwG0d9gdFO3RqkwdwjdQKqjZwfxa24LpNdtT9Y+yiwbSDDotWSinlE9qHo5RSyic04CillPIJDThKKaV8QgOOUkopn9CAo5RSyic04CillPIJDThKKaV8QgOOUgFCRKJF5AERmebvuig1EBpwlAocc4GfA6H+rohSA6EBR6nAMQfXMgEF/q6IUgOhqW2UCgAishPI6LT5n8aYq/xRH6UGQgOOUgFARObhymaeD/zKvfmwMabEf7VSqn/s/q6AUqpPtuFaCO0RY8xGf1dGqYHQPhylAsMMIAz4xN8VUWqgNOAoFRjOwLVez1Y/10OpAdOAo1RgmAPsNcYc93dFlBooDThKBYbp6HBoFeB00IBSgeEocIaILMa1lvxuY8wR/1ZJqf7RKxylAsPPgHLgdWADoOltVMDReThKKaV8Qq9wlFJK+YQGHKWUUj6hAUcppZRPaMBRSinlExpwlFJK+YQGHKWUUj6hAUcppZRPaMBRSinlE/8fVg04iTfL6W4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = model.predict(X_valid)\n",
    "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ABkCAvHeFpaa"
   },
   "source": [
    "### Simple RNN\n",
    "- It uses the tanh for implementation of RNNs.\n",
    "- For each recurrent neuron, there is one parameter per input and one per hidden state dimension plus a bias."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "enea04fkFpab",
    "outputId": "dd777a8e-fec5-4ef3-98d3-44ec7102b9e7"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\keras\\optimizer_v2\\optimizer_v2.py:355: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "219/219 [==============================] - 2s 6ms/step - loss: 0.0967 - val_loss: 0.0489\n",
      "Epoch 2/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0369 - val_loss: 0.0296\n",
      "Epoch 3/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0253 - val_loss: 0.0218\n",
      "Epoch 4/20\n",
      "219/219 [==============================] - ETA: 0s - loss: 0.019 - 1s 5ms/step - loss: 0.0198 - val_loss: 0.0177\n",
      "Epoch 5/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0166 - val_loss: 0.0151\n",
      "Epoch 6/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0146 - val_loss: 0.0134\n",
      "Epoch 7/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0132 - val_loss: 0.0123\n",
      "Epoch 8/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0124 - val_loss: 0.0116\n",
      "Epoch 9/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0118 - val_loss: 0.0112\n",
      "Epoch 10/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0116 - val_loss: 0.0110\n",
      "Epoch 11/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 12/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 13/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 14/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 15/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 16/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 17/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 18/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 19/20\n",
      "219/219 [==============================] - 1s 5ms/step - loss: 0.0114 - val_loss: 0.0109\n",
      "Epoch 20/20\n",
      "219/219 [==============================] - 1s 6ms/step - loss: 0.0114 - val_loss: 0.0109\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(1, input_shape=[None, 1])\n",
    "])\n",
    "\n",
    "optimizer = keras.optimizers.Adam(lr=0.005)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer)\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "W9MYV17tFpab",
    "outputId": "a1a928e9-433c-4ffd-d1e0-a66ca47b9ec9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "63/63 [==============================] - 0s 2ms/step - loss: 0.0109\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.010881561785936356"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ym9UcP2pFpac",
    "outputId": "57732bed-2d3b-46d8-ce5b-e4f0675cb6db"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA9sklEQVR4nO3dd3hUZfr/8fedQhokJJQgTYoB6R2MSBMBy6KugKugwgpi+aGiooiNKjZsqwiIFBVEdIHVXf2KiERFUIqCggiCFAHpCAmEtLl/f5xJCHGAJEyLuV/XNdfMnPPMOZ9JJnPnOeU5oqoYY4wx3hYS6ADGGGP+mqzAGGOM8QkrMMYYY3zCCowxxhifsAJjjDHGJ6zAGGOM8QkrMMYYY3zCrwVGRBJEZIGIHBOR7SLS9wxt7xORPSJyRESmi0hEvnkpInJCRNLct43+eQfGGGMKy989mIlAJpAI9AMmiUijgo1EpAfwMNAVqAXUAUYXaDZEVcu6b/V9mtoYY0yR+a3AiEgM0At4XFXTVHUp8CFws4fm/YFpqrpeVQ8DY4EB/spqjDHm3IX5cV31gBxV3ZRv2lqgk4e2jYAPCrRLFJEKqnrQPe0pEXka2Ag8qqopnlYqIoOBwQBRUVGtatSokTfvl1/KUb58JpUqZRT6TbhcLkJCgmfXVTDlsSyeBVMWCK48lsWzYMoCsGnTpgOqWqnIL1RVv9yADsCeAtNuA1I8tN0CXJ7veTigQC3383ZAOSACp7eTCtQ9W4ZWrVppflWrqt56qxbJkiVLivYCHwumPJbFs2DKohpceSyLZ8GURVUVWKXF+N73Z4lMA2ILTIt1F4eztc19nAqgqt+qaqqqZqjqm8DXwJVFDZSQAIcPF/VVxhhjCsOfBWYTECYiSfmmNQPWe2i73j0vf7u9enLzWEEKyFkT7NkDy5fnPY2Ph0OHzvoqY4wxxeC3AqOqx4D5wBgRiRGR9sA1wNsemr8FDBSRhiISDzwGzAQQkfIi0kNEIkUkTET6AR2BhWcNsWsXdO2aV2Ti460HY4wxvuLvvUh3AVHAPmAOcKeqrheRmu7zWWoCqOonwLPAEmC7+zbSvYxwYBywHzgA3A1cq6qFOxcmMxNSUgDbRGaMMb7kz6PIUNVDwLUepu8AyhaY9gLwgoe2+4E2xQ5Rpgx07gzYJjJjjPElvxaYoPD++5CcDDgF5tgxyMqC8PAA5zKmEI4ePcq+ffvIysoq9jLi4uLYsGGDF1MVn2XxzJ9ZwsPDqVy5MrGxBY/BOnelr8Dku0R0QoJzf/gwVK4coDzGFNLRo0fZu3cv1apVIyoqCpGzH9fiSWpqKuXKlfNyuuKxLJ75K4uqkp6ezq5duwC8XmSC50wef1mxIu9hfLxzb5vJTEmwb98+qlWrRnR0dLGLizH5iQjR0dFUq1aNffv2eX35pavAREXBypV5T3MLjO3oNyVBVlYWUVFRgY5h/oKioqLOabPr6ZSuAhMT4/Rg3JvJcjeRWQ/GlBTWczG+4KvPVekqMNHRTjX59VfAejDGGONLpavAxMQ49+7NZPl38htjjPGu0lVgoqIgMjJvR3/58s5k20RmTMkyYMAA/va3vxXpNZ07d2bIkCE+SnTSqFGjaNy4sc/XUxKUrsOURaBly7wCExYG5cpZD8YYXznbtv2+ffsye/bsIi/35Zdfzh1pvdDmz59PuJ3w5lelq8AAtG0LU6ZAdjaEhdlwMcb40O+//573+H//+x+33XbbKdOys7NPaZ+VlVWoIhAXF1fkLAm528SN35SuTWQAbdpAejqsdwZxtuFiTGm0fDk89dQpg4v7RJUqVfJu5d3bpHOfnzhxgho1ajBnzhwuvfRSoqKimDJlCgcPHuTGG2+kevXqREVF0ahRI2bMmHHKcgtuIuvcuTN33XUXjzzyCBUrVqRy5coMGzYMl8t1Spv8m8hq1arFuHHjuP3224mNjeXCCy/kueeeO2U9mzZtolOnTkRGRlK/fn0+/vhjypYty8yZMwv9M3C5XIwdO5YaNWoQERFBkyZN+OCDD05pM2bMGM4//3wiIiKoUqUKgwcPzpv35ZdfctFFF1G2bFni4uJo164d69atK/T6A6l09mDA2UzWrJmNqGxKtKFDYc2aor3m8OEo1q0DlwtCQqBpUyhKh6B5c3jppaKt80xGjBjBhAkTmDZtGuHh4Zw4cYKWLVsyfPhwYmNj+eyzz7j99tupWbMmXbt2Pe1yZs+ezb333suyZctYs2YNffv2pVWrVtx4442nfc2LL77I6NGjefDBB1mwYAEPPfQQl1xyCcnJybhcLv7+979TpUoVvvnmG9LT0xk6dCgZGYW/Ai44m/Oee+45Jk+eTOvWrZk1axbXXXcdq1evpnnz5sybN48JEyYwZ84cmjRpwr59+0hxD8ibnZ3NNddcw8CBA5k9ezZZWVl89913hIaGFilDoJS+AlO3rtNtWbECbruNhAT46adAhzLGf44cEXL/sXe54MiRohUYb7v77rvp3bv3KdMefPDBvMeDBw/m888/Z86cOWcsMA0bNmTMmDEA1KtXj6lTp7J48eIzFpju3bvn9WruuOMOXn/9dRYvXkxycjKLFi1i48aNfPrpp1SrVg1wClL79u2L9P4mTJjAsGHD6Nu3L+D0Vr788ksmTJjArFmz2L59O+eddx7du3cnPDycmjVrUr9+fcAZHuiPP/6gZ8+e1K1bF4ALL7ywSOsPpNJXYESczWTuHf3WgzElWXF6Ep99doKrr44hM9MZXHz27LzxXwOidevWpzzPycnh6aefZu7cuezatYuMjAwyMzPp7B4F/XSaNm16yvOqVauedfiTM73m559/pmrVqnnFBaBNmzaEhBR+z8LRo0fZvXv3n4rSJZdcwscffwxAnz59ePnll6lduzY9evTg8ssvp0uXLpQrV46EhAQGDBhAjx496Nq1K127dqVPnz7UqFGj0BkCqfTtgwFnM9n69XDsWF6BKeIBKcaUWO3auVi8GMaOhcWLA1tcAGJyz09zmzBhAs8//zwPPvggixcvZs2aNVx77bVkZmaecTkFDw4QkVP2wRT1NarqtTPcPS0nd1qNGjXYuHEjU6ZMITY2lgceeICOHTty7NgxAGbMmMG3335Lx44d+fDDD6lXrx4LF579+orBoPQWmJwc+P57EhIgI8PZ729MaZGcDCNGBL64eLJ06VJ69uzJzTffTPPmzalbty6bNm3ye44GDRqwa9cudu/enTdt1apVZy1a+cXGxlK1alWWLl16yvSlS5fSsGHDvOeRkZFcddVVvPjii6xcuZINGzbw9ddf581v1qwZw4cPJyUlhc6dO/Pmm2+ewzvzn9K3iQycTWQAK1YQH38J4PRioqMDmMkYAzj7T+bOncvSpUupWLEir7zyClu3bqVFixZ+zdGtWzfq169P//79mTBhAunp6dx///2EhYUVqWfz4IMP8sQTT5CUlESrVq2YNWsWX331FatXrwZg5syZZGdn065dO8qWLcvcuXMJDw8nKSmJrVu3MmXKFK6++mqqVavGr7/+yg8//MCdd97pq7ftVaWzwFSpAjVqwIoVJPRyJh0+DPk2tRpjAuSxxx5j69atXHHFFURFRTFgwAD69evHT34+GickJIQFCxYwaNAg2rZtS61atXj++ee57rrriIyMLPRy7rnnHlJTU3nooYfYu3cv9evXZ968eTRv3hyA8uXL88wzzzBs2DCysrJo2LAhs2bNonbt2uzdu5dNmzbRp08fDhw4QGJiIv369WP48OE+etdepqql5taqVSvN06uXap06umiRKqh+8YUWypIlSwrX0E+CKY9l8cxbWX766SevLOfo0aNeWY43lLQsa9asUUBXrVoV8CzedqbPF7BKi/GdWzp7MODsh5k3j4pyAKhoR5IZY/5kwYIFxMTEkJSUxLZt27j//vtp1qwZLVu2DHS0EqF07uSHvBMuq+xwRla2AmOMKSg1NZUhQ4bQsGFD+vXrR4MGDVi4cKFdl6eQSm8PplUrECHul5XAFTZcjDHmT2655RZuueWWQMcosUpvD6ZcOWjQgMi1KxCxHowxxnhb6S0wAG3bIitXEF9erQdjjDFeVroLTJs2sH8/jctttx6MMcZ4WekuMO4d/clhK63AGGOMl5XuAtO0KZQpQ6ucFbaJzBhjvKx0F5gyZaBFCxodX2E9GGOM8bLSXWAA2rShzuHVHDmUE+gkxpjTGDVqFI0bNz7tc0+GDBly1iH+i7NuXyl4lc6/AiswbdsSmX2MKoc32JD9xnhZz549ueyyyzzO27BhA7GxsSxatKjIyx02bBhffPHFucY7xfbt2xERVq1a5fN1lRZWYNw7+lu5VpCaGuAsxvzFDBo0iM8//5xt27b9ad60adPOehnk0ylbtiwVKlTwQsLgWtdfjRWYpCQyouJoi+3oN6XI8uXw1FPOvQ9dddVVJCYmMmPGjFOmZ2Vl8fbbb3PTTTehqgwcOJDatWsTFRVFUlISzz777Bmvu1Jws1VOTg7Dhg0jPj6e+Ph4hg4dSk7OqZu9P/nkEzp06EB8fDwJCQn06NGDDRs25M1v0qQJ4Fy1UkTyNq8VXJfL5WLs2LHUqFGDiIgImjRpwgcffJA3f9u2bYgI8+bNo1u3bkRHR9OwYcMi99QyMjIYOnQoiYmJREZGctFFF51yXZmsrCzuueceqlatSkREBDVq1ODhhx/Omz9//nyaNm1KVFQUCQkJdOrUib179xYpw7kqvUPF5AoJ4UhSa9r84ByqXKtWoAMZUwRDh8KaNUV6SdThw7BuHbhcEBLiHE0ZF1f4BTRvXuhrNYeFhdG/f39mzpzJyJEj8y43/N///pcDBw5w00034XK5qFatGu+99x6VKlVixYoVDB48mAoVKjBw4MBCref5559n6tSpTJ06laZNmzJx4kRmz559yqCUx44dY+jQoTRt2pT09HTGjRtHz549+emnnyhTpgxLliyhS5cufPLJJzRr1owyZcp4XNfLL7/Mc889x+TJk2ndujWzZs3iuuuuY/Xq1XlD8AM8+uijPPfcc7z22muMGzeOG264ge3bt1O2bNlCvaeHHnqI9957j+nTp1OnTh1eeOEFLr/8cn755RfOO+88/vWvf7FgwQLeffddatWqxc6dO9m4cSMAe/bs4YYbbuCpp56iV69epKWl8c033xRqvd5kPRggvXFbmvIDR/bYZS3NX58cOeIUF3Dujxzx6foGDhzIjh07+Oyzz/KmTZs2je7du1O9enXCw8MZM2YMbdq0oVatWlx//fXccccdzJkzp9DreOmll3jooYe4/vrrufDCC3n55ZepUqXKKW169epFr169SEpKomnTpsyYMYOtW7eyYsUKACpWrAhAhQoVqFKlCgkJCR7XNWHCBIYNG0bfvn2pV68eY8aMoUOHDkyYMOGUdvfddx89e/YkKSmJ8ePHc+jQIdYU8p+BY8eOMWnSJJ555hmuuuoqGjRowOTJk0lMTGTixImAs8+oXr16dOjQgZo1a3LxxRfzz3/+E4Ddu3eTlZVF7969qVWrFo0bN2bQoEEkJiYW+mfqDX7twYhIAjAN6A4cAEao6junaXsfMByIAuYBd6pqRoE2ScCPwL9V9abi5tI2bQl/JxvXd2vgiiC8hqwxp1PInkR+Jz77jJirr4bMTOdQ/dmzfXrt5KSkJDp27Mj06dPp3r07u3fvZuHChcydOzevzeTJk3njjTfYvn076enpZGVlcf755xdq+UeOHOH3338nOd97CAkJoV27dvz2229507Zs2cLjjz/Ot99+y/79+3G5XLhcLnbs2FHo93L06FF2795N+/btT5l+ySWX8PHHH58yrWnTpnmPq1atCsC+ffsKtZ6tW7eSlZV1ynpCQ0NJTk7Ou/DagAED6NatG/Xq1aN79+5ceeWVXHHFFYSEhNCsWTMuu+wyGjduTPfu3bnsssvo3bs3lSpVKvR79QZ/92AmAplAItAPmCQijQo2EpEewMNAV6AWUAcYfZrlrTzXUOHtnR39kT+uONdFGRP0XO3aweLFMHasc+/D4pJr0KBB/Oc//+HQoUPMnDmThIQErr76agDmzp3L0KFDGTBgAAsXLmTNmjXcddddZGZmejVDz5492b9/P1OmTOHbb7/l+++/JywsrFjr8TRcf8Fp4eHhf5p3pv1K+an7kNYzradly5Zs27aN8ePH43K56N+/P926dcPlchEaGsqnn37Kp59+StOmTZk2bRpJSUmsXbu2cG/QS/xWYEQkBugFPK6qaaq6FPgQuNlD8/7ANFVdr6qHgbHAgALLuwH4A1h8rtniGlRlF1WJ23TOtcqYkiE5GUaM8EtxAejduzeRkZHMmjWL6dOnc8stt+R9AS9dupR27doxZMgQWrZsyQUXXMCWLVsKvey4uDjOO++8U/YxqGrepi+AgwcPsmHDBh555BEuu+wyGjRoQGpqKtnZ2Xltcve5FDw4IL/Y2FiqVq16ys723PfQsGHDQmc+mzp16lCmTJlT1pOTk8Py5ctPWU+5cuXo06cPkyZN4qOPPuLzzz9n8+bNgFOIkpOTGTlyJCtXrqRq1aqn9Br9wZ+byOoBOaq6Kd+0tUAnD20bAR8UaJcoIhVU9aCIxAJjcHo4Z9wLKCKDgcEAiYmJpKSk/KmNKhyRNrTbvMzj/PzS0tLO2safgimPZfHMW1ni4uJI9cKx9Dk5OV5ZTlH17t2bkSNH8scff3DDDTeQmppKTk4ONWvWZObMmcybN486deowb948vvjiC8qXL5+XMyMjA5fLddrnd9xxB88++yw1atSgUaNGTJ06ld9//53KlSuTmppKWFgYFSpU4LXXXiM+Pp7ff/+dxx57jLCwME6cOEFqaioJCQlERUXx4YcfUrFiRSIiIoiLi/vTuu6++27Gjx9P9erVad68OXPnzuWrr77iyy+/JDU1lbS0NMDZj1Lw55yenn7an31WVhbZ2dmkpqYSGRnJwIEDGT58ONHR0dSqVYuJEyeyd+9ebrnlFlJTU3n11VdJTEykadOmhIWFMXPmTGJjY4mLi2Px4sWkpKTQtWtXKleuzA8//MBvv/1G7dq1T7v+EydOeP9vpjjXWS7ODegA7Ckw7TYgxUPbLcDl+Z6HAwrUcj9/GRjufjwKmFWYDK1atTrtNafHxYxXBdWDB0/bRjW4rvWuGlx5LItn3spypmumF0Ugrveuqrp69WoF9OKLLz4lS0ZGht56661avnx5jYuL01tvvVVHjx6t559/fl67kSNHaqNGjU77PCsrS4cOHapxcXEaFxenQ4YM0TvuuEM7deqU12bx4sXaqFEjjYiI0EaNGuknn3yiMTExOmPGjLwsU6dO1Ro1amhISEjeawuuKycnR8eMGaPVq1fX8PBwbdy4sS5YsCBv/tatWxXQlStXnvL+AX3//fdP+/Pp37+/XnXVVXlZTpw4offee69WrlxZy5Qpo+3atdOvvvoqr/3rr7+uLVq00LJly2q5cuW0Y8eO+vXXX6uq81m5/PLL815bt25dfeaZZ0677tzXnA6wSovzvV+cFxVrRdACOF5g2gPAfz20XQtcn+95BXeBqQA0B9YDZdSLBWZA9UXOj2PhwtP/BjS4vrhUgyuPZfHMCszpWRbPApHFFwXGnzv5NwFh7iO/cjVzF4uC1rvn5W+3V1UPAp1xdvzvEJE9wDCgl4h8dy7hdp3X2nmw0vbDGGOMN/itwKjqMWA+MEZEYkSkPXAN8LaH5m8BA0WkoYjEA48BM93zXgfq4vRkmgOTgY+AHueSL7xSebZF1IcVdiSZMcZ4g78PU74L57yWfcAcnHNb1otITRFJE5GaAKr6CfAssATY7r6NdM87rqp7cm9AGnBCVfefS7CEBPgurC18+y026qUxxpw7v55oqaqHgGs9TN8BlC0w7QXghUIsc5Q3ssXHwzc5bbhu79uwcyfUqOGNxRpjTKllQ8W4xcfDFyecEy5tP4wJVmq9a+MDvvpcWYFxS0iAtTRDw8NtP4wJSuHh4aSn23h5xvvS09NPGXnAW6zAuMXHQwaRZFzYzAqMCUqVK1dm165dHD9+3HoyxitUlePHj7Nr1y4qV67s9eXbcP1u8fHO/dF6bYj8dNbJocyNCRKxsbHAyZFyi+vEiRNERkZ6K9Y5sSye+TNLeHg4iYmJeZ8vb7IC45Y7MveHe9oyKHUSbNwIDRoENpQxBcTGxp7zF0FKSgotWrTwUqJzY1k8C6Ys58L+RXfbvt25f2mZs6N/8zu2mcwYY86FFRi3Te4hODdofVIpy5FFVmCMMeZcWIFx69HD2eXiIpTvQlpT/6gdqmyMMefCCoxbcjKMHOk8LndpW8puXgMZGWd8jTHGmNOzApPPAw9AdDSsoC1kZYGfr/5mjDF/JVZg8omJgWuugYkr2zgT7HwYY4wpNiswBfTtC+uO1OBE+UQbMsYYY86BFZgCuneH+HhhfUxb68EYY8w5sAJTQJky0Ls3fLSvLfz8Mxw5EuhIxhhTIlmB8eDGG+HrLPd+mFWrAhvGGGNKKCswHnTsCDuruAuM7YcxxphisQLjQWgo9LgxgV+4gMyvbT+MMcYUhxWY07jxRlhJG7KswBhjTLFYgTmN1q3h1wptiTm8C3bvDnQcY4wpcazAnIYIJPZ0RlY+tND2wxhjTFFZgTmDDve0IJtQNs+xzWTGGFNUVmDO4MIWUWyObALfWoExxpiisgJzFieatiXp6Cp+3ewKdBRjjClRrMCcRY1ebYnnDxZO3BzoKMYYU6JYgTmLClc4O/q3v2+byYwxpiiswJxNgwZklYmm6q4V/PhjoMMYY0zJYQXmbMLC0JataMtK5swJdBhjjCk5rMAUQpn2bWkZ8j3/ficT1UCnMcaYksEKTGG0bUsZVwbltv/ITz/FBjqNMcaUCFZgCqONM7LyxWErWLy4coDDGGNMyWAFpjBq1YKKFbm22kpSUiqTnR3oQMYYE/yswBSGCLRtS2vXCg4fLkNKSqADGWNM8LMCU1ht2hC78ycqRx3mnXcCHcYYY4KfXwuMiCSIyAIROSYi20Wk7xna3icie0TkiIhMF5GIfPNmicjvInJURDaJyCCfh2/bFlHl5kZLmD8fMjJ8vkZjjCnR/N2DmQhkAolAP2CSiDQq2EhEegAPA12BWkAdYHS+Jk8BtVQ1FrgaGCcirXya3L2j/2+Vl3LkCPzf//l0bcYYU+L5rcCISAzQC3hcVdNUdSnwIXCzh+b9gWmqul5VDwNjgQG5M93Tc/sQ6r7V9WV+KlWC2rVpkPodFStiJ10aY8xZiPrpzEERaQEsU9WofNOGAZ1UtWeBtmuB8ao61/28IrAfqKiqB93TXsMpOlHA90BHVU3zsN7BwGCAxMTEVu+++26x30PT+++n7KZNPN7yHV5ecRXz5y8jOjqn2MvzhrS0NMqWLRvQDLksi2fBlAWCK49l8SyYsgB06dJltaq2LvILVdUvN6ADsKfAtNuAFA9ttwCX53sejtNLqVWgXShwCfAYEH62DK1atdJiW7ZMNTxcFTS7TKRexDJ9++3iL85blixZEugIeSyLZ8GURTW48lgWz4Ipi6oqsEqL8b3vz30waUDB0+BjgdRCtM19fEpbVc1RZ1NbdeBOL+X0LCUFcpzeSkhWBtfGpdhmMmOMOQN/FphNQJiIJOWb1gxY76Hteve8/O32qnvzmAdh+HofTOfOEBGBAqJKhcua8+mncPB0iYwxppTzW4FR1WPAfGCMiMSISHvgGuBtD83fAgaKSEMRicfZBDYTQEQqi8gNIlJWRELdR5zdCHzu0zeQnAyLF7OzVy8Aro75nOxs+Pe/fbpWY4wpsfx9mPJdODvl9wFzgDtVdb2I1BSRNBGpCaCqnwDPAkuA7e7bSPcyFGdz2E7gMDABGKqqH/g8fXIyW4YMgZtvptJ7r9Kh7m476dIYY04jzJ8rU9VDwLUepu8AyhaY9gLwgoe2+4FOPopYOKNGIXPm8HzCk7T7aiI7d0L16gFNZIwxQceGiimOOnVg0CBaf/865+tW5s4NdCBjjAk+VmCK67HHkLAwXqkw2o4mM8YYD6zAFFe1anDXXVx56G2Ord7Apk2BDmSMMcHlnAuMiIR7I0iJ9PDDEB3NaEZaL8YYYwooUoERkXtEpFe+59OAdBHZKCL1vZ4u2FWqRMj993E977Nmxvf4adQdY4wpEYrag7kHZ0wwRKQjcD3QF1gDPO/VZCXFAw9wIjqegdsfZ82aQIcxxpjgUdQCUw3Y5n7cE3hfVd8DRgEXeS9WCRIXR84DD/E3PmLZhGWBTmOMMUGjqAXmKFDJ/bgbsNj9OAuI9FaokiZm+N0cjkikxbxHceXYdjJjjIGiF5hPganufS8XALmX3WoEbPVmsBIlJoYt1z/CxRkprP/X4rO3N8aYUqCoBeb/AV8DFYHe7jPzAVriDP1Sal344u38JjUo+/Sj2N5+Y4wpYoFR1aOqereqXuMeLyx3+khVHe/9eCVH2QoRfNx6JLX3rSB7wX8DHccYYwKuqIcpN8x/OLKIdBORWSIyQkRCvR+vZDnv4f5sIon0Bx4DlyvQcYwxJqCKuolsGtACQESqAx8ACTibzsZ5N1rJ0+OqMJ6KGE25bT+yaawNUGaMKd2KWmAaAN+5H/cBvlXVK4Gbca7JUqp99x3MyvoHP9AEGT2S5V9lBzqSMcYETFELTCiQ6X7cFfjY/XgLkOitUCVVSgq4COFxxpKkv3DwhTcDHckYYwKmqAVmHXCniHTAKTC5O/qrAQe8Gawkcl9Vmf/J1XxLWy5aNAYyMgIdyxhjAqKoBWY4cBuQAsxR1R/d068GVngxV4nkvqoy454UPmz3JBWP7WDTsNcDHcsYYwKiSFe0VNUvRaQSEKuqh/PNmgIc92qyEio52bkdu7sr3yZ2pvZrT7Lv/lupXDsm0NGMMcavijxcv6rm4Iyg3FhEGolIpKpuU9V9PshXYsWUFSpMepLKrr387/JX7KhlY0ypU9TzYMJE5DngMLAW+BE4LCLPlurrwpzGBbdczPZGV3LtpmeZ+OQfgY5jjDF+VdQezLPATcAdQD0gCbgT5zDlp7wb7a+h5lvjSOAwR0a+wIpSv5fKGFOaFLXA9AUGquqbqrrFfZsJDAL6eT3dX4C0bEHmNX0Yyovc1Wc/R44EOpExxvhHUQtMHM45LwVtAcqfc5q/qDJPjyFGjtP3t2e47TYbC9MYUzoUtcCsxbmqZUH3uucZTy68ELn5Zu4OncjX7+9i6tRABzLGGN8raoF5COgvIptE5E0RmSkiG3H2ywzzfry/kJEjCZMcJlcfx733wrp1gQ5kjDG+VdTh+r/E2bn/PlAWiHU/7oHnno3JVbs2MmgQf9vzBk1ifuUf/4DjduaQMeYvrDjnwexW1UdVtZeqXqeqjwHHgF7ej/cX89hjSFgY/2kxmg0b4N57Ax3IGGN8p8gFxpyDqlVhyBCqLn6bL1vfx7o3lvPuu4EOZYwxvmEFxt8uvRRUab/yJZaEdOWNgcv59ddAhzLGGO+zAuNva9aACAJEuE7QKedz/vEPyMw82wuNMaZkKdRglyLy4VmaxHohS+nQuTNERsKJE4gqgxp+zahVOTzySCgTJgQ6nDHGeE9hR1M+WIj5W88xS+mQO6Z/Sgps3cp5U6ey9MLbaP/8G1x6aQhXXhnogMYY4x2FKjCq+k9fBylVcsf0B6haleTRo3k/IZQBt0zh+7UhVKsW2HjGGOMNRboejPGBkSMhJ4de48ZxOCyUm/u9xqLFIYSGBjqYMcacG7/u5BeRBBFZICLHRGS7iPQ9Q9v7RGSPiBwRkekiEuGeHiEi09yvTxWR70XkCv+9Cy8TgTFjYMQIBmVPofcXQxj/pA1WZowp+fx9FNlEIBNIxBl9eZKINCrYSER6AA8DXYFaQB1gtHt2GPAb0Aln8M3HgfdEpJaPs/uOCDz5JPrgQ9zFJOJH3sOXX1iRMcaUbH4rMCISg3O2/+OqmqaqS4EPca4lU1B/YJqqrndfmnksMABAVY+p6ij3VTRdqvo/nAMMWvnljfiKCPLM02Te/QBDeJUNl9/H448py5cHOpgxxhSPqJ/GjheRFsAyVY3KN20Y0ElVexZouxYYr6pz3c8rAvuBiqp6sEDbRGA70FxVf/aw3sHAYIDExMRW757jqfNpaWmULVv2nJZxRqqEPjiTDqvf4nnu45HwZ3nhxR9o1OhoYPIUgWXxLJiyQHDlsSyeBVMWgC5duqxW1dZFfqGq+uUGdAD2FJh2G5Dioe0W4PJ8z8MBBWoVaBcOfAZMKUyGVq1a6blasmTJOS/jbMY/6dJ/cbcq6DM8qIMGugKap7Asi2fBlEU1uPJYFs+CKYuqKrBKi/G978+jyNL48wmZsUBqIdrmPs5rKyIhwNs4+3SGeC9m4HXuInSNfJnQjBwe0ud4aloYLzZ8kqH3CSKBTmeMMYXjz538m4AwEUnKN60ZsN5D2/Xuefnb7VX35jEREWAazsECvVQ1yzeRAyM5GRZ/LhwZ+wo7r7qdETxF2gNP0K+vcuxYoNMZY0zh+K3AqOoxYD4wRkRiRKQ9cA1OL6Sgt4CBItJQROKBx4CZ+eZPAhoAPVU13bfJAyM5GUY8GkL1D19DBw7iccZR/93RJCfDFk8XrTbGmCDj78OU7wKigH3AHOBOVV0vIjVFJE1EagKo6ifAs8ASnB3424GRACJyPnA70BzY435dmoj08/N78Y+QEOT1KfDPfzKS0dzwy1hat4b/+79ABzPGmDPz65n8qnoIuNbD9B04V8jMP+0F4AUPbbcDpWtPREgITJ0KOTk88tYTlIsP5aqrHmH0aGjfPtDhjDHGMxsqpqQIDYXp08Hl4u5Zj1KpRSg3PjGciy9uTKtWEBcX6IDGGHMqKzAlSWgozJwJOTncMOdhGnTcyXtfncfgJl0Y+UkyDRsGOqAxxpxkBaakCQ2Ft96CfftotvhVmgInfoviqlaLuevtZHr3DnRAY4xx2BUtS6KwMOjSJe/KmJGk80DZyfTpA8OHQ3Z2oAMaY4wVmJLr0kshMhINCUFEuOrAW6y44EZmPruXK66AAwcCHdAYU9pZgSmp3FfG3Hrrrc7VMUePps2O+eyIbkDtlBm0bqV8912gQxpjSjMrMCVZcjI7+vWDjh3hiSdgzRoiWjbi9exbmb33Mm66aDN9+sCyZYEOaowpjazA/JU0aABffAGTJ9M2dBWrs5pQ999P0+WSLF59Ffw0cLYxxgBWYP56QkLg9tuZfM8G/o8reZoRfKttePPulXTuDEuWWKExxviHFZi/qNZXV+WmqHn0CplPIvv4Vi6i3+r76XlpmhUaY4xfWIH5i3IfA0DrcX/nt083EHL7YAYfe5HdCY2pvu4TLr0UKzTGGJ+yAvMXlpwMI0ZA225xMGkSfPUVsZWjmH3oCn5u3Y/DG/dZoTHG+IwVmNLkkktgzRp44gnqr32ftVkNWNHjcS7/fjyPXLrcCo0xxquswJQ2EREwejR8/z1SrRptFo7j4dRHWRrWhbj1X1uPxhjjNVZgSqtGjeCGG/KGmwnNzuA/ejVfXvcS+zb9cUqhWbYMnnoKli8PdGhjTElig12WZl26QGQkZGZCaCgh1arRYf59/BT9KD9d3I+hG/4fl17ajBD3vyEREc6BA8nJgY1tjCkZrAdTmuUeajZ2rDPczA8/wHffITfeSKPv3mbR/uasS+hAb9dcQl2ZpKfD3Xc7V9PMzAx0eGNMsLMCU9rlHmqW2y1p0QLeeAN27YIJE6gbuZu53MB2zme0jOLw+t1ceSVUrgz9+8N//wsnTgT2LRhjgpMVGONZQgI88ACRv/3ChgkfcbxeCx5nDJuzz2dXh38wPPlLPvxAufpqp9j06wdLl1YkPT3QwY0xwcIKjDmzkBAaPHAldTd+jPzyC3LvvVT98VNGfNKJgzWase7uKdz89zQO/Hc5xx7/L1clLOeGG+Df/4ZjxwId3hgTSFZgTOHVrQsTJjibz954g5CwUBq9cgcT/53IJ+kdeVIe45OsrhxduJw+faBSJejdG+bOhbQ05yg0OxrNmNLDjiIzRRcdDQMHwq23OtXizjuRH35AgDI56fyvxh1suu1eZu3txrRPazBvHpQp41xpU9V5vHAhdOoU6DdijPEl68GY4hOBiy+GyZOdq2uKOIc779rJhc8NZNxbNdkddyE7e93DPbX/S7QrFVXIyICuXeGii+Cee2D2bPjlFzux05i/GuvBmHOXnAyff87W6dOpc+utTuVYtw4WLUIWLaLax2/wXPorjCeMb7mIxaHdcF3ajaUZbZg+PYxXXnEWEx8PbdtCu3bOfdu2zmY2Y0zJZAXGeEdyMjsyMqiTe7hzkybO7f77nS7LsmXsnbmIuos/pf3uUciikRAXh6vbpexu1I1l0d1Y9GtdTqR8Q9anKYzTznxDMrVrnyw47do5598sX+6MMmAnfBoT3KzAGN+LiIAuXajepQswHg4edE7wXLSIkEWLqP6fBVwPXF+lCuzfj+JCw8vw71s+4L0/evD11/Duu6cuMjTUGemmQweoU8e51awJ4eGBeIPGGE+swBj/q1ABrr/euanC5s3w6acwcSLs2YMAkpXB9dMu5/pq1aBxY9KubMw7PzRm6vLG/EQDjufE8O67zv6bXKGhTpGpUwciI+vxzTcni0+dOs4mOBGn7fLlzuAF1hMyxneswJjAEoGkJOfWsqWz9989NhqDBkFqKqxfT9kvJjL4xAkGAy6ErVKH8smNCWvRmF3lG7MxvDFr0+vxy/YyRK9dTo0ti/ngo658w8nqERfnFJq4OFi6FHJynB7PuHHOsQrx8SdvkZGFi2+FypjTswJjgkfu2GievrFzcuDXX9k4bx17F6+joa6jwu/r4LX/EZeTQ0Pg72FhUKMG7NiB5uTwePgY9tw9no3VLmVjWjXW7a3Elq0hrFzpHDINTi176KE/R4mKOllsEhI8Pz5wwDmvJzvbKVSvv+7sJ4qMdF4fGencznZ0nLeKlDeWE0xZvLWcYMrireUEU5YzsQJjgktysudPemgoJCVR/+Ek6j/895PTMzJg0ybnqLV162DBAsjJQQCysjjvhQc5D+gMEBYGVaqQWrkqnx+qxk6tyt7QqnS9uRoxSVXZH16V3VKVvSfKc/gPIf7n5dT8NYXlaZ1ZeiSZ77+Hw4edk0YLyshwxmbzJCSkU16xyb3lFqDMTFi/HlwuCAmB9u2dI+dCQpy3HBp68vGZpu3dC++959Th0FBn6J4aNZwOYkiIc5972769JsuW/Xnejh0wZYpTMMPC4K67oFatov8Kt22D1147uZw773SWo3qy2Obeb95cg5Ur/zxdFbZvh2nTTi5n4EDnPeUqWLjzP899vGMHzJx58ucyYMDpl7Ft2/l88YXnZf/2G7z11snl9O9/8ucLp957mpZ7v2OHM9Rf/vd0/vl/XueWLc7vyNP72rEDZsw49T3VrEmR5f5sXC7fjZQuWopOPmjdurWuWrXqnJaRkpJC586dvRPIC4IpT1BkWb4cunbFlZFBSJky8Oqrzj6f3budEQh274bduzn+yy5kz26i0g//eRlRUU5X5fffnb++0FC44gqoXRtiYsiOiCY9JIafd8Qw6c1ojubEkBEazfX/jKHi+TEcJ5pjxJDqiqHcth+I/O4j9jW8jo0JyaSnO4OD5t42bIBffz256ipVoGJF58sjJ8dZfU4ONE9fTtvjKXwd3pmVYcmnzMvJcQpcVtbJ5eR+qRX8876I5XQmhRQ6n7L5sCi8sYxgXI43lNSfb2ioM6j6iBEeZi5fTvWLL961U7V6UTNYD8b8tbg3s23LPSfnNP+SRec+SE/PKzp5t127YNEi5x6cb/Avv3R23Bw7RlhWFuWANu4bANnAVM+RFJA1E6B8eWcHUFSUMxpCVBR/VIjmq23RHNMoMkKi6dE+iip1ovPmEx3tZHrhhZP/9j76KFxwgVNhVMHlYvMvyoRnXeRkK2XCXNx/n1K3tjNfc1yoKmzZgrw2MW85OXcMwVWrDoqgImzdJkyaJGRlC6Fhwp13Qe3ackr3J2TrFiJemQDZWRAWzomhw3HVueCU9/vrFnj55ZNx773XGWUITha+kF83U+bFZ/KWkznsYaib5FwAL8RptHmL8PQzQnY2hIYJI0ZAUj2BkJNdAvllE6HjRuetzDV6HDRs6HTNQkLYsDGEhx4OITM7hJCwEJ5/MYRGTUJOdt1CQpDQEFi3jl0ffki1nj2hcWNwuRB15f2M1/3g4oH7XLiyXZQJc/Hs0y4a1Hfmq8v5HbDhJ0JGPXHy5zv8EbR23bz/AjQ7h61bXEx61XlcJiyH2wflcH51938Krhxk+zZC3555chn9/wm1aiNhYXnd162/hfGviaFkZIciYaHc/1AYF9R35mluu19+IWTk4+7lhON68ilo0OCU39OGn+HRR5wmoaHCNZHAwgIf3p9+ghEjqALVPH+6z8x6MEUUFP+l5xNMef5SWdw9ITIznbFt8m8/yMqC48ed0TwL3ud//J//wMcfO0VABNq0gQsvdIra8eN598cOppNx6DgxIceJyHHPs2sgmCDSGlilKkV9nV97MCKSAEwDugMHgBGq+s5p2t4HDAeigHnAnaqa4Z43BBgANAHmqOoAn4c3pcuZDjgID3d6InFxZ15Go0bw+efO5rqICHjpJY89qhj37RQul1Nkjh93ek433ugUtvBwmD4dWrU6dSdKYe5XrYLrrju5+XD+fGjd+uTOkfw38Dx99WpnB09ulrffdo7+K6rvvoObb0YzM5EyZZwdHM2b/3nduY89TVOFtWudow1zu0uTJuX1QAp1U3VGY5016+SOsJtuck6yyv+zO90t//wff4Q77jh51MeMGc7vKXfHWcEdaJ6mrVgB3buf/MwsXOj8Y5KT4yw3d5to/see5n3/vbPzK/f39MorzonPBZ2tg/Hjj3DPPWhGRvF6IqrqtxswB5gLlAUuAY4AjTy06wHsBRoB8UAK8HS++dcB1wKTgJmFXX+rVq30XC1ZsuScl+FNwZTHsniwbJluGTRIddmyc16Ojh/vleWcc55gyuKNPMuWqUZFaU5IiGpUVOB/NsHyc8m3nGqwU4vznV+cFxVrRc4/aZlAvXzT3s5fOPJNfwcYn+95V2CPh3bjrMAsCXSEPJbFs2DKohpceYImi7e+1L0kaH4ubsAqLcb3vt/2wYhIC2CZqkblmzYM6KSqPQu0XesuMHPdzysC+4GKqnowX7txQHU9wyYyERkMDAZITExs9W7BMUeKKC0tjbJly57TMrwpmPJYFs+CKQsEVx7L4lkwZQHo0qXLalVtXeQXFqcqFecGdKBALwS4DUjx0HYLcHm+5+E4B+PUKtDOejBBlMeyeBZMWVSDK49l8SyYsqgWvwfjz+vBpAGxBabFAqmFaJv72FNbY4wxQcifBWYTECYiSfmmNQPWe2i73j0vf7u9mm/zmDHGmODmtwKjqseA+cAYEYkRkfbANTg7+gt6CxgoIg1FJB54DJiZO1NEwkQkEggFQkUkUkTspFFjjAki/r5k8l0457Xswzlk+U5VXS8iNUUkTURqAqjqJ8CzwBJgu/s2Mt9yHgPSgYeBm9yPH/PbuzDGGHNWfv2vX1UP4Zy/UnD6DpxzY/JPewF44TTLGQWM8npAY4wxXuPvHowxxphSwgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn7ACY4wxxieswBhjjPEJKzDGGGN8wgqMMcYYn/BrgRGRBBFZICLHRGS7iPQ9Q9v7RGSPiBwRkekiElGc5RhjjAkMf/dgJgKZQCLQD5gkIo0KNhKRHsDDQFegFlAHGF3U5RhjjAkcvxUYEYkBegGPq2qaqi4FPgRu9tC8PzBNVder6mFgLDCgGMsxxhgTIGF+XFc9IEdVN+Wbthbo5KFtI+CDAu0SRaQCULMIy0FEBgOD3U/TRGRjMfPnqggcOMdleFMw5bEsngVTFgiuPJbFs2DKAlC/OC/yZ4EpCxwpMO0IUK4QbXMflyviclDV14HXixr2dERklaq29tbyzlUw5bEsngVTFgiuPJbFs2DKAk6e4rzOn/tg0oDYAtNigdRCtM19nFrE5RhjjAkQfxaYTUCYiCTlm9YMWO+h7Xr3vPzt9qrqwSIuxxhjTID4rcCo6jFgPjBGRGJEpD1wDfC2h+ZvAQNFpKGIxAOPATOLsRxf8NrmNi8JpjyWxbNgygLBlceyeBZMWaCYeURVvR3k9CsTSQCmA92Ag8DDqvqOiNQEfgIaquoOd9v7geFAFDAPuENVM860HL+9EWOMMWfl1wJjjDGm9LChYowxxviEFRhjjDE+YQWmkEQkQkSmucc+SxWR70XkigBnShKREyIyK5A53FluEJEN7vHhtohIhwDlqCUiH4vIYfdYdq+KiF/O9xKRISKySkQyRGRmgXldReRnETkuIktE5PxAZBGRi0RkkYgcEpH9IvK+iJznyyxnylOgzUgRURG5LFBZRCRaRF4TkQPucRC/DGCW691/U6ki8pOIXOvjLGf8jivOZ9gKTOGFAb/hjBgQBzwOvCcitQKYaSKwMoDrB0BEugHPAP/EOeG1I/BrgOK8BuwDzgOa4/y+7vLTuncD43AOQMkjIhVxjnx8HEgAVgFzA5EFiMc5IqgWcD7O+WMzfJzlTHkAEJG6QG/g9wBneR3nd9TAfX9fILKISDVgFnA/znl+DwLviEhlH2Y57XdccT/D/jyTv0RzHx49Kt+k/4nIVqAVsM3feUTkBuAPYBlwgb/XX8BoYIyqfuN+viuAWWoDr6rqCWCPiHyCM/SQz6nqfAARaQ1UzzfrOmC9qr7vnj8KOCAiF6rqz/7Moqr/l7+diLwKfOGLDIXJk8+rOEeNvhaoLCJSH7gaqK6qR92TVwcii/vxH/l+Xx+JyDGgLs4/UL7IcqbvuAoU4zNsPZhiEpFEnPHV/H6Cp4jEAmOAB/y9bg9ZQoHWQCUR2SwiO92bpaICFOll4Ab3po5qwBXAJwHKkqsRznh5QN4f8hb8VPjOoiMBPklZRPoAmar6cSBzAO2A7cBo9yayH0WkV4CyrAI2iMjVIhLq3jyWAfzgrwAFvuOK9Rm2AlMMIhIOzAbe9NV/oGcxFme06d8CsO6CEoFwnM0bHXA2S7XAOTk2EL7A+dAfBXbi/KH+J0BZchVp/Dx/EZGmwBM4m18ClaEsMB4YGqgM+VQHGuP8bqoCQ4A3RaSBv4Ooag7OCefv4BSWd4Db3V/sPufhO65Yn2ErMEUkIiE4owZk4nwA/b3+5sBlwIv+XvdppLvvX1HV31X1APACcKW/g7h/NwtxthXH4IxIG4+zfyiQgm78PBG5APg/4F5V/SpQOXA2r76tqlsDmCFXOpAFjFPVTFX9AlgCdPd3EPeBDs8CnYEyOPtF3nD//ft63Z6+44r1GbYCUwQiIsA0nP/ae6lqVgBidMbZQbtDRPYAw4BeIvJdALLgvl7PTiAYzthNAGrg7IPJcI9dN4MAFLsCThlbT5xrGtUlQJum3Ef/fAaMVVV/DbF0Ol2Be9xH/O3B+f29JyLDA5DFb5ufCqE58KWqrlJVl6quBL7F+efSZ87wHVesz7AVmKKZhHN0SU9VTT9bYx95HecX29x9mwx8BPQIUB5wvsTvFpHK4owdNxT4n79DuHtPW4E7RSRMRMrjXLxu7Rlf6CXudUYCoUCoiESKc4j0AqCxiPRyz38C+MGXm1dPl8W9X+pzYKKqTvbV+gubB6fANObk53k3cDvOEZL+zvIlsAMY4W7THucfuoUByLIS6JDbYxGRFjiboH1dBE/3HVe8z7Cq2q0QN5xDOhU4gdNdzL31C3CuUcCsAGcIxzn65w9gD/AvIDJAWZoDKcBhnAs2vQ9U9uPvQgvcRrnnXQb8jLMZJgWoFYgswEj34/yf4bRA/mwKtNsGXBbA31MjYDlwDGd8xL8HMMsQYDPOZqhfgQd8nOWM33HF+QzbWGTGGGN8wjaRGWOM8QkrMMYYY3zCCowxxhifsAJjjDHGJ6zAGGOM8QkrMMYYY3zCCowxJYg410vpHegcxhSGFRhjCklEZrq/4Avevjn7q40pfex6MMYUzWfAzQWmZQYiiDHBznowxhRNhqruKXA7BHmbr4aIyEfuy8puF5Gb8r9YRJqIyGciki7OZYtnikhcgTb93dciyRCRvfLnSwwniHOp42Mi8quHdTzhXneGexDJt3zxgzDmbKzAGONdo4EPccZEex14y321QkQkGufiZ2lAW+DvwMXku1yuiNwOTMEZQLQpzkjQBUesfQL4AGd027nAdPcIybgvkDUM5zLRScDfgBXef5vGnJ2NRWZMIbl7EjfhDAaY30RVHS4iCryhqrfle81nwB5VvUlEbgMm4FySN9U9vzPONUeSVHWziOzEGbz04dNkUOBpVR3hfh6Gc3G1wao6S0TuxxmNuLEG5nISxuSxfTDGFM2XwOAC0/7I93h5gXnLgavcjxvgDHGe/yJNywAX0FBEjgLVgMVnyZA3ZLuqZovIfqCye9L7wL3AVhFZiNNj+lBVM86yTGO8zjaRGVM0x1V1c4HbgUK+Vjj9hdnUPb8wCvZMFPffsjqX0a6P04s5CjwPrHZfIMoYv7ICY4x3XeTh+Qb345+AZiKS/zrmF+P8HW5Q1b3ALpyLcBWbqp5Q1Y9U9T6gDc41TtqfyzKNKQ7bRGZM0USISJUC03JUdb/78XUishLngky9cYpFO/e82TgHAbwlIk8A8Tg79Oer6mZ3myeBF0VkL86VSqOBrqr6fGHCicgAnL/rb3EOJvgHTo/nlyK+T2POmRUYY4rmMuD3AtN2AdXdj0cBvXCu6rkf+Kc611NHVY+LSA/gJZwju07gHA12b+6CVHWSiGQCDwDPAIeAj4uQ7w9gOM7BBOE4vabrVHVrEZZhjFfYUWTGeIn7CK8+qvrvQGcxJhjYPhhjjDE+YQXGGGOMT9gmMmOMMT5hPRhjjDE+YQXGGGOMT1iBMcYY4xNWYIwxxviEFRhjjDE+8f8BGpnsRLwmbRQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0uFJTP3hFpac",
    "outputId": "bd19c068-b58d-4f5f-9b90-e4f40ca30124"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA+8UlEQVR4nO3deXiU1dn48e89mSyQkJCNAEISwpKwKGCgRlEJqFC1Vlu1Lmi1VWm19u3baqtdbK3118W3u6+vrVvd97rbKqhEQdkSWSSBBAgkIGQjAbKQbeb8/phJOoTsmeWZzP25rrlknuXMmeNM7nmec859xBiDUkop5Wu2QFdAKaVUaNCAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILSwYcEblVRPJFpEVEHu/j2O+LSIWIHBGRx0Qk0mNfgoi8KiKNIlImIlf7vPJKKaW6ZcmAAxwA7gUe6+0gEVkK3AmcA6QDGcAvPQ55AGgFUoBlwIMiMtMH9VVKKdUHsXKmARG5F5hgjLm+h/3PAnuNMT9xPz8HeMYYM1ZEooE6YJYxpsS9/yngc2PMnX55A0oppTrZA12BIZoJvO7xfAuQIiKJQCrg6Ag2HvsXdleQiCwHlgNERUVlp6am+qbGQcTpdGKzWfUi2H+0HbQNOmg7uPTUDiUlJTXGmOSezgv2gBMDHPF43vHvUd3s69g/qruCjDEPAQ8BZGZmmuLiYu/WNAjl5eWRm5sb6GoEnLaDtkEHbQeXntpBRMp6Oy/YQ3UDEOvxvOPf9d3s69hf74d6KaWU6iLYA04hMNvj+Wyg0hhzCCgB7CIytcv+Qj/WTymllJslA46I2EUkCggDwkQkSkS6u/33JHCDiMwQkXjgZ8DjAMaYRuAV4B4RiRaRBcDFwFN+eRNKKaWOY8mAgytwHMM15Pka979/JiKpItIgIqkAxph3gPuAVUCZ+/ELj3JuAUYAVcBzwM3GGL3CUUqpALDkoAFjzN3A3T3sjuly7B+BP/ZQTi1wiRerppRSapCseoWjlFJqmNGAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv7BswBGRBBF5VUQaRaRMRK7u4bi/uZed7ni0iEi9x/48EWn22F/sv3ehlFKqgyWXmHZ7AGgFUoA5wNsissUYU+h5kDHm28C3O56LyOOAs0tZtxpjHvFpbZVSSvXKklc4IhINXArcZYxpMMasAd4Aru3neU/4vpZKKaUGwpIBB5gGOIwxJR7btgAz+zjvUqAa+KjL9t+ISI2IfCwiuV6rpVJKqX6z6i21GOBIl21HgFF9nHcd8KQxxnhsuwMownV77krgTRGZY4zZ7XmiiCwHlgMkJyeTl5c3+NoPEw0NDdoOdN8Ou+ocFB1yMCMxjCnxYYGpmB/pZ8FF28FlsO0gx/9ttgYRmQt8bIwZ6bHtNiDXGHNRD+dMBPYA04wxpb2U/Q7wtjHm/p6OyczMNMXFOrYgLy+P3NzcQFcj4Lq2Q0FZHVc+tJY2hyHSbuPZm3LITosPXAX9QD8LLtoOLj21g4gUGGPm9XSeVW+plQB2EZnqsW02UNjD8QBfBz7pLdi4GUCGWD8Vwj4qqabN4fqh1tLuZF3poQDXSKngYMmAY4xpBF4B7hGRaBFZAFwMPNXLaV8HHvfcICKjRWSpiESJiF1ElgFnA+/6qOoqBFQdbT7ueVS4Jb9GSlmOlb8ptwAjgCrgOeBmY0yhiKS659OkdhwoIqcDE4CXupQRDtyLayBBDfBd4BJjjN4vU4NyuKmVt7YeZH56PD84bypjYyN5bsM+2hxdR+Irpbqy6qABjDG1wCXdbC/HNajAc9taILqbY6uB+T6qogpBf/uwlIbWdn51ySyyxsYyY1wcNz6Zz1Nry/jmmZMCXT2lLM3KVzhKWUrV0WYe/2QPF88eT9bYWADOmT6Gs6cl86f3SjjU0BLgGiplbRpwlOqn/121i3aH4b/Pnda5TUT4+Zemc6zVwe9XlPRytlJKA45S/bCvtonnNpTztfkTSU86/u7tlDGj+Prp6Ty/sZzCA12njymlOmjAUaof/vzeTkSE/1o8tdv93zt3KvEjI/jlG0VYcW6bUlagAUepPnze4OTVTfu57vQ0xsZFdXtM3Ihwbl+SyYa9tby19aCfa6hUcNCAo1QfXt3ZyojwMG7OndLrcVfMn8jM8bH88o1C/vJeCQVldX6qoVLBQQOOUr14cWM5+ZUOLjxlHAnREb0eG2YTrpqfSk1jK396byfLHlmnQUcpDxpwlOpBQVkdd77yGQBvbD7Qr+BxpLmtM29Sc5uTNTurfVhDpYKLBhylerCyqAKnu/+/zdG/nGk5GYlEhts6g86bWw6ckApHqVClAUepnriDjQDhdhs5GYl9npKdFs8zN+Zw+9JM7jw/iwNHmrnkgY8pOnDUt3VVahA27DnEA6t2+e3Wr2VT2ygVaLuqG0mKiWDhOMPV587v9xIE2WnxnceeNTWJGx7P57K/fcJfr5zLuTNSfFllpU5QUFbHutJD5GQkMmVMDBv21PLJ7hreL6qivK4JgIgwG8/ddBrZ6Qk+rYsGHKW60dzm4ONdNVyWPYFzRtcMer2bmePjeP3WBdz4RD43PpnPosxkbl08ddivn6OsoaCsjqsfXkdru9N1qW5cF+5R4TbGxkZ1bKLV4eSWZz/l51+ayRdnjSXM5psVXDTgKNWN9XtqOdbmYHHWGKioGVJZKbFR/Pj8LK55dD2riqv5ePchnguBRdtU4K0rPURLuzuTuYHTJyfyvXOmMjd1NNs+P8qyR9bR1u7EZhME+M6zn5KeOJKlM8cyMiKMM6cme/VzqgFHqW6s2lFFVLiN0ycnsq5i6OVt2ne4899t7kXbNOAoX5uc7ErDJEBkuI3bl2R2fu46+hs7brfNmTiadwsr+MOKYv7+kWsdywc/3M0zN3rvx5EGHKW6MMbwwY4qzpicRFR4mFfKzMlIJMJuo7nNiQj9GoCg1FBVHnVlML/hzEmcf/K4EwKHZ38jwAUnj2NPTQN/WFGC03j/x5GOUlOqi93VjZTXNrEoa4zXyuz4NZk5dhQJ0RF6daP8YmVRJRnJ0fzsSzP6/ZnLyUgiwm4jTPo/OrO/NOAo1cWqHVUArv4bL8pOi+fy7AnUNLTq3Bzlc0eb21hXeojzBjgysuPH0Q+WZJ54O+2ZZyA9nYWLF0N6uuv5AOgtNaW6WFVcRWbKKE4aPcLrZc9NHQ24+nSWzhzr9fKV6vBhcTXtTsN50wc+FL/rrTbAFVyWL4emJtfE5rIy13OAZcv6Va5lr3BEJEFEXhWRRhEpE5GrezjuehFxiEiDxyN3oOUoBVDf3MaGPbXkZiX7pPyZ4+MIDxM2ewwiUMoX3tteSWJ0BHNTvXT79qc/haam47c1Nbm295OVr3AeAFqBFGAO8LaIbDHGFHZz7FpjzJleKEeFuDU7a2h3GhZnevd2Woeo8DCmj4tlU7km9VS+0+ZwsmpHFUtnenFOTXn5wLZ3w5JXOCISDVwK3GWMaTDGrAHeAK4NRDkqdHywo4rYKLtPO/XnThzN1v1HcDh1oTblGxv31HK0ud27mS1SUwe2vRtWvcKZBjiMMZ6LxG8BFvZw/FwRqQFqgaeA3xhj2gdSjogsB5YDJCcnk5eXN+Q3EewaGhpCqh2cxvDuZ8fISrCxZvVHndu93Q6Rje00tTp49u1VTBxlyd98Jwi1z0JPgqUdntnegt0GHNxOXvUOr5Q55ppryPz97wlraenc5oiMpPiaa6jqb5sYYyz3AM4CKrpsuwnI6+bYDGASrqu1k4Ei4McDLcfzMW3aNKOMWbVqVaCr4Fdb9tWZtDveMv8s2Hfcdm+3w57qBpN2x1vmmXVlXi3Xl0Lts9CTYGgHp9Npzvzd++Yb/9jg/cKfftqYtDTjFDEmLc313AOQb3r522rVn1cNQGyXbbFAfdcDjTGlxpg9xhinMeYz4B7gsoGWo9QHO6oQgYXTfDNgoENa4kjiR4azeZ/24yjvK66sZ1/tsQEPh+6XZctg714+/OAD2Lu336PTOlg14JQAdhGZ6rFtNtCfjn4DncuRDKUcFWJW7ahizsTRJMZE+vR1RIQ5E0ezqfywT19Hhab3iioBOMfL88i8wZIBxxjTCLwC3CMi0SKyALgYV//McUTkfBFJcf87C7gLeH2g5ajQVl3fwpb9R3w2Oq2ruanx7Kpu4Ghzm19eT4WOldurmD1xNGNiowJdlRNYMuC43QKMAKqA54CbjTGFIpLqnmvTMTTiHGCriDQC/8IVYH7dVzn+ehMqOOQVu7ILeDOdTW/mTByNMbB13xG/vJ4KDVVHm9my7zDnTbfe1Q1Yd5Qaxpha4JJutpcDMR7PbwduH2g5SnnKK65mzKhIZo7v2uXnG7MnjgZgU3kdZ05N8strquHvve2uH07nzfBeFov77oP582HRop6PWbUKNm7suywrX+Eo5RdtDicflVSzKHMMIr5ZeKqruBHhTE6O1owDyqve217JxIQRTEuJ6fvgfpo/H772NVdQ6c6qVa798+f3XZYGHBXynl1fTn1LO2mJI/36unNT49m073DHcH2lhqSptZ01u2o4d3qKV384LVoEL77YfdDpCDYvvtj7FVAHDTgqpBWU1XHPW0UA/PX9nRSU+W+o8tzU0dQ2trKv9pjfXlMNX6t31tDa7vTJcOjugs5Agw1owFEh7uNdNZ0pZtocrsWm/GVORz+OzsdRXrCyqJLYKDvz0xN8Ur5n0HnssfQBBxvQgKNCnN39DbD5YLGpvmSmjGJEeJjOx1FDtnFvLW9vPcjsiaMJD/Pdn/VFi+Dmm+Gpp9K5+eaBBRvQgKNCXH7ZYRJGRvCD86Z5de32/rCH2Th5QhybdOCAGoKCsjqWPbKeY20O1pUe8ult4VWr4MEH4dpr9/Lggz0PJOiJBhwVsiqONJNXXMVVp03k1sVTA7Ls89zU0RQdOEJzm8Pvr62Gh3Wlh2hrdwLgdBqf3Rb27LP55jf39jiQoDcacFTI+uen+3EauDx7YsDqMHfiaNochqKDRwNWBxXccjIS6RiU5qvbwt0NEOht9FpPNOCokGSM4aX8fXxhUgLpSdEBq0fHaozaj6MGa+7E0USFh3HKSXE+uS3c22i0gQYdDTgqJG3cW8feQ01cMS9wVzcAKbFRjI+L0hVA1aDtqm6gqdXBtaen+eS28MaNvY9G6wg6mmlAqR68sHEfMZF2zj/ZeylABmtO6mjNOKAGLX+v68fKPB8Nh/7Rj/oejbZokeu4vmjAUSGnvrmNf312kItmj2NkRODTCc6dGM/+umNU17f0fbBSXeSX1ZIYHUG6nzNlDIYGHBVy3t56kGNtDi4P8O20DnNSRwPoVY4alIKyOrLT4v2WB3AoNOCokPNC/j6mjIlhrnumf6DNGh9HmMAjq0v9mlpHBb/q+hbKDjUxL93/Q/oHQwOOCik7K+vZVH6YK+ZNtMwvwqKDR3EC6/fUsuyRdRp0VL8VlNUCkJ3mm/4bb9OAo0LKSwX7sduES+aeFOiqdFpXesi1MDrQ1u7ffG4quOXvrSPCbmPWSf5Zx2moNOCokNHmcPLKp/tZnDWG5FGRga5Op5yMRMLdSd1sNvFrPjcV3PLL6pgzYTSR9rBAV6VfLBtwRCRBRF4VkUYRKRORq3s47joRKRCRoyKyX0TuExG7x/48EWl2L0vdICLF/nsXyko+2FFFTUMrX7PIYIEO2WnxPHvjaURHhDEvLT4gKXZU8DnW6mDb50fIDpL+G7BwwAEeAFqBFGAZ8KCIzOzmuJHAfwNJwGnAOZy45PStxpgY9yPTd1VW3lRQVscDq3Z5rU/j4Y9KiY4II3ZE4IdCdzUvPYGL557Elv2aV031z5b9h2l3GuYF0Q8USwYcEYkGLgXuMsY0GGPWAG8A13Y91hjzoDFmtTGm1RjzOfAMsMC/NVbeVlBWx5UPreUPK4qH3JHe0NLOD1/aTH5ZHU2tDr7+2AZLdswvmZFCU6uDT3bXBLoqKgh0fIaD6YrYej/1XKYBDmNMice2LcDCfpx7NlDYZdtvROS3QDHwU2NMXteTRGQ5sBwgOTmZvLwTDgk5DQ0NXmuHXXUOdtQ6yEoIY0p83/ebf59/jDaHqye9pc3Jc+9tpH5yxIBes91pyNvXzuu7W6lvdW0zQOsAy/NmO/SmzWmICoPH39uMrcI6fUzgvzawOiu1w7ufNjM+Wti84RO/v/Zg28GqAScGONJl2xFgVG8nicg3gHnAjR6b7wCKcN2euxJ4U0TmGGN2e55rjHkIeAggMzPT5ObmDqX+w0JeXh7eaIeCsjp+t3ItDqchwu7oM8HgQx/tZlvNDmwC7sU4uTT3VE6fnNSv11pXWoPDCa98up+9h1o5bVICXzn1JO5+o5C2difhdhtXnTu/378MvdUO/XFuxaesKz3EWWcvJMxmjWHb4N82sDKrtIPTafivvBVccPIEcnNP8fvrD7YdrBpwGoCu4/xigfqeThCRS4DfAucaYzrvSRhj1nsc9oSIXAVcANzvtdqqXr392cHjrlY+3lXT4x/7R1aX8ut/7eDCU8Zx3elpvJS/n5cK9vPW1oN9BpyCsjquengdre61QVLjR/KP6+eTm5mMiDB1zCjWlR4iJyPRsrchlswcy1tbD7KpvM5nubFU8NtV3cDR5nbLfo57YtWAUwLYRWSqMWane9tsTrxVBoCIfBF4GLjQGPNZH2UbwDo/HUNAWU1j578NriuPpTPHkjn2+AvWR9fs4d63t3PhyeP4yxVzsIfZ+MKkRBKiI/j7R6WcOSWJ808e1+PrPLV2b2ewEeBr8yewKGtM5/7sIBgBlpuZTHiYsKKoUgOO6pGvE3b6iiUHDRhjGoFXgHtEJFpEFgAXA091PVZEFuMaKHCpMWZDl32jRWSpiESJiF1EluHq43nX9+9CAVQebWb1zhqWzkzhh0szufP8LBpa2rnof9fwyOpSnO57Zv/4eA+/equI82eN5c9XuoJNh9uWZDJ7Qhx3/HMr++uaun2dJz7Zy2ubDyACYQKR4bZ+3YKzmtiocE6fnMSKwgqMMYGujrKoYErY6cmqVzgAtwCPAVXAIeBmY0yhiKTi6pOZYYwpB+4C4oB/eaQqWW2MOR8IB+4FsgAHsAO4xBijc3H85LGP99DudPKTC6aTluha6Oyy7An8+JXPuPft7by6aT/xIyNZs8sVlP561VzCw47/HRRht3H/VadywV9X873nN/PC8pzOgOR0Gn737g7+/mEpS2akcP0Z6Wzad9jSt836smRGCj97bRu7qhqYmtJrt6UKUQVldcxLD46EnZ4sG3CMMbXAJd1sL8c1qKDjeY8rNRhjqoH5vqif6tvR5jaeXVfO+SeP6ww2AEkxkTx0bTa/X1HMA6t2A/XYBL65YNIJwaZDauJI/t9XZvG95zfz5/d2cvvSTFrbnfzo5S28tvkA1+akcfeXZxJmE86YEnxXNp7OcwecFUWVGnDUCarqmyk71MQ1p6UFuioDZtmAo4Lfs+vLqW9p5+aFk0/YJyKMjLB3jkQTXGk6TuslrcvFc07i4101PJC3i5GRYbywYR9ltU38cGkmt+RODrpfez1JiY1i9sTRrCis4DuLpgS6OspiCtz9N8GUYaCDJftwVPBraXfw2Jo9nDkliVknxXV7TE5GIhF2G2EC4XZbv3KI3f3lmYyPi+K+d4opq20iPMyVe2y4BJsOS2aksGX/EQ4eORboqiiLyS+rI9JuY9b47r9XVqYBR/nEa5s+p6q+hW8tzOjxmOy0eJ65MYcfLMnsc25Oh5ERds6b8Z9loZ1OMyyzKy+dmQLAe0WVAa6Jspr8sjpmTxhNhD34/nwHX42V5Tmdhr9/VMrM8bGc2Ud/SnZaPN9ZNGVAHfwXzR5PVPjAroyCzeTkGDKSolmhAUd5ONbqoDDIEnZ60j4c5XUrt1dSWt3IX6+a65NbXR1XRlafxDkUIsJ5M1N4dPUejhxrI25EeKCrpCwgGBN2evL6FY6I3C8ib3azPVZE7haR6R7bvi8iW0VEr7SGCWMMf/twNxMTRnDBrLF9nzBIg7kyCjZLZox15YMrrgp0VZRFvLHlAAB2C6U9Ggiv/qEXkcnAt4BfdrN7HvALXHNjOvwNGANc5816qMDZuLeOTeWHuemsjOMmb6qBmztxNEkxkXpbTQGuuTfPbygH4FtPF1gy43lfvP0X4b+BLcaY/G72zQVacE3aBMAYcwx4khPXr1FBqKCsjh+/spVRUXYuz7bWImfByGYTzpuRwvtFlfz1/ZKg/AOjvOfjXdWdyWyDdSnyfgUcEZkiIm0i8ssu2x8UkXoRmScikcA1wLPdnL8d+D0QCbSJiBGRl927nwdmiMgZQ3onKqAKyuq4+uF17K5u5Firg6KDRwNdpWEhIzma5nYnf1q5c8jrAqngFh3p6nK3BfFgmX4FHGPMLuAR4PsikgQgIj8Hvgl8xX1FkwOMBlZ3U8TXgVLgTeB09+M2977NwFHgi4N9Eyrw1pUe6kycaczwHKocCB2rfxqgNUh/1Srv2FPTSESYje+dM7Xf0wisZiC31H4JhAF3iMgNuPpjrjXGvOfen4Pre7G1m3O3ABOAD4wx69yPMgBjjNN9Ts4g34OyAM9fW8H668uKzpicRJR7voXTQPzIgS1Cp4YHYwzvFVWxKCuZ7507LSiDDQwg4BhjKoA/A98F/g78lzHmRY9DxgNHjTGt3Zw+E4gAPu2h+Gr3+SpInTR6BAZYOC05aH99WVF2WjzP3JTDLbmTGRsbye/e2UFJZY/LQqlhatvnR6k42nzcpOdgNNBBAztx9cOsNcY80GVfFK5BAd05FdfVz+Ye9h8DRgywLspC1uxyrXl3xxezNNh4WXZaPD/6YhYvffsMIu02vv7ohh6XaVDD08rtldgEFnus7xSM+h1w3OvO/B1YCywQkdldDjkE9PSXZi6w2xjTU09yAlDTwz4VBNbsrCYpJoKssZrd2FcmJozkyRu+QFNrO19/dAOHGnr6faeGm5VFlcxLSyAhOrhvqfZ3lNqpwGu4Bg7kAuXAr7sctgMIF5EJ3RQxA4/h0N2YBOgaNUHK6TSs2VXDmVOSsAXphLRgkTU2lseun8/nh49x+d/W8qeVOlx6uNtX28T2g0c5b0ZKoKsyZH0GHBGZAvwbWAF8191H80vgAhE52+PQj9z//UI3xRwGZrtX38wRkc4eZREZDUzzOF8FmR0V9dQ0tHLm1ORAVyUkzEtP4LYl0yitaeQv7+/kir+v5dE1pdQ3t3UeU1BWxwOrdmkwGgbe3+6a+HvuMAg4veZSE5GxuALNdmCZe0QZuCZr/gj4LXAGgDFmr4hsAC7CtTy0p58Dj+K6SooCzgLWuPddCLQCrw7xvagAWb2zGqDPRJ3Ke9ocBsHVMdruNPzqre38v7e3M31cLGmJI1lZVInDabDbbPz8SzOYPn4UIsLOynoKDxxlzsTRnDIhDrvNRphNCA+zUXjgCIUHjrJgStJx/XAFZXW8tbuVUZPqtH8uAFZur2TKmBgmJUX3fbDF9Rpw3CPTTsgvb4xxANNPPIMHgb+IyHeMMU0ex28DTuvhZa4BXjLGHDfBQEQScAWpJbj6d35sjDlhUqn72O8Dd+AaePBPXMtRtwy0HDU4a3bVMC0lhrFxUYGuSsjIyUgkMtxGW7uT8DAbd56fRV1TG/lltawsqqTN4ZqS3upw8rPXt51w/pNry3os+48rS0hLHMmkpGjCRPiwpBqH0/DW3nU6AtHPjhxrY31pLTed3fMyH8HE29min8J15XMLrswCvRKROcAiYFY3ux/AdeWTAswB3haRLcaYwi5lLAXuBBYDB3BdKf3Sva3f5ajBaW5zsH5PbVAudxvMesuYvWHPIa55dAPtDid2m40fn5/F5DExvL7lc14p+ByDa7b6BSePY3HWGNqdhveKKllZVInBtfqq3SYcamiltLqBdnc+lY6Jpxpw/CevuIp2p+Hc6cF/Ow28HHCMMQ4R+SauYdD9MRb4hjuTQScRiQYuBWYZYxqANSLyBnAt/wkkHa4DHu0IICLyK+AZ4M4BlqMGYePeWlrbnZw1TW+n+Vt2Wny3f/y/MCmR5246MRhFR9p5e+tB11WR3cY3Fkzq3Dc5OYaPdlZ37rvvstlkp8VTUFbHsofX0ezOIqETev1rZVElSTERzJ04OtBV8QoxxgS6DicQkbnAJ8aYER7bbgcWGmMu6nLsFuDXxpgX3M+TcE0kTQJSB1DOcmA5wKgxE7MTvvGgT96bUkoNV2W/+1KBMWZeT/utugBbDHCky7YjQHeTPLoe2/HvUQMpxxjzEPAQQGZmpin+7YUDr/Uwk5eXR25ubq/HnP+X1cSNsPP88tP9U6kA6E87DHePv/4+d69t5leXzOLanNC9ferPz8LqndVc++gGHvn6PMuNUOupHeR3vZ9n1QVLGoDYLttige5yenQ9tuPf9QMsRw1QdX0L2w8e5SwdDj3spcXayBo7ipfz9wW6KiFjZVElUeE2zpw6fG5XWzXglAB2EZnqsW020F1Hf6F7n+dxle5RbwMpRw3Qx+50NmcNoy+E6p6IcPm8iWzZf0RzufmBK1lnJWdNTSYqPCzQ1fEaSwYcY0wjrrk894hItIgsAC7GNQquqyeBG0RkhojEAz8DHh9EOWqAVu+sIX5kODPHxwW6KsoPLpkzHrtNeEmvcnyu8MBRDhxpHhbZBTxZMuC43YJrXk0V8ByuuTWFIpIqIg0ikgpgjHkHuA9YBZS5H7/oqxz/vY3hyRjD6p3VnDEliTBNZxMSEmMiWZw1hlc3fU6bw9n3CWrQ3tteiQicE+TJOruy6qABjDG1wCXdbC/HNRjAc9sfgT8OpBw1NDurGqiqb+EszS4QUi6fN5EVRZXkFVcPu1/fVrKyqJLs1HgSYyIDXRWvsvIVjrKwj0rc6Wy0/yak5GYmkxQTobfVfOjdwgoKDxxl+riu452CnwYcNShrdtWQkRTNhPiRga6K8qPwMBtfmXsSH+yo0uURfKCgrI7vPONap/LF/H3DLvmqBhw1YC3tDtaX1urotBB1+byJtDsNr20+EOiqDDvrSg91phJqd7hSCQ0nGnDUgBWU1XGszaHLEYSoaSmjmD0hjpfy92HFTCXB7GT3iE8Bwu22YZdKSAOOGrA1O2sIswk5GQmBrooKkMuyJ7CjwrXUgfKeQ02u25TLctKGZWZuDThqwN7ZVsHY2EhKKhsCXRUVIF+efRIRdpsOHvCyFYWVpMRGcs+XZw67YAMacNQAfVhcRWlNIwcON7PskXXDrlNT9U/cyHCWzEjh9S0HaGl3BLo6XtPbSqkdC9H56jPf3ObgwxLXcPPhulS7ZefhKGt6fYuro9gAbbo+Ski7fN5E3tp6kB++tJXrzkgP+s9BQVkdyx5ZR2u7kzCb8L1zpjIpyTXlr7S6gb9+sJN2h+8Woluzs4amVgdLZ471arlWogFHDYjTPcE8TIZnp6bqv5ERrhxfb2w5wIqiiqDvc1hXeoiWNicGcDoMv19R0u1xvlqIbkVRBaOi7Jw2afh+pzTgqAHZX9fEtJQYLp5z0gkrTarQsmFPbee/h8PV7ri4KDrG3EXabfz2q6cwY3wsIlB04Cg/enkrre6UPqdN8u6AGYfT8N72KhZnjSHCPnx7OjTgqH5rbnOwdf8Rrl+QzncWTQl0dVSA5WQkEhEmtDoMYbbgvtptbXfy8Oo9xI2wc01OGouzUo4LntNSRjExYSS/eWU9+ZUO8svqmJfuvaBTUFZHbWMrS2YM39tpoIMG1ABs+/wIrQ5nUP+KVd6TnRbP49/4AmE2YcmMlKD+XPzl/RK2HzzK7y+fww+XZnX7XrLT4vnOnEguOHksf1hRzGf7u67tOHjvFlYQYbexMHN4z23TgKP6beNe1+iceUH8h0V51xlTkjhjciLbK4J3Ps6n5XU8mLeby7Mn9JmQVET49VdOJjE6ku+9sImm1vYhv74xhhVFFZw5JYmYyOF900kDjuq3/L21ZCRHD7sMtmpocjPHsLu6kX21TYGuyoA1tbZz24tbGBc3gp9fNKNf54weGcEfr5jNnppG7n17+5DrsKOinn21x1gSAtm3NeCofnE6DflldcxP0+wC6ni57ttAee4M4sHkd//ewZ6aRv7n8lMYFRXe7/POmJzE8rMzeHZ9OSsKK4ZUhxWF7rVvpmvAUQqAXdUNHDnWxrx0vZ2mjpeRFM3EhBF8WFwV6KoMyMe7anhibRnXn5HOGZMHnoj2tvMymXVSLHf8cytVR5sHXY93CyvITo0nedTwv3OgAUf1S767/2a+F0fmqOFBRMidNoZPdh8KmqwDq3dW8+2nChgfF8UdX8waVBkRdht/vmIux9oc3PRkPv/7wc4BZyHYV9tE0cGjLJk5/K9uwIIBR0QSRORVEWkUkTIRubqXY68TkQIROSoi+0XkPhGxe+zPE5Fm95LUDSJS7J93Mfzk760lKSaCtERd/0adaOG0ZJpaHZ0/TKysoKyO6x/bSH1LOzWNrRQdHPyAhyljYrju9HS27D/C71eUcPXDA0v3tLKoEmDYD4fuYLmAAzwAtAIpwDLgQRGZ2cOxI4H/BpKA04BzgNu7HHOrMSbG/cj0TZWHv41ltcxLS0BkeOZ4UkNzxpREIsJs5AXBbbXVO6txuJdVcHhhzZnYEXY6vhUt7U7+8fGefi/bsKKogsyUUaQnRQ+pDsHCUgFHRKKBS4G7jDENxpg1wBvAtd0db4x50Biz2hjTaoz5HHgGWOC/GoeGiiPN7Ks9pv03qkcjI+x8YVICecXWHzjQMZPf5qX0TDkZSUSG27AJiMBbWw9y05MFffbr1Da2smFPbcjcTgMQKy2gJCJzgU+MMSM8tt0OLDTGXNSP818Ddhhj7nQ/zwNm4lrPqBj4qTEmr4dzlwPLAZKTk7NffPHFIb2X/thV52BHrYOshDCmxIf5/PUGqqGhgZiYGDYcbOf/trTw89OjyIizXj19raMdQll/2uCdPW08X9zKHxaOIHGEpX7LHueRz1rYUNHOlyaFMyNxYN+9ntqh47s8LcHG7sOGV3a2Em6Dc1PthIcJ07v5jq/e38aj21q5+/Qo0oPse9VTOyxatKjAGDOvp/OsNssoBug6ffcIMKqvE0XkG8A84EaPzXcARbhu0V0JvCkic4wxu7ueb4x5CHgIIDMz0+Tm5g6m/ifYsOcQr28+wLi4KEZFhVNV30zV0RZ2VTeweV8jGIgMd1gy8WFeXh65ubnkvVHIiPB9XPulRYSHWfcPia90tEMo608bTJhRz/PFH9GSMIXc01L9U7EBcjgNP1j9Hl+cNZ4/XjV3wOf31A5dt3yruoFbnvmUN0rrAbBJGxeePI6ZJ8WRHBNJ0qhINn5WzKhIJyfPnkt2kA3GGex3wq8Bx33FsbCH3R8D3wViu2yPBer7KPcS4LfAucaYmo7txpj1Hoc9ISJXARcA9w+o4oO0dncNyx5Zj9PjIjLMJiTFRLjr59rW0uZkXWmN5QJOh/yyWuamjg7JYKP6b3JyDCeNHkFecRVXWzTgfFruylnWV0aBocpIjuFLp4yjuKLelX3awL+3VfDm1oMnHLvs0fWW/MHpC34NOMaY3N72u/tw7CIy1Riz0715NlDYyzlfBB4GLjTGfNZXFQC/9Ho7nIZfvFHYGWxsAjfnTua28zKx2aRz7Y1mdzr0ksoGjDGW65RvaGmn6MBRbl08NdBVURYnIizMTOb1TZ/T2u60ZNbjlUWVhIdJ52RVXzp9chKR4btoa3cSbrfxzA2nkTkulpr6Fh76qJTnNpSH3LpSlvpEGGMagVeAe0QkWkQWABcDT3V3vIgsxjVQ4FJjzIYu+0aLyFIRiRIRu4gsA84G3vXtu3DlRvrVW0WUVDZgtwlh4uqoXJz1n5X8stPieebGHG5fMo3zZ43l9c0H+NHLW2l3pz+3ik3ldTiN5k9T/ZM7LZnGVgf5ZbV9H+xnxhhWFlWSk5E4oKwCg9XxHf/BkkzXFUx6AjGRdtKTork0ewKR4baQW1fKan04ALcAjwFVwCHgZmNMIYCIpOLqk5lhjCkH7gLigH95XBmsNsacD4QD9wJZgAPYAVxijPH5XJyHV5fy+Cd7ueHMSVxw8jjWlR7qdu2Y7LR4stPiMcbw5/d28pf3d3L4WBv3XzWXqHBrdCJu3FuHTWBu6uhAV0UFgTOmJBEeJnxYUj2o2fu+tLu6gT01jXxzQbrfXrPjO97d9mduzOnxb8NwZbmAY4ypBS7pYV85roEFHc8X9VJONTDf2/Xry+ubP+fX/9rBhaeM46cXTMdmkz4/TCLC98+bRkJ0BL94o5Cv/t/HnDcjhbOnjQn4BzF/by3Tx8X65RehCn4xkXbmpyfwYXE1Pz5/eqCrc5wV7kmWVslZ1lMwGs4sdUst2H28q4bbX9pCTkYCf/za7M7bZ/113Rnp/Pe5Uyk6WM9f3t/FskcGNmvZ29qdhs37Dms6GzUgC6cls6OinoNHjgW6Ksd5r6iSWSfFMn70iL4PVj6hAcdL/lmwn2/8YyPjYkfw92vnEWkf3C2x8DBb56iGjrXTA2VfvZOmVodO+FQDkps5BoAPLTQJtLq+hU37DnPe9NBIIWNVGnC84L3tldz20hZaHU4q65vZVdUw6LJyMhKJDLcd9zxQSupcAxjm6ZIEagCmpcQwLi6KDy20XMH72ysxBp8Ph1a904AzRE6nOW4RpvYh5mbq6Ew8a0oSTgPxIwPXd7KzzsGE+BGMjYsKWB1U8BFxDTv+sLia+weRQdkXVhZVctLoEUwf1+cccuVDGnCG6Im1e9lb09g5/NkbQxyz0+L5wxWzsduEZ9aXe6mmA2OMoaTOqf03alAmxo+kqc3Bn1aWBLwvsqm1nTW7ajhvRorl5rmFGg043TjSYvr1BSmuqOc3/97B4qwxvLDcY7y9F0aejBkVxdJZY3m5YD/Nbf5fY6S8tomjrUb7b9SgtLnnkznNfyY2BsrqnTW0tDv1dpoFaMDpRl2L4ao+1rVobnPwvec3ERtl577LTiE7PYHvLJri1WGO15yWxpFjbby55YDXyuyvlwr2AxAdYbmR8yoInDk1Gbt7lKY9LLATG1cWVRIb5cpmrQJLA04PWtudPLK6tMf9971TzI6Kev7n8tkkxfhmadicjASmjInhaT/fVisoq+PBVa78pne+stUS9+BVcMlOi+dv12YTJsLirMDNJ3M4DR/sqGJR1hjNBWgB+n+gBzZxJdv7+evbaG0/Pt3MhyXVPPbxHq4/I51F7iGgviAiLDstlS37DrPt865JtH1n7e6azgWqAn07RAWvc6en8NVTTyKvuJrDTa0BqUNBmX+Sdar+0YDTjfhI4fnlOSw/O4Mn15Zx9cPrOhdTOtTQwu0vbWFaSgx3nj+4tdAH4qunTmBEeBhPryvz+Wt16MgqIIRWniflfTeelcGxNkfABr+8t92VrHPhNN8n61R904DTjbhI4QuTEvnJBdO5/6q5FB44ypfuX8Oz68u44qF11DW28pcr/ZPvLG5EOF+ePZ7XNx/gaHObz18PYE9NI+FhwsWTw0MmbbryjcyxozhrahKPf7KXlnb/Dn4p2FvLS/n7mKGpmSxDA04fLpo9nle/cwY2gZ+8uq1zUmdTq/++PNfkpHGszcEr7o58X3I6De9sq2BR5hgumRqhwUYN2U1nZVBd38KbW05cC8ZXCsrquPqR9dQ1tVF44Kj2Q1qEBpx+yBoby9fmTex8bozxa7/GyRPimD0hjqfXl+PrJcG3fn6EiqPNfHGWpgBR3nHW1CQyU0bxyOpSn39+O6wrPdTZ9+r08/dV9UwDTj8tzBxDVADXr1iWk8auqgbW7/HtOiPvbKvAbhPOydJOVuUdIsINZ01iR0U9H+/yzx/+nIzEzqUWI7Qf0jI04PTTCYsp+flW00WnjCc2yu7TwQPGGN7ZdpDTJycSF8CUOmr4uXjOeJJiInm4l6kG3pSRFA0GTs9I0H5IC9GAMwDZafFen9zZXyMiwrgseyL/3naQ+97Z4ZN70sWV9ew91MT5s8Z5vWwV2iLtYVx3ehofllRTUlnv89fLK6nCAHecP12DjYVowAkic1LjcDjhwbzdPslP9c62CkQ0o67yjWU5aUSF23h09R6fv9b726tIionklJPifP5aqv8sF3BEJEFEXhWRRhEpE5Grezn2ehFxiEiDxyN3MGUFg321rgWtDL6ZkPnOtgrmpyWQPMo3mRNUaEuIjuDSUyfwz0/3++wqHVx53D4sqWZxVvKAF0FUvmW5gAM8ALQCKcAy4EERmdnL8WuNMTEej7whlGVpORmJRNrd/8tEvNoRuremkR0V9SzV0WnKh3IyEmh3Gp9dpQNs3FtLfXO7ZZaSVv9hqYAjItHApcBdxpgGY8wa4A3g2kCWZRXZafE8e1MOp0yIA2NIiI7wWtnvFlYAsHSmfkmV75T7+Cod4IPtVUSE2ThzSpLXy1ZDY7VUwNMAhzGmxGPbFmBhL+fMFZEaoBZ4CviNMaZ9oGWJyHJgOUBycjJ5eXmDfhO+9s2pTu44CD98ajW3zvXO4mgvrD1GeqyNXVs2sMu9raGhwdLt4C/aDt5rg8jDDuwC7cY1ajnycBl5ed6d0Pzmp01Mi7exce0ar5YL+lnoMNh2sFrAiQG6Zqk8AvS0TN9HwCygDJgJvAC0A78ZaFnGmIeAhwAyMzNNbm7uwGvvR+X2nfxxZQkx6acwb4iLpB08cozSdz7gh0szyc2d0rk9Ly8Pq7eDP2g7eK8NcoHZc2v5xj82MjUlhhu/smDIZXoqrW6g8p0P+c55WeSenu7VskE/Cx0G2w5+vaUmInkiYnp4rAEagNgup8UC3Y6jNMaUGmP2GGOcxpjPgHuAy9y7B1RWsLnxrEmkxEZy79vbhzx7e0VhJYBmF1B+MT89gW8sSGfLvsMcOHzMq2W/v70KgMVZvsvirgbPrwHHGJNrjJEeHmcCJYBdRKZ6nDYbKOzvS9A5v3jIZVnayAg7ty3JZPO+w7z92dByVL2zrYKpY2KYnBzjpdop1bvLsyfiNPCyl/MDvr+jkqyxo5gQP9Kr5SrvsNSgAWNMI/AKcI+IRIvIAuBiXH0zJxCR80Ukxf3vLOAu4PXBlBWMLj11AlljR/G7d3YMOhNvbWMr6/cc0qsb5VepiSNZMCWRF/P34XR6J7/akWNtbNxbp1c3FmapgON2CzACqAKeA242xhQCiEiqe65NqvvYc4CtItII/AtXgPl1f8oaDsJswk8vnM6+2mM8tXZwKW9WFlXgNLB0pgYc5V9XzE9lf90xPtntnZFqH5ZU43AazpmuAceqrDZoAGNMLXBJD/vKcQ0G6Hh+O3D7YMoaLs6amszCacn89f2dXJY9gdEjBzZU+oWN+4gbYaelzb9rlSi1ZEYKcSPCeX5jOWdOHfoQ5g+2V5IQHcGciZrKxqqseIWjBugnF0ynvrmdG5/IH9BEujc2f86n5Yc5eqydZY+u1zVDlF9FhYfxlbknsaKwkrrGoS1B3e5wsqq4mtzMZMI0u4BlacAZBhpa2rGJkF9Wx9f+vpaV7lFnPamub+Fnr33G917YDPh2Ep5Svbli/kRaHU5e3fT5kMr5tPwwR4616bIaFqcBZxhYV3oIg6vj1eE0fOvpfH7wwmaKDhw97rhjrQ7uf38nuf+ziuc27GPpjLFE2gO3xo9S08fFMntCHC9s3Dek4f3v76jEbhPOnqbZBazMcn04auByMhKJsNtoa3diD7Nx7vQxvFNYwSubPmfBlERyM8dQsLeO9XsOUdfUxtKZKfzoi1lMTo6hoKyOdaWHyMlI1DTuKiCumJ/KT179jC37jzBn4uhBlfH+9ipOy0hgVJSu42RlGnCGgY7F4TwDx5FjbTy3oZyHPiztXGVRBO69ZBbX5KQdd64GGhVIF80ex6/eKuKFjeWDCjhlhxrZVdXA1V9I7ftgFVB6S22Y6Lo4XNyIcL69cDLXL0jvnAlrwzVXQSkrGRUVzoWnjOONzQdobGkf8PmPf7IXgJRYXVbD6jTgDHMLpiQRGa79NMrarpg/kcZWx4CzZhSU1fGEO+Dc9tIWHWlpcXpLbZjr7nabUlYzLy2ejORoHl2zh+r6ln5/Vp9au5eORAUdIy31M25dGnBCgPbTKKsTERZMTuKpdWX8obKYCLuNZ27M6fVzu+3zI/zrM9ey6Db0Cj4YaMBRSlnCqCjXnyOngdY+rlaqjjZz05P5JMVEcM8lsyiuqNcr+CCgAUcpZQnnTE/hkTV7aG134jSQHNN9mqbmNgc3PVXA4aY2Xr75dGaOj+NcXU46KOigAaWUJWSnxfPcTTnccOYkkkdFcPebRawqrjruGGMMP3x5K1v3H+bPV85h5vi4ANVWDYYGHKWUZWSnxXPXl2bw9nfPIj0xmhufyOel/H2d++//YBdvbjnAj5ZmaYbzIKS31JRSljMmNooXvpXDzU9/yg9f3sqmfYdpaG7njS0H+OqpJ/HthRmBrqIaBA04SilLGhUVzmPXz+eGJzby7PpywJUt42vzJiCiGaGDkd5SU0pZVoTdxmmTEo7LllFQdjiANVJDoQFHKWVpp0/WbBnDheUCjogkiMirItIoImUicnUvx/7NveR0x6NFROo99ueJSLPH/mL/vAullLd0ZMv4wZLMPieDKmuzYh/OA0ArkALMAd4WkS3GmMKuBxpjvg18u+O5iDwOOLscdqsx5hGf1VYp5XOaLWN4sNQVjohEA5cCdxljGowxa4A3gGsHcO4Tvq2lUkqpwbBUwAGmAQ5jTInHti3AzH6ceylQDXzUZftvRKRGRD4WkVyv1FIppdSAWe2WWgxwpMu2I8Cofpx7HfCkOX6d2juAIly36K4E3hSROcaY3V1PFpHlwHKA5ORk8vLyBl77YaahoUHbAW0H0DbooO3gMuh2MMb47QHkAaaHxxpgLtDU5ZzbgDf7KHci0A5k9HHcO8B3+6rntGnTjDJm1apVga6CJWg7aBt00HZw6akdgHzTy99Wv17hGGNye9vv7oexi8hUY8xO9+bZwAkDBrr4OvCJMaa0ryoAOmNMKaUCwFJ9OMaYRuAV4B4RiRaRBcDFwFN9nPp14HHPDSIyWkSWikiUiNhFZBlwNvCuD6qulFKqD5YKOG63ACOAKuA54GbjHhItIqnu+TSpHQeLyOnABOClLuWEA/fiGkhQA3wXuMQYo3NxlFIqAKw2aABjTC1wSQ/7ynENLPDcthaI7ubYamC+D6qolFJqEKx4haOUUmoY0oCjlFLKLzTgKKWU8gsNOEoppfxCA45SSim/0ICjlFLKLzTgKKWU8gsNOEoppfxCA45SSim/0ICjlFLKLzTgKKWU8gsNOEoppfxCA45SSim/0ICjlFLKLzTgKKWU8gsNOEoppfxCA45SSim/0ICjlFLKLywXcETkVhHJF5EWEXm8H8d/X0QqROSIiDwmIpEe+xJE5FURaRSRMhG52qeVV0op1SPLBRzgAHAv8FhfB4rIUuBO4BwgHcgAfulxyANAK5ACLAMeFJGZXq6vUkqpfrBcwDHGvGKMeQ041I/DrwMeNcYUGmPqgF8B1wOISDRwKXCXMabBGLMGeAO41icVV0op1St7oCswRDOB1z2ebwFSRCQRSAUcxpiSLvsXdleQiCwHlruftojINh/UN9gkATWBroQFaDtoG3TQdnDpqR3Sejsp2ANODHDE43nHv0d1s69j/6juCjLGPAQ8BCAi+caYed6tavDRdnDRdtA26KDt4DLYdvDrLTURyRMR08NjzSCKbABiPZ53/Lu+m30d++sH8TpKKaWGyK8BxxiTa4yRHh5nDqLIQmC2x/PZQKUx5hBQAthFZGqX/YWDfwdKKaUGy3KDBkTELiJRQBgQJiJRItLTrb8ngRtEZIaIxAM/Ax4HMMY0Aq8A94hItIgsAC4GnupHNR4a6vsYJrQdXLQdtA06aDu4DKodxBjj7YoMiYjcDfyiy+ZfGmPuFpFUoAiYYYwpdx//A+AOYATwT+DbxpgW974EXMOrz8M16u1OY8yzfnkjSimljmO5gKOUUmp4stwtNaWUUsOTBhyllFJ+oQHHQ6jmXustf52InCMiO0SkSURWiUivE7uClYhEisij7v/v9SKySUTO99gfEu0AICJPi8hBETkqIiUicqPHvpBpBwARmSoizSLytMe2kGkD91SWZhFpcD+KPfYNuB004BwvVHOvdZu/TkSScI30uwtIAPKBF/xeO/+wA/twZaKIw/WeXxSR9BBrB4DfAOnGmFjgy8C9IpIdgu0Arr8JGzuehGgb3GqMiXE/MmHw7aCDBtzcudfqgFkd6XBE5Cngc2PMnQGtnJ+IyL3ABGPM9e7ny4HrjTFnuJ9H40pnMdcYsyNgFfUTEdmKKxlsIiHaDiKSCeQB3wNGE0LtICJXAl/FNTJ2ijHmmlD7TohIHvC0MeaRLtsH1Q56hfMf0+g+91ooXOH0ZCauNgA65zbtJgTaRERScH0mCgnBdhCR/xORJmAHcBD4FyHUDiISC9wD3NZlV8i0gYffiEiNiHwsIrnubYNqBw04/zGg3GshIiTbRETCgWeAJ9y/1kKuHYwxt+B6f2fhunXSQmi1w69wZaLf12V7KLUBuOY4ZgAn4Zrs+aaITGaQ7aAB5z8099qJQq5NRMSGKxtFK3Cre3PItQOAMcbhXtZjAnAzIdIOIjIHOBf4Uze7Q6INOhhj1htj6o0xLcaYJ4CPgQsYZDtowPkPzb12ouNy1bnv005mmLaJiAjwKK5BI5caY9rcu0KqHbph5z/vNxTaIRfXgo7lIlIB3A5cKiKfEjpt0BMDCINtB2OMPtwP4HngOSAaWIDrEnFmoOvlh/dtB6JwjU56yv1vO5DsboNL3dt+B6wLdH192A5/A9YBMV22h0w7AGOAK3HdMgkDlgKNuPIQhkQ7ACOBsR6P3wMvu99/SLSBux1Gu///d/w9WOb+LGQOth0C/qas9MA1vO81d6OWA1cHuk5+et93u3+5eD7udu87F1fH8TFco5XSA11fH7VBmvt9N+O6XdDxWBZi7ZAMfAgcBo4CnwE3eewPiXbo0iZ34xqpFVJt4P4sbMR1m+yw+8fYeUNpBx0WrZRSyi+0D0cppZRfaMBRSinlFxpwlFJK+YUGHKWUUn6hAUcppZRfaMBRSinlFxpwlFJK+YUGHKWChIjEisjdIjI90HVRajA04CgVPOYBvwDCA10RpQZDA45SwWMurmUCigJdEaUGQ1PbKBUERGQ7kNVl8z+NMZcFoj5KDYYGHKWCgIjMx5XNvBD4tXvzQWNMWeBqpdTA2ANdAaVUv2zBtRDa/caYdYGujFKDoX04SgWHmUAE8GmgK6LUYGnAUSo4nIprvZ7NAa6HUoOmAUep4DAX2G2MORroiig1WBpwlAoOM9Dh0CrI6aABpYLDYeBUEVmKay35ncaYQ4GtklIDo1c4SgWHnwOVwGvAWkDT26igo/NwlFJK+YVe4SillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi/+P1tlMhnA7GfBAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = model.predict(X_valid)\n",
    "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nJF6l-EjFpac"
   },
   "source": [
    "- `return_sequences=True` argument returns the output for all the neurons (not only the last time step), and therefore the output shape becomes (batch_size, timesteps, units). It also permits ***deep RNNs***.\n",
    "- In time series, we can use a dense layer as the last layer that is faster and allows application of the other activation functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_EDzjBZYFpac",
    "outputId": "6eaef49f-0383-434b-8fde-add72fda3097"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 7s 998us/sample - loss: 0.0433 - val_loss: 0.0073\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 5s 750us/sample - loss: 0.0066 - val_loss: 0.0052\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 5s 743us/sample - loss: 0.0051 - val_loss: 0.0043\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 5s 710us/sample - loss: 0.0045 - val_loss: 0.0044\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 5s 715us/sample - loss: 0.0040 - val_loss: 0.0038\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 5s 714us/sample - loss: 0.0038 - val_loss: 0.0042\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 5s 713us/sample - loss: 0.0037 - val_loss: 0.0033\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 5s 703us/sample - loss: 0.0036 - val_loss: 0.0035\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 5s 708us/sample - loss: 0.0034 - val_loss: 0.0038\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 5s 709us/sample - loss: 0.0035 - val_loss: 0.0034\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 5s 703us/sample - loss: 0.0034 - val_loss: 0.0031\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 5s 705us/sample - loss: 0.0034 - val_loss: 0.0030\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 5s 694us/sample - loss: 0.0034 - val_loss: 0.0031\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 5s 685us/sample - loss: 0.0033 - val_loss: 0.0039\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 5s 698us/sample - loss: 0.0033 - val_loss: 0.0030\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 5s 694us/sample - loss: 0.0033 - val_loss: 0.0033\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 5s 708us/sample - loss: 0.0034 - val_loss: 0.0033\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 5s 710us/sample - loss: 0.0032 - val_loss: 0.0030\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 5s 707us/sample - loss: 0.0032 - val_loss: 0.0032\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 5s 704us/sample - loss: 0.0032 - val_loss: 0.0029\n"
     ]
    }
   ],
   "source": [
    "# Deep RNN:\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
    "    keras.layers.SimpleRNN(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\")\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6P35D86fFpac",
    "outputId": "2da34075-5293-473b-cad9-35eaf042db49"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 180us/sample - loss: 0.0029\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.0029341063220053912"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CNmhQk7lFpad",
    "outputId": "f2cd7f6e-ef86-4564-bd96-f5e86f8c6a42"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXwU9f348dc7yeYgEBKOhCPchwIKyCGnggXFG6m2+hMVqohHrUKrLVbkkGqLX7CKotUqIiig1Fs8ChhUJCgoqA0oigpyJtwJ5M7798dsQohLyCY7u4t5Px+PeWR35rMz79ndzHs/n5nPfERVMcYYYwItItQBGGOM+WWyBGOMMcYVlmCMMca4whKMMcYYV1iCMcYY4wpLMMYYY1xhCcYYY4wrgppgRKSBiLwqIodFZIuIXF1J2fEisktEDorIHBGJKbdshYjkiUiOd/omOHtgjDGmqoJdg5kNFAApwEjgCRHpUrGQiAwDJgBDgNZAW2BqhWK3qWpd73SKq1EbY4zxW9ASjIjEA5cD96pqjqquBN4ArvVRfBTwjKpmqOp+YBowOlixGmOMqbmoIG6rI1CsqpvKzfsCGOSjbBfg9QrlUkSkoaru9c77u4j8A/gGuEdVV/jaqIiMBcYCxMXF9WzRokWNdqKkpISIiPA5dRVO8VgsvoVTLBBe8VgsvoVTLACbNm3ao6qN/X6hqgZlAs4CdlWYdyOwwkfZzcD55Z57AAVae5/3AeoBMTi1nWyg3Yli6Nmzp9ZUWlpajdcRSOEUj8XiWzjFohpe8VgsvoVTLKqqwFqtxnE/mCkyB0ioMC/BmxxOVLb0cTaAqn6iqtmqmq+qzwEfAxcGOF5jjDE1EMwEswmIEpEO5eZ1AzJ8lM3wLitfbrcebR6rSAEJSJTGGGMCImgJRlUPA68A94lIvIgMAIYD830UnwfcICKdRSQJmAjMBRCRRBEZJiKxIhIlIiOBs4H3grIjxhhjqiTYZ5FuBeKATGAhcIuqZohIS29/lpYAqvou8CCQBmzxTpO96/AAfwOygD3AH4DLVNX6whhjTBgJ5lVkqOo+4DIf87cCdSvMewh4yEfZLKC3WzEaY4wJjKAmGGNM9R06dIjMzEwKCwtrtJ769euzcePGAEVVMxaLb8GMxePxkJycTEJCxWuwas4SjDEngUOHDrF7926aN29OXFwcItW/piU7O5t69eoFMLrqs1h8C1Ysqkpubi7bt28HCHiSCZ+ePMaY48rMzKR58+bUqVOnRsnFmPJEhDp16tC8eXMyMzMDvn5LMMacBAoLC4mLiwt1GOYXKi4ursZNr75YgjHmJGE1F+MWt75blmCMMca4whKMMcYYV1iCMcaclK666iquuOIKv17Tt29f7rzzTpciOmrChAn06tXL9e2EO7tM2RjjihO1648aNYpHH3202ut/8sknS++2XmVvv/02Ho+n2ts0/rEEY4xxxc6dO8sev/XWW9x4443HzDveVXGFhYVVSgL169f3O6YGDRr4/RpTfdZEZkwtkp4OM2dGk57u/raaNGlSNiUmJv5sXv369dm0aRMiwuLFixk0aBCxsbE899xz7N69myuvvLKs789pp53GCy+8cMz6KzaR9e3bl/Hjx3PXXXfRoEEDmjRpwl//+tdjajkVm8iaNGnC9OnTuf7662nWrBktWrRg1qxZx2xnw4YNDBgwgNjYWDp37szSpUuJiopi0aJFVX4viouLmTRpEqmpqcTExNC9e3fefvvtsuWqyr333kvLli2JiYmhY8eOjBkzpmz58uXLOfPMM4mPjycxMZG+ffvyzTfhf/tFq8EYcxIaNw7Wr/fvNQcPwpdfQklJNNOmQdeu4E8loHt3ePhh/7ZZVRMmTGDmzJl069aNmJgYcnNz6du3L3fffTcJCQm88847jBo1ilatWjFw4MDjrmfOnDncddddfPLJJ3z66adcd9119O7dmxEjRhz3NTNmzGDatGncfvvtLFu2jDvuuIOBAwfSo0cPioqKGD58OO3atePTTz/l0KFDjB8/npKSEr/278EHH2TWrFk8+eSTdO/enTlz5jB8+HC+/PJLOnXqxIIFC5g9ezYLFy6kU6dOfP/992W3isnPz+eyyy7jD3/4A4sWLSI/P5+1a9eG1YiXx2MJxpha4uBBcI6LQkmJ87warUyu+OMf/8hllx17H9zx48eXPf7973/P0qVLWbRoUaUJpkePHkycOBGADh068K9//Yvly5dXmmAuvvhibr75ZrKzs+nevTuPPPII77//Pj169GDJkiVs2bKFjz/+mOTkZACmT5/OkCFD/Nq/GTNm8Ne//pUrr7yybB0ffPABM2fO5Omnn2bLli00b96coUOHEhkZSVJSEoMHDwZg79695OTkMHz4cNq2bQtAp06d/Np+qFiCMeYkVJ2aRHo6DBkCBQVKdLTwwgvQr1/gY6uOildcFRUVcf/99/Of//yH7du3U1BQQH5+PhdccEGl6+natesxz5s1a3bCW6BU9pqvv/6a1q1blyUXgD59+pxwf8rLzMxk3759DBgw4Jj5AwcOZNWqVYDT3Dd79mzatGnDsGHDGDRoEFdeeSUej4dmzZpx1VVXMXjwYIYOHcqQIUP4zW9+Q/Pmzf2KIxTCv45ljAmIfv1g+XKYOLGA5cvDJ7kAxMfHH/P8/vvvZ/bs2dx9992kpaWxfv16LrzwQgoKCipdT8WLA0TkhM1Zlb1GVWvcy730HJCv9ZTOa9u2Ld999x2PPfYYderU4c9//jN9+vQhLy8PgIULF/Lxxx/Tv39/Xn75ZTp06EBaWlqN4goGSzDG1CL9+sGf/lQQVsnFl5UrVzJixAiuvvpqunXrRtu2bdm0aVPQ4+jUqRM//vgjWVlZZfM+/fRTv9aRkpJCw4YNWbly5THzV65cSefOncuex8XFcemll/LII4+wdOlS1q1bd8y2evTowd13381HH31Enz59mD/f12DA4cWayIwxYadjx44sWbKE9PR0EhMTeeihh9ixYwetWrUKahwXXXQRLVu2ZNSoUfzjH/8gOzubCRMmICJ+1WzuvPNOHnjgAdq0aUO3bt2YM2cOn332Gc8++ywA//73v4mKiqJ3797Ex8ezYMECoqOjadeuHd988w3z5s3j4osvplmzZnz77bds2LCBiy66yK3dDhhLMMaYsDN16lR++uknzj33XOLj4xkzZgxXXHFF2bglwRIVFcXrr7/OmDFj6N27N23btmXGjBlccsklxMbGVnk9d911F0eOHGHcuHFkZWXRqVMnXnvttbKT9YmJicyYMYNx48ZRXFzMqaeeyuuvv07z5s3Zvn07GRkZPPvss+zdu5emTZsyZsyYYy6CCFuqWmumnj17ak2lpaXVeB2BFE7xWCy+BSKWDRs21DwQr0OHDgVsXTV1MsayevVqBfR///tfyGMJpMq+Y8BarcYx12owxhhTicWLF5OUlET79u3ZvHkz48aN48wzz6RLly6hDi3sWYIxxphKHDx4kLvvvptt27bRsGFDhgwZwsyZM0Md1knBEowxxlRizJgxx9y2xVSdXaZsjDHGFZZgjDHGuMISjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyzBGGPC3tNPP102Kqav57784x//oH379gHftlsmTpxI9+7dXd9OMFmCMca44pJLLmHo0KE+l23cuBER4f3336/WukeOHBnwuysXFRUhIrz22muub6u2sARjjHHFmDFjeP/99/nxxx9/tuyZZ56hVatWZaM2+isuLu6YQcDcFMxt/dJYgjGmNklPJ3rmTGd4S5dddNFFpKSklN2SvlRhYSHz58/n+uuvLxtX/s4776Rjx47ExcXRpk0bJkyYQH5+/nHX7avZ6u9//zspKSnUq1eP0aNHc+TIkWOWf/LJJ5x77rk0atSI+vXrc9ZZZx0z3sppp50GwIgRIxCRsuY1X9t6/PHHadeuHdHR0XTo0IE5c+aULSutCT399NNcfvnlxMfH065dOxYuXFjVtw6AkpISpk6dSmpqKjExMXTt2pU333yzbLmqMmXKFFq1akVMTAxNmzbld7/7XdnyFStW0KdPH+Lj40lMTKRPnz5s3LjRrxhqym4VY8zJaNw4WL/ev9ccPAhffkl0SQlMmwZdu0L9+lV/fffufo3VHBUVxahRo5g7dy6TJ08uSyZvvvkme/bsOeZgmJCQwNy5c2nWrBkZGRncdNNNxMXFMXny5Cpta8GCBUyZMoXHHnuMQYMGsWjRImbMmHFMzSM7O5tRo0Yxa9YsAB599FEuuOACvvvuO5KSklixYgUdO3bk2Wef5fzzzycqyvfhcfHixYwbN46HH36YoUOH8vbbbzN27FiaNm16zJDOU6dOZfr06UyfPp0nn3yS0aNHc9ZZZ5GamlqlfZo5cyYPPfQQ//rXv+jRowfz5s1jxIgRrF+/ntNOO42XXnqJhx9+mIULF9KlSxd2797NmjVrACeJDx8+nJtvvpkFCxZQWFjI2rVryz6DYLEajDG1xcGDUFKCAJSUOM9ddsMNN7B161aWLVtWNu+ZZ57hvPPOo0WLFmXzJk2aRP/+/WndujUXXXQREyZM8OsX/8MPP8z111/PjTfeSMeOHZk0aRI9evQ4pszQoUO55ppr6NSpE506dWL27NlERETw3nvvAdCoUSPAGZulSZMmZc8rmjFjBqNHj+bWW2+lY8eOjBs3jquuuorp06cfU2706NFcffXVtG/fnvvvvx/gZ6NaVmbGjBn85S9/4f/9v//HKaecwv3330/fvn2ZMWMGAFu2bKFZs2ace+65tGzZkt69e3PrrbcCsH//fg4dOsSll15Ku3btOPXUU7nmmms45ZRTqrz9QAhqDUZEGgDPAOcBe4C7VXXBccqOB/4CxAEvA7eoan6FMh2Ar4D/qOo1bsZuTFjxoyZRJj0dhgxBCwqQ6Gh44QXcHju5Q4cOnH322cyZM4fzzjuPHTt28N577/Hiiy8eU+7FF19k1qxZbN68mZycHIqKivz6tb1x40Zuu+22Y+b169ePxYsXlz3fvXs39957LytWrGD37t0UFxdz5MgRtm7d6tc+bdy4sexAXmrgwIFMmjTpmHldu3YtexwdHU2jRo3IzMys0jb27dtHZmYmAwYM+Nl2Si+MuPLKK3n00Udp06YNw4YN4/zzz+fSSy8lOjqa5ORkrrnmGoYOHcqQIUMYMmQIv/nNb6pcewqUYNdgZgMFQAowEnhCRH42qIKIDAMmAEOA1kBbYOpx1rfGrWCN+UXp1w+WL6dg4kRYvtz15FJqzJgxvPbaa+zbt4+5c+fSoEEDLr300rLlK1euZOTIkVx44YW8+eabrFu3jvvuu4+CgoKAxnHNNdewbt06Hn74YVatWsX69etp1qxZtbbja7jkivM8Hs/PlpeUlFRp/c4YX5Vvp1WrVmzatInHH3+cunXrMn78eHr37l127mn+/Pmkp6czcOBAXn31VTp27HhMTTIYgpZgRCQeuBy4V1VzVHUl8AZwrY/io4BnVDVDVfcD04DRFdZ3FXAAWO5q4Mb8kvTrR8Gf/hS05AJwxRVXEBsby/PPP8+cOXO47rrrjjn4fvzxx7Rq1Yp77rmH3r1706FDB59XnlWmU6dOrF69+ph5FZ+vXLmS22+/nQsvvJAuXboQHx/Prl27ypZHRkYSGRlJcXHxCbdVsalr5cqVdO7c2a+YK9OwYUOSk5NPuJ24uDguueQSHn74YVavXs2XX355zH53796dCRMm8OGHHzJgwADmzZsXsBirIphNZB2BYlUtf0H5F8AgH2W7AK9XKJciIg1Vda+IJAD34dRwbqhsoyIyFhgLkJKSwooVK6q/B0BOTk6N1xFI4RSPxeJbIGKpX78+2dnZAYmnuLg4YOuqqiuuuILJkydz4MABrrrqqrLtFxcX06JFC7Zu3cqcOXPo2bMnS5cuLWtCKy2Xl5dX6fOxY8dy22230aVLF/r168err77KZ599RqNGjcrKtG/fnmeffZZOnTqRnZ3NvffeS0xMDPn5+WRnZ1NSUkKLFi1455136NatG9HR0SQlJf1sW7fddhs33HADnTp1YvDgwbz33nssWrSIF198kezsbIqKigDIzc095n1WVfLy8o773ufn51NSUkJ2djbFxcXcfvvtTJ8+nWbNmtG1a1cWLlzI6tWrmTlzJtnZ2cyfPx+Anj17UqdOHRYvXozH46FJkyasX7+e+fPnc8EFF9CsWTM2b97MV199xcCBA4+7/by8vMD/z1RnnOXqTMBZwK4K824EVvgouxk4v9xzD6BAa+/zR4C/eB9PAZ6vSgw9e/asytDUlQqnsd5Vwysei8W3QMRS2Xjp/grFeO+fffaZAtq/f3+fsdx1113aqFEjrVu3rl5++eX62GOPaWRkZFm5f//731q/fv3jPldVnTZtmjZu3Fjj4+N15MiROnHiRG3Xrl3Z8s8//1x79+6tMTEx2q5dO33hhRf0lFNO0WnTppXF8uqrr2r79u01Kiqq7LW+tjV79mxt27atejwebd++vT799NNlywoLCxXQV1999ZjXNG/eXP/5z38e9z265557tFu3bmWxFBcX65QpU7R58+bq8Xj09NNP1zfeeKOs/Msvv6x9+vTR+vXra3x8vPbu3VuXLFmiqqo7duzQyy67TJs2barR0dHasmVLnTBhghYWFh53+5V9x4C1Wp3jfnVeVK0NwRnAkQrz/gS86aPsF8Bvyz1v6E0wDYHuQAYQrZZgwioei8U3SzDHZ7H4FopY3EgwwWwi2wREiUgHVf3WO6+bN1lUlOFd9lK5crvVaR67FufE/1bvya66QKSIdFbVHj7WZYwxJgSCdpJfVQ8DrwD3iUi8iAwAhgPzfRSfB9wgIp1FJAmYCMz1LnsKaIdTk+kO/AtYAgxzdw+MMcb4I9iXKd+K068lE1iI07clQ0RaikiOiLQEUNV3gQeBNGCLd5rsXXZEVXeVTkAOkKeqWUHeF2OMMZUIakdLVd0HXOZj/lacpq7y8x4CHqrCOqcEKj5jjDGBY7eKMeYkod7Od8YEmlvfLUswxpwEPB4Pubm5oQ7D/ELl5ub+7M4DgWAJxpiTQHJyMtu3b+fIkSNWkzEBo6ocOXKE7du3uzLmjd2u35iTQEJCAgA7duygsLCwRuvKy8sjNjY2EGHVmMXiWzBj8Xg8pKSklH3HAskSjDEniYSEhIAcBFasWMEZZ5wRgIhqzmLxLZxiqQlrIjPGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcYUlGGOMMa6wBGOMMcYVlmCMMca4whKMMcYYV1iCMcYY4wpLMMYYY1xhCcYYY4wrLMEYY4xxhSUYY4wxrrAEY4wxxhWWYIwxxrjCEowxxhhXWIIxxhjjCkswxhhjXGEJxhhjjCsswRhjjHGFJRhjjDGusARjjDHGFZZgjDHGuMISjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcUVQE4yINBCRV0XksIhsEZGrKyk7XkR2ichBEZkjIjHllj0vIjtF5JCIbBKRMcHZA2OMMVUV7BrMbKAASAFGAk+ISJeKhURkGDABGAK0BtoCU8sV+TvQWlUTgEuBv4lIT3dDN8YY44+gJRgRiQcuB+5V1RxVXQm8AVzro/go4BlVzVDV/cA0YHTpQu/8/NKn3qmdm/EbY4zxj6hqcDYkcgawSlXjys27ExikqpdUKPsF8ICqvuh93gjIAhqp6l7vvMdxkk4csA44W1VzfGx3LDAWICUlpeeiRYtqtB85OTnUrVu3RusIpHCKx2LxLZxigfCKx2LxLZxiATjnnHM+U9Vefr9QVYMyAWcBuyrMuxFY4aPsZuD8cs89OLWU1hXKRQIDgYmA50Qx9OzZU2sqLS2txusIpHCKx2LxLZxiUQ2veCwW38IpFlVVYK1W47gfzHMwOUBChXkJQHYVypY+Pqasqhar09SWCtwSoDiNMcYEQDATzCYgSkQ6lJvXDcjwUTbDu6x8ud3qbR7zIQo7B2OMMWElaAlGVQ8DrwD3iUi8iAwAhgPzfRSfB9wgIp1FJAmnCWwugIgki8hVIlJXRCK9V5z9P+D9oOyIMcaYKgn2Zcq34pyUzwQWAreoaoaItBSRHBFpCaCq7wIPAmnAFu802bsOxWkO2wbsB2YA41T19aDuiTHGmEpFBXNjqroPuMzH/K1A3QrzHgIe8lE2CxjkVozGGGMCw24VY4wxxhWWYIwxxrjCEowxxhhX1CjBiEiciAwVkVaBCsgYY8wvg18JRkTmisit3sfRwKfAf4FvROQCF+IzxhhzkvK3BjMMWO19fClQD2gCTPFOxhhjDOB/gknC6cMCcD7wsqpmAouAzoEMzBhjzMnN3wSzCzhNRCJxajPLvPPrAoWBDMwYY8zJzd+OlnOAF4EdQDGw3Du/D/B1AOMyxhhzkvMrwajqfSKSAbQEFqtqgXdRETA90MEZY4w5efl9qxhVfdnHvOcCE467SkpCHYExxtQe/l6m/FsROa/c80kisk1E3hORpoEPL7AKCk5cxhhjTGD4e5J/SukDEekB/BWYhTPi5MzAheWOQrsMwRhjgsbfJrJWwDfexyOA11T1QRH5L/BeQCNzgdVgjDEmePytweThdK4EGMLRy5QPlpsftizBGGNM8Phbg/kImCkiK4FewBXe+R2BnwIZmBsswRhjTPD4W4O5DSjASSw3q+oO7/wLOAmayOwcjDHGBI+//WC2AZf4mD8uYBG5yGowxhgTPNUaMllEfoVz7zEFNqhqWkCjcoklGGOMCR6/EoyINAdeBXri3C4GoJmIrAVGlGsyC0vFxXDkCNSpE+pIjDHml8/fczCzcO5B1l5VW6hqC6CDd96sQAfnhu3bQx2BMcbUDv4mmHOB36vqD6UzVPV74HbvsrC3bVuoIzDGmNqhRkMml3PS3OXLEowxxgSHvwlmOTBLRFqUzhCRlsAjwPuBDMwtP4V9bx1jjPll8DfB3A7UAb4XkS0i8iOwGYgD/hDg2AIuMtJqMMYYEyz+9oP5CeghIucCpwICbAC+Ax4CfhvwCAMoOtoSjDHGBEu1+sGo6lJgaelzEekGXB6ooNxiCcYYY4InUCf5TwoejyUYY4wJllqVYKKjISsL8vJCHYkxxvzy1boEA9bZ0hhjgqFK52BE5I0TFEkIQCyuK00w27ZBu3ahjcUYY37pqnqSf28Vlv9wgjIh5/E4f+08jDHGuK9KCUZVf+d2IMFQvgZjjDHGXbXqHExEBCQmWm9+Y4wJhqAmGBFpICKvishh750Arq6k7HgR2SUiB0VkjojEeOfHiMgz3tdni8g6EbmgqjGkploNxhhjgiHYNZjZOEMupwAjgSdEpEvFQiIyDJgADAFaA22Bqd7FUcBPwCCgPnAv8JKItK5KAC1aWIIxxphgCFqCEZF4nN7+96pqjqquBN4ArvVRfBTwjKpmqOp+YBowGkBVD6vqFFX9UVVLVPUtnAsMelYlDqvBGGNMcIiqBmdDImcAq1Q1rty8O4FBqnpJhbJfAA+o6ove542ALKCRqu6tUDYF2AJ0V9WvfWx3LDAWICUlpecFF6Qzd24b/vvfD/B4/N/3nJwc6tat6/fr3BJO8VgsvoVTLBBe8VgsvoVTLADnnHPOZ6ray+8XqmpQJuAsYFeFeTcCK3yU3QycX+65B1CgdYVyHmAZ8GRVYujZs6c+84wqqH7/vVZLWlpa9V7oknCKx2LxLZxiUQ2veCwW38IpFlVVYK1W47gfzHMwOfy8Q2YCkF2FsqWPy8qKSAQwH+eczm1VDSI11flrzWTGGOOuYCaYTUCUiHQoN68bkOGjbIZ3Wflyu9XbPCYiAjyDc7HA5apaWNUgWniHSrMEY4wx7gpaglHVw8ArwH0iEi8iA4DhOLWQiuYBN4hIZxFJAiYCc8stfwLoBFyiqrn+xGE1GGOMCY5gX6Z8K87ol5nAQuAWVc0QkZYikuMdfhlVfRd4EEjDOYG/BZgMICKtgJuA7sAu7+tyRGRkVQKoVw8SEizBGGOM26o14Fh1qeo+4DIf87cCdSvMewhnlMyKZbfgjKRZbamp1pvfGGPcVqtuFVPKOlsaY4z7amWCsc6WxhjjvlqbYHbtgsIqX3tmjDHGX7U2wajCzp2hjsQYY365am2CATvRb4wxbqqVCcY6WxpjjPtqZYKxzpbGGOO+WplgEhKgbl1LMMYY46ZamWBE7FJlY4xxW61MMGC9+Y0xxm21NsFYb35jjHFXrU0wqalOP5iiolBHYowxv0y1OsGUlDg9+o0xxgRerU4wYOdhjDHGLbU2wVhnS2OMcVetTTDW2dIYY9xVaxNMYiLUqWMJxhhj3FJrE4x1tjTGGHfV2gQD1tnSGGPcVKsTjHW2NMYY99TqBJOaCjt2QHFxqCMxxphfnlqfYIqLYffuUEdijDG/PLU+wYA1kxljjBsswWAn+o0xxg21OsFYb35jjHFPrU4wDRpAbKwlGGOMcUOtTjDW2dIYY9xTqxMMWIIxxhi31PoE06KFneQ3xhg31PoEk5oK27c7g48ZY4wJHEswqc6wyZmZoY7EGGN+WSzBWGdLY4xxhSUYSzDGGOOKoCYYEWkgIq+KyGER2SIiV1dSdryI7BKRgyIyR0Riyi27TUTWiki+iMytSUylnS3tRL8xxgRWsGsws4ECIAUYCTwhIl0qFhKRYcAEYAjQGmgLTC1XZAfwN2BOTQNq1Aiio60GY4wxgRa0BCMi8cDlwL2qmqOqK4E3gGt9FB8FPKOqGaq6H5gGjC5dqKqvqOprwN6ax2V9YYwxxg2iqsHZkMgZwCpVjSs3705gkKpeUqHsF8ADqvqi93kjIAtopKp7y5X7G5CqqqMr2e5YYCxASkpKz0WLFv2szB13dAfgkUfWn3A/cnJyqFu37gnLBUs4xWOx+BZOsUB4xWOx+BZOsQCcc845n6lqL79fqKpBmYCzgF0V5t0IrPBRdjNwfrnnHkCB1hXK/Q2YW9UYevbsqb5cfbVq27Y+F/1MWlpa1QoGSTjFY7H4Fk6xqIZXPBaLb+EUi6oqsFarcdwP5jmYHCChwrwEILsKZUsf+ypbY6VDJ1tnS2OMCZxgJphNQJSIdCg3rxuQ4Yo+iCgAABidSURBVKNshndZ+XK7tVzzWCClpkJBAezZ48bajTGmdgpaglHVw8ArwH0iEi8iA4DhwHwfxecBN4hIZxFJAiYCc0sXikiUiMQCkUCkiMSKSFR1Y7O+MMYYE3jBvkz5ViAOyAQWAreoaoaItBSRHBFpCaCq7wIPAmnAFu80udx6JgK5OJcyX+N9PLG6QVmCMcaYwKv2r/7qUNV9wGU+5m8F6laY9xDw0HHWMwWYEqi4bGRLY4wJvFp/qxiAxo3B47He/MYYE0iWYICICGje3GowxhgTSJZgvKw3vzHGBFbtSjC7dkF6us9FlmCMMSawaleC2b4dhgzxmWRKO1sG6c45xhjzi1e7EgxAbi489tjPMklqKuTlwV5XunIaY0ztU/sSjAgsWAC9e8N//1uWaEr7wkydetxWNGOMMX6oXQmmeXP48EN47jmnqjJsGJxzDqSns3+/U2T27OO2ohljjPFD7UowTZrAwIFw3XXwzTdOU9nXX0P//gyYfimn8yWqTivatGmwc2eoAzbGmJNX7Uow5UVHw+9/D5s3wwMP0H7XR6ynOy8wkvaymXfecZrNhg2D55+HnJxQB2yMMSeX2ptgSsXHw913E7Xle3ZcN4ErPK/xTcSp7L/yZv7+hx1s2gTXXutUfq69FtasSaK4ONRBG2NM+LMEUyopidTnHiB662YibrmZxFfm8Ocn2/H9FX9m1Zt7ufpqePNN+POfu9GiBdx5J3zxRaiDNsaY8GUJpqImTeDRR51zNL/9LTJzBv1GtuWpFtPYvWAZy4aM49r26cyaBd27Q9eu8H//53SxMcYYc5QlmONp08a52uyrr5zLyiZNIuaic/nV+7OYvuZXZL66ktmznRa2P//Z6ag5dCjcey9MmWJXoRljTFBv139S6tIFXnkFbr0VnngCUYW8PBJH/Ipb+/bl1n692Pnb3ry0uRf/90o7li93cvZ990G3bk4tp107aNvWmdq1g0aNnO44xhjzS2YJpqquvRbmzkXz85GoKBg+3Lm3zBNP0DTvn9wB3BRbn5X0Yi29WKu92ZnVi/++15IdO4/NJnXrHk025RNP27bQqpVzgVt6OqxYAYMHQ79+odhhY4ypGUswVdWvHyxfzg9z5tD2+uuPHvWLiiAjA9au5cCStTR4bQ1/1IeIphC2A40bU3xeL/a17cUPjXvzpacX2V/9SIOvVvDeusHMfqcfeXlHNxMRAcnJkJnp3GQgOrKYqfcUcMGQAtqmFlA3ugAKvNOaNbReuhRiYiwLGWPCjiUYf/Trx9b8fNqWP5hHRTltYd260eSGG/ghHR5els/5zb6ka8FaWLOGyLVrabzsPRqXlHBmudWNEkHbtKFIPBQfKaA4rwDNL0Az84kqKSCaAiKLSmAqzuRDa0DnzaOoey+iBvZDunSGzt6pUSP33otSqs5dEZYsoc2771qyM8aUsQQTYP36Qb9+MUBvZ7rlFmfB4cOwfj38/e+wZIkzTxWJjcVz2ml4YmKctrHoaHbujWbhK9HkFUdTFBnDRSOiiaoTze4D0ezaG82OPdF03vIOF+UuJpISFGHX+p0krn+WehztEZpTpzHbEzpT0qkzif06oad2JvL0zkS3bEJcHSEmBmR1JW1xJSWQmcmXb29jU9o2eiZvo41nm9M0WH7KzwegFcCLLzpXO4wY4dw1oUsXp1p2MrJ2Snelp9PyhRfsR8kvmCWYYImPhwED4J574P33nSau6Gh4+umf/XM1Bfp5j23nDoaevv730juiQ96gJL8AjYrm45sX81FhXw5mbMPz7QaSdm2g05ENdD6ygc67FpKUdqDspftJ5DM6s4eGXMC7RFJMMZG8E/8bPJElNCv+iZSibTQq2IFHC+kKdPW+tijCw8F6qeTUT+VwgzPJHfBrGmVtpOX/3ka0BFUo/ng1UUuXAlCckEh+z/4U9xvI143PYum+XgwaFsuAAdV7G1075hcWwp49zpSVBR9/7NwvqKjIGU/7ySfh3HOd9kuPJ4AbrkVUnev5P/8c3ngD5s6lTXGxc7XmY4855znj4kIdpQkg0Vo0AEqvXr107dq1NVrHihUrGDx4cM0CCdRRMj2d7yueE/L6299g8mSnEhIhythLdzG840bit24g4acNJO7cQJNta4kpPFz2miKJIjO+DVkxqez2pLIzMpWvs1PJOJTKNpwpO7oRJURQUHB0W31JZzlD8FBAIdEMYRm7aMpAVjKQlZzFR3RmIwD5RLOG3qySgXwSPZDPY/qTG9egtPLmcyqt3GVnO/cqLSmByEj47W+hQwcnd9epc3Rq/lM6Ecvm0XDweUS0aUn84SzqHNlDzKEsYg5lEbE3i33fZHFkSxYNNYu47Cw4cICqKmnQEE1OoaRxE7IiUvgxrwkpXVNo3SeFiGZNkCYpTn+qxo1hzZrjfkblPsZAfR2qtJ7KvsMBi+XjEr585TvOSVxHx8PrnKSybp2TwI8nKgpOP92507l3Sj/UhRUro2oUT5X2qZL/JX/XE4g3MCDHmQASkc9UtZffr7ME459w++CPF096utN9p7SitHy5j+97ejr86lfOr/foaFi2DPr3r9J6ioud8XPy8pybg+qqdHa9+DzxF15DVvt+5OYeXZabC++/tIfs91YxwJt0estaPFoIwPakLmxKHsjXDQeSWxxNm6xP+L7O6WzztCE6P5vovEPEFGRTdCAbyckmgUPUI5sEyaauHn1ej2yS2EciBzneVeCFRLGHRmTRuOzv/qjG7I1oTJY2JlMbk6WNSCnezr+5EQ+FFBHFZKZwgCSasIsUdpf9LX1cl8M/21YJlMVRQgSr6/yKLXU6sz8mhezYZLLjktlZnMyHXyezS5PJjajLwIHQsOHRy9hL/56yL53OWSvYkDyY7xr1O2aZiHPcTktzPpfISBg0CBo0cCoNqk5SLn28Z88ekpIa/WzZ/v3w2WfeHyURTgtn06ZOgi8/tc9Kp8P2FezoOJi9HfsRF1VIyt4NpOxYR6OfPidu4zrq/7i+rLm2ONJDTssuHGjbg/2tz2BvyzMgP49BMy5BCgvQKA9rL56M50g2jX9cQ/LWNcTmOgn/CHF8Tg8+i+hN/aFnor16k5PSjohIITLS2deIiGP/ln/87bfOD62iIid/TZrkXLFZWKBw5AiRB/eRvGEFgxfcSERRASWRHj44dxp7kk6hpLiEkiKluEjZu6eET1c7b1ZkhHL22UrTlBKioxSPR4mOLKHB3m859a3/Q4qL0SgPmyc+S/7Z5+JJaUBsnQhiYyE21smzH3/s5KD+/X13WTjRcSZQya6q+dASTBXUpgQDwf8SniiW8okq7e1c+kSsgZUr4aOPYNUqOHTo+CsvJ5u6ZJNAUst6xCbXo6ROPYrqJFAYW4+Izd8S99WnCIpKBFvPupqNZ9/M/qjG7ItszH5NZNly4cMPnYOqiLNPffs6B6XyB6tWO9Jpu3UFW9oMZlebfscsi4hwWjqXLHHWE89hRvTfzdkddxF7cDdxh3Zz+teL6bg9zYkFyI5uSGRJIfFFvvfzMHXYG5HMgZhk9kUlsy/S+RtdnMtVB54gUosolihmNPoHm2K6UiKRFGkkJRLJvoOR7DsUSTHOVD8pkoQkZxmRzt8SieTUvHWcdjCNTYkD+S6+Ox4pIgpn2p9VRNauIiIpJooikpOKSKhThBYWUVxQjBYW0S4/g78W3UcUhSgRfEt72vIDMThV2hzi+YJurOMMPqcH6ziDDLpQSPTP9rcv6QxmBSsYzGrKf7GU9nxHb9bQmzWcyaecwTrqkAvAPpK8S5wpkmIG8DEbOZXtpNKAfTRgH0nsP+ZvxXmlMbutmAj20pAsGpNFYzJJLnucRWP2SmOyJJl9kc53tAObGFD8AenR57AxpjsJ4vyYqifZ1OcQsYXZFB8o/S/IplndQyREZBNf4kx1S7JpUrSNU4u+QlBKiOATz0B+jGrPIUnkoCRyQJLYV5LI9iOJHCCJ3JhEnnopkZ5Dk5xmyvJZLz2d1P79t29TTfV33y3B+OlkSjDBVqNfXcXFMH780dFGIyLgd7+Dm26CevUgIQHq1SP9y3hWfBhx/GTnzWQl+flExMT4rLpVqXZXBSdcz/FiyctzzvNkZrLxg0z+OWE3DYoyaRKZycihmTTWTNi927lWPTPT+fkdhhQoadOO3At+Te4pZ5DTsQc5Tdqz5vNIbr3VqRh7PM5H2r370dpFZKRTm4iMhDVrVjNgQN9j5pVOa9fCxRc772+cp4ilD2dweu6nRH6+hsh1a4ja+BVygjvPFsTWY2deA/bSgIOSRNteDajfOglNauBU8ZKSiDqQRd2Z96GFhUi0h6JZTxDRozsRkYJECEREsP4L4foxERQUClEe4ZlnI+hympBfIOQXRpBfIMj6dTS7ayRSVIhGRvHdlX8lNyaRyL1ZRB3IIvqA0ywbuT+LZDJpyL7jvq+lqtofOzeqLnmeemVTfP4+Guf8gHjXdyg2mZJID3H5B4gt+nlt+xgeDyQmQlKS80F88w29SkpYq+p393BLMH4KpwM6hFc8NY4lgEf+sDnvUdNYVJ2RVy+7zDliR0XBI49Ap05OUi43fZ1RTMYXRZzWqZhT2hf/bDlvvQWvv340gf/61856o6LKpo3fRvFFRhRdz4ikc9eoY5YRFQX/+x/ceOPRZtXjfEaBqPWecD1HjsAf/whPPXV0n8aOdX6oJCU5B0mPJ6zOwZT/isd5ikj7z156tcpyfnBkZTljg7z11tHq9XnnwaWXOj+yvNOXPyZw3e/rsa+wHvnR9XhtWV36DYg4/oYqfk4FBXDwIOvSDnDHdfuJLzxAw8gDTB2/n3YNDjjnI/fvd/5+/jl8+y29oFoJBlWtNVPPnj21ptLS0mq8jkAKp3gCEsuqVaoPPOD8DXUsARI278uqVapxcVocEaEaF1f9dQXoM1INwHvj3SeNjKzZPgUiliqq9O2r4mdUpY+gCoVOWMQbTw8o0Wocc+0yZRNenI5EoY4i/ATiffHejeLHE/xSD0osgeLdp5Opv1Klb18VP6MqfQRVKHTCIt54dvfvv+MEW/PJEowxtYmvu1Gc7MIp4QVCuH1G/fqxHXZV56UnaRdrY4wx4c4SjDHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcUVQE4yINBCRV0XksIhsEZGrKyk7XkR2ichBEZkjIjHVWY8xxpjQCHYNZjZQAKQAI4EnRKRLxUIiMgyYAAzBGbSxLceO6Vil9RhjjAmdoCUYEYkHLgfuVdUcVV0JvAFc66P4KOAZVc1Q1f3ANGB0NdZjjDEmRILZk78jUKyqm8rN+wIY5KNsF+D1CuVSRKQh0NKP9SAiY4Gx3qc5IvJNNeMv1QioZNSkoAuneCwW38IpFgiveCwW38IpFoBTqvOiYCaYusDBCvMOAvWqULb0cT0/14OqPgU85W+wxyMia7UaA++4JZzisVh8C6dYILzisVh8C6dYwImnOq8L5jmYHCChwrwEILsKZUsfZ/u5HmOMMSESzASzCYgSkQ7l5nUDMnyUzfAuK19ut6ru9XM9xhhjQiRoCUZVDwOvAPeJSLyIDACGA/N9FJ8H3CAinUUkCZgIzK3GetwQsOa2AAmneCwW38IpFgiveCwW38IpFqhmPEEd0VJEGgBzgHOBvcAEVV0gIi2BDUBnVd3qLftH4C9AHPAycLOq5le2nqDtiDHGmBOqVUMmG2OMCR67VYwxxhhXWIIxxhjjCkswVSQiMSLyjPfeZ9kisk5ELghxTB1EJE9Eng9lHN5YrhKRjd77w20WkbNCFEdrEXlbRPZ772X3mIgEpb+XiNwmImtFJF9E5lZYNkREvhaRIyKSJiKtQhGLiPQVkaUisk9EskRksYg0dTOWyuKpUGayiKiIDA1VLCJSR0QeF5E93vsgfhjCWH7r/Z/KFpENInKZy7FUeoyrznfYEkzVRQE/4dwxoD5wL/CSiLQOYUyzgTUh3D4AInIuMB34HU6H17OB70MUzuNAJtAU6I7zed0apG3vAP6GcwFKGRFphHPl471AA2At8GIoYgGScK4Iag20wuk/9qzLsVQWDwAi0g64AtgZ4liewvmMOnn/jg9FLCLSHHge+CNOP7+7gAUikuxiLMc9xlX3OxzMnvwnNe/l0VPKzXpLRH4AegI/BjseEbkKOACsAtoHe/sVTAXuU9XV3ufbQxhLG+AxVc0DdonIuzi3HnKdqr4CICK9gNRyi34NZKjqYu/yKcAeETlVVb8OZiyq+k75ciLyGPCBGzFUJZ5yHsO5avTxUMUiIqcAlwKpqnrIO/uzUMTifXyg3Oe1REQOA+1wfkC5EUtlx7iGVOM7bDWYahKRFJz7qwW9g6eIJAD3AX8K9rZ9xBIJ9AIai8h3IrLN2ywVF6KQHgGu8jZ1NAcuAN4NUSyluuDcLw8o+0feTJAS3wmcTYg7KYvIb4ACVX07lHEAfYAtwFRvE9lXInJ5iGJZC2wUkUtFJNLbPJYPfBmsACoc46r1HbYEUw0i4gFeAJ5z6xfoCUzDudv0TyHYdkUpgAeneeMsnGapM3A6x4bCBzhf+kPANpx/1NdCFEspv+6fFywi0hWYhNP8EqoY6gIPAONCFUM5qcBpOJ9NM+A24DkR6RTsQFS1GKfD+QKcxLIAuMl7YHedj2Nctb7DlmD8JCIROHcNKMD5AgZ7+92BocA/g73t48j1/n1UVXeq6h7gIeDCYAfi/Wzew2krjse5I20SzvmhUAq7++eJSHvgHeAOVf0oVHHgNK/OV9UfQhhDqVygEPibqhao6gdAGnBesAPxXujwIDAYiMY5L/K09//f7W37OsZV6ztsCcYPIiLAMzi/2i9X1cIQhDEY5wTtVhHZBdwJXC4in4cgFrzj9WwDwqHHbgOgBc45mHzvveueJQTJroJj7q0nzphG7QhR05T36p9lwDRVDdYtlo5nCHC794q/XTif30si8pcQxBK05qcq6A58qKprVbVEVdcAn+D8uHRNJce4an2HLcH45wmcq0suUdXcExV2yVM4H2x37/QvYAkwLETxgHMQ/4OIJItz77hxwFvBDsJbe/oBuEVEokQkEWfwui8qf2VgeLcZC0QCkSISK84l0q8Cp4nI5d7lk4Av3WxePV4s3vNS7wOzVfVfbm2/qvHgJJjTOPp93gHchHOFZLBj+RDYCtztLTMA5wfdeyGIZQ1wVmmNRUTOwGmCdjsJHu8YV73vsKraVIUJ55JOBfJwqoul08gQxzUFeD7EMXhwrv45AOwCZgGxIYqlO7AC2I8zYNNiIDmIn4VWmKZ4lw0FvsZphlkBtA5FLMBk7+Py3+GcUL43Fcr9CAwN4efUBUgHDuPcH3FECGO5DfgOpxnqe+BPLsdS6TGuOt9huxeZMcYYV1gTmTHGGFdYgjHGGOMKSzDGGGNcYQnGGGOMKyzBGGOMcYUlGGOMMa6wBGPMSUSc8VKuCHUcxlSFJRhjqkhE5noP8BWn1Sd+tTG1j40HY4x/lgHXVphXEIpAjAl3VoMxxj/5qrqrwrQPypqvbhORJd5hZbeIyDXlXywip4vIMhHJFWfY4rkiUr9CmVHesUjyRWS3/HyI4QbiDHV8WES+97GNSd5t53tvIjnPjTfCmBOxBGNMYE0F3sC5J9pTwDzvaIWISB2cwc9ygDOBEUB/yg2XKyI3AU/i3EC0K86doCvesXYS8DrO3W1fBOZ475CMd4CsO3GGie4AXAx86sJ+GnNCdi8yY6rIW5O4BudmgOXNVtW/iIgCT6vqjeVeswzYparXiMiNwAycIXmzvcsH44w50kFVvxORbTg3L51wnBgU+Ieq3u19HoUzuNpYVX1eRP6Iczfi0zQ0w0kYU8bOwRjjnw+BsRXmHSj3OL3CsnTgIu/jTji3OC8/SNMqoAToLCKHgObA8hPEUHbLdlUtEpEsINk7azFwB/CDiLyHU2N6Q1XzT7BOYwLOmsiM8c8RVf2uwrSniq8Vjj8wm3qXV0XFmoni/V9WZxjtU3BqMYeAmcBn3gGijAkqSzDGBFZfH883eh9vALqJSPlxzPvj/B9uVNXdwHacQbiqTVXzVHWJqo4HeuOMcTKgJus0pjqsicwY/8SISJMK84pVNcv7+NcisgZnQKYrcJJFH++yF3AuApgnIpOAJJwT+q+o6nfeMvcD/xSR3TgjldYBhqjqzKoEJyKjcf6vP8G5mOBKnBrPt37upzE1ZgnGGP8MBXZWmLcdSPU+ngJcjjOqZxbwO3XGU0dVj4jIMOBhnCu78nCuBrujdEWq+oSIFAB/AqYD+4C3/YjvAPAXnIsJPDi1pl+r6g9+rMOYgLCryIwJEO8VXr9R1f+EOhZjwoGdgzHGGOMKSzDGGGNcYU1kxhhjXGE1GGOMMa6wBGOMMcYVlmCMMca4whKMMcYYV1iCMcYY44r/D1c0CFIfwbFMAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RCTK-_WRFpad",
    "outputId": "64a846ee-5ec9-4d21-8916-0bfb73e62690"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXiU1dn48e89mSwkISEbAYQEwpKwKGCgRlEJLlDtoq271Gqr0mrt27fVVrvYWuvbxbe7rz9bt7pUrUu1bq0CShSULVFAEkiAQMKWjYSQhWwz5/fHTOIQsmeWZzL357rmknmWM2eOM7nnec459xFjDEoppZSv2QJdAaWUUqFBA45SSim/0ICjlFLKLzTgKKWU8gsNOEoppfxCA45SSim/0ICjlFLKLywZcETkNhHJF5FWEXmin2O/KyIVIlIvIo+LSKTHvkQReUVEmkSkTESu9XnllVJK9ciSAQc4BNwHPN7XQSKyDLgLOB+YDGQAP/c45EGgDUgFlgMPichsH9RXKaVUP8TKmQZE5D5gojHmhl72PwvsM8b8yP38fOAZY8w4EYkB6oA5xpgS9/6ngYPGmLv88gaUUkp1sQe6AsM0G3jV4/lWIFVEkoA0wNEZbDz2L+6pIBFZAawAiIqKyk5LS/NNjYOI0+nEZrPqRbD/aDtoG3TSdnDprR1KSkpqjDEpvZ0X7AEnFqj3eN7579E97OvcP7qngowxDwMPA2RmZpri4mLv1jQI5eXlkZubG+hqBJy2g7ZBJ20Hl97aQUTK+jov2EN1IxDn8bzz3w097Ovc3+CHeimllOom2ANOITDX4/lcoNIYcwQoAewiMr3b/kI/1k8ppZSbJQOOiNhFJAoIA8JEJEpEerr99xRwo4jMEpEE4CfAEwDGmCbgZeBeEYkRkUXAJcDTfnkTSimlTmDJgIMrcBzHNeT5K+5//0RE0kSkUUTSAIwxbwH3A2uAMvfjZx7l3AqMAqqA54BbjDF6haOUUgFgyUEDxph7gHt62R3b7djfA7/vpZxa4FJv1k0ppdTQWPUKRyml1AijAUcppZRfaMBRSinlFxpwlFJK+YUGHKWUUn6hAUcppZRfaMBRSinlFxpwlFJK+YUGHKWUUn6hAUcppZRfaMBRSinlFxpwlFJK+YUGHKWUUn6hAUcppZRfaMBRSinlFxpwlFJK+YUGHKWUUn5h2YAjIoki8oqINIlImYhc28txf3EvO935aBWRBo/9eSLS4rG/2H/vQimlVCdLLjHt9iDQBqQC84A3RWSrMabQ8yBjzDeBb3Y+F5EnAGe3sm4zxjzq2+oqpZTqiyWvcEQkBrgMuNsY02iMWQe8Blw3wPOe9H0tlVJKDYYlAw4wA3AYY0o8tm0FZvdz3mVANfB+t+2/EpEaEflARHK9V02llFIDZdVbarFAfbdt9cDofs67HnjKGGM8tt0JFOG6PXc18LqIzDPG7PE8UURWACsAUlJSyMvLG3rtR4jGxkZtB3puh911DoqOOJiVFMa0hLDAVMyP9LPgou3gMtR2kBP/NluDiMwHPjDGRHtsux3INcZ8oZdzJgF7gRnGmNI+yn4LeNMY80Bvx2RmZpriYh1bkJeXR25ubqCrEXDd26GgrI6rH15Pu8MQabfx7M05ZKcnBK6CfqCfBRdtB5fe2kFECowxC3o7z6q31EoAu4hM99g2Fyjs5XiArwIf9hVs3Awgw6yfCmHvl1TT7nD9UGvtcLKh9EiAa6RUcLBkwDHGNAEvA/eKSIyILAIuAZ7u47SvAk94bhCRMSKyTESiRMQuIsuBc4G3fVR1FQKqjrWc8Dwq3JJfI6Usx8rflFuBUUAV8BxwizGmUETS3PNp0joPFJEzgYnAi93KCAfuwzWQoAb4NnCpMUbvl6khOdrcxhvbDrNwcgLfu3A64+IieW7Tftod3UfiK6W6s+qgAYwxtcClPWwvxzWowHPbeiCmh2OrgYW+qqMKPX95r5TGtg5+cekcssbFMWt8PDc9lc/T68v4+tlTAl09pSzNylc4SllK1bEWnvhwL5fMnUDWuDgAzp85lnNnpPCH1SUcaWwNcA2VsjYNOEoN0P+t2U2Hw/DfF8zo2iYi/PTzMzne5uC3K0v6OFsppQFHqQHYX9vMc5vKuXLhJCYnn3j3dtrY0Xz1zMn8Y3M5hYe6Tx9TSnXSgKPUAPxx9S5EhP86b3qP+79zwXQSoiP4+WtFWHFum1JWoAFHqX4cbHTyyscHuP7MdMbFR/V4TPyocO5YmsmmfbW8se2wn2uoVHDQgKNUP17Z1cao8DBuyZ3W53FXLZzE7Alx/Py1Qv60uoSCsjo/1VCp4KABR6k+vLC5nPxKB587bTyJMRF9HhtmE65ZmEZNUxt/WL2L5Y9u0KCjlAcNOEr1oqCsjrte/gSA17YcGlDwqG9p78qb1NLuZN2uah/WUKngogFHqV6sKqrA6e7/b3cMLGdaTkYSkeG2rqDz+tZDJ6XCUSpUacBRqjfuYCNAuN1GTkZSv6dkpyfwzE053LEsk7suyuJQfQuXPvgBRYeO+bauSg3Bpr1HeHDNbr/d+rVsahulAm13dRPJsREsHm+49oKFA16CIDs9oevYc6Ync+MT+Vz+lw/589XzuWBWqi+rrNRJCsrq2FB6hJyMJKaNjWXT3lo+3FPDO0VVlNc1AxARZuO5m88ge3KiT+uiAUepHrS0O/hgdw2XZ0/k/DE1Q17vZvaEeF69bRE3PZnPTU/lsyQzhdvOmz7i189R1lBQVse1j2ygrcPpulQ3rgv3qHAb4+KiOjfR5nBy67Mf8dPPz+azc8YRZvPNCi4acJTqwca9tRxvd3Be1lioqBlWWalxUfzwoiy+8thG1hRX88GeIzwXAou2qcDbUHqE1g53JnMDZ05N4jvnT2d+2hi2HzzG8kc30N7hxGYTBPjWsx8xOSmaZbPHER0RxtnTU7z6OdWAo1QP1uysIircxplTk9hQMfzyPt5/tOvf7e5F2zTgKF+bmuJKwyRAZLiNO5Zmdn3uOvsbO2+3zZs0hrcLK/jdymL++r5rHcuH3tvDMzd578eRBhylujHG8O7OKs6amkxUeJhXyszJSCLCbqOl3YkIAxqAoNRwVR5zZTC/8ewpXHTq+JMCh2d/I8DFp45nb00jv1tZgtN4/8eRjlJTqps91U2U1zazJGus18rs/DWZOW40iTERenWj/GJVUSUZKTH85POzBvyZy8lIJsJuI0wGPjpzoDTgKNXNmp1VAK7+Gy/KTk/giuyJ1DS26dwc5XPHWtrZUHqECwc5MrLzx9H3lmZ69XYa6C01pU6ypriKzNTRnDJmlNfLnp82BnD16SybPc7r5SvV6b3iajqchgtnDn4ofvdbbd5i2SscEUkUkVdEpElEykTk2l6Ou0FEHCLS6PHIHWw5SgE0tLSzaW8tuVkpPil/9oR4wsOELR6DCJTyhdU7KkmKiWB+mnVu31r5CudBoA1IBeYBb4rIVmNMYQ/HrjfGnO2FclSIW7erhg6n4bxM795O6xQVHsbM8XF8XK5JPZXvtDucrNlZxbLZvptTMxSWvMIRkRjgMuBuY0yjMWYd8BpwXSDKUaHj3Z1VxEXZfdqpP3/SGLYdqMfh1IXalG9s3lvLsZYOy2W2sOoVzgzAYYzxXCR+K7C4l+Pni0gNUAs8DfzKGNMxmHJEZAWwAiAlJYW8vLxhv4lg19jYGFLt4DSGtz85TlaijXVr3+/a7u12iGzqoLnNwbNvrmHSaEv+5jtJqH0WehMs7fDMjlbsNuDwDvKqd3qt3LGrV5Px6KMsrqqiZexYSm+6iaoLLhh4AcYYyz2Ac4CKbttuBvJ6ODYDmILrau1UoAj44WDL8XzMmDHDKGPWrFkT6Cr41db9dSb9zjfMPwv2n7Dd2+2wt7rRpN/5hnlmQ5lXy/WlUPss9CYY2sHpdJqzf/OO+drfNnm34L//3ZjoaGPg00d0tGu7G5Bv+vjbatWfV41AXLdtcUBD9wONMaXGmL3GGKcx5hPgXuDywZaj1Ls7qxCBxTN8M2CgU3pSNAnR4WzZr/04yvuKKxvYX3t80MOh+/XjH0Nz84nbmptd2wfIqgGnBLCLyHSPbXOBgXT0G+hajmQ45agQs2ZnFfMmjSEpNtKnryMizJs0ho/LdaSa8r7VRZUAnO/leWSUlw9uew8sGXCMMU3Ay8C9IhIjIouAS3D1z5xARC4SkVT3v7OAu4FXB1uOCm3VDa1sPVDvs9Fp3c1PS2B3dSPHWtr98noqdKzaUcXcSWMYGxfl3YLT0ga3vQeWDDhutwKjgCrgOeAWY0yhiKS559p0vsvzgW0i0gT8G1eA+WV/5fjrTajgkFfsyi7gzXQ2fZk3aQzGwLb99X55PRUaqo61sHX/US6c6YPP8f/8D0RHn7gtOtq1fYCsOkoNY0wtcGkP28uBWI/ndwB3DLYcpTzlFVczdnQksyd07/LzjbmT3BkHyus4e3qyX15TjXyrd7h+OF04y3tZLO6/HxYuhCXLl7s2/PjHmPJyJC3NFWyWL2fNGti8uf+yrHyFo5RftDucvF9SzZLMsYj4Z5Jc/KhwpqbEaMYB5VWrd1QyKXEUM1Jj+z94gBYuhCuvhDVrgOXLYd8+3nv3Xdi3ryvYXHml67j+aMBRIe/ZjeU0tHaQnhTd/8FeND8tgY/3H+0crq/UsDS3dbBudw0XzEz16g+nJUvghRc8go6HzmDzwguu4/qjAUeFtIKyOu59owiAP7+zi4Iy/w1Vnp82htqmNvbXHvfba6qRa+2uGto6nN4fDk3PQWewwQY04KgQ98Humq4UM+0O12JT/jKvsx9H5+MoL1hVVElclJ2FkxN9Ur5n0Hn88cmDDjagAUeFOLv7G2DzwWJT/clMHc2o8DCdj6OGbfO+Wt7cdpi5k8YQHua7P+tLlsAtt8DTT0/mllsGF2xAA44KcfllR0mMjuB7F87w+mJT/bGH2Th1Yjwf68ABNQwFZXUsf3Qjx9sdbCg94tPbwmvWwEMPwXXX7eOhh07u0+mPBhwVsirqW8grruKaMyZx23nTA7Ls8/y0MRQdqqel3eH311Yjw4bSI7R3OAFwOo3Pbgt79tl8/ev7eh1I0BcNOCpk/fOjAzgNXJE9KWB1mD9pDO0OQ9HhYwGrgwpuORlJdA5K89Vt4Z4GCPQ1eq03GnBUSDLG8GL+fj4zJZHJyTEBq0fnaozaj6OGav6kMUSFh3HaKfE+uS3c12i0wQYdDTgqJG3eV8e+I81ctSBwVzcAqXFRTIiP0hVA1ZDtrm6kuc3BdWem++S28ObNfY9G6ww6mmlAqV48v3k/sZF2LjrVeylAhmpe2hjNOKCGLH+f68fKAh8Nh/7BD/ofjbZkieu4/mjAUSGnoaWdf39ymC/MHU90RODTCc6flMCBuuNUN7QGuioqCOWX1ZIUE8FkP2fKGAoNOCrkvLntMMfbHVwR4NtpnealuSaA6lWOGoqCsjqy0xP8lgdwODTgqJDzfP5+po2NZb57pn+gzZkQT5jAo2tL/ZpaRwW/6oZWyo40s2Cy/4f0D4UGHBVSdlU28HH5Ua5aMMkyvwiLDh/DCWzcW8vyRzdo0FEDVlBWC0B2um/6b7xNA44KKS8WHMBuEy6df0qgq9JlQ+kR18LoQHuHf/O5qeCWv6+OCLuNOaf4Zx2n4dKAo0JGu8PJyx8d4LyssaSMjgx0dbrkZCQR7k7qZrOJX/O5qeCWX1bHvIljiLSHBboqA2LZgCMiiSLyiog0iUiZiFzby3HXi0iBiBwTkQMicr+I2D3254lIi3tZ6kYRKfbfu1BW8u7OKmoa27jSIoMFOmWnJ/DsTWcQExHGgvSEgKTYUcHneJuD7QfryQ6S/huwcMABHgTagFRgOfCQiMzu4bho4L+BZOAM4HxOXnL6NmNMrPuR6cM6Ky8qKKvjwTW7vdan8cj7pcREhBE3KvBDobtbMDmRS+afwtYDmldNDczWA0fpcBoWBNEPFEsGHBGJAS4D7jbGNBpj1gGvAdd1P9YY85AxZq0xps0YcxB4Bljk3xorbysoq+Pqh9fzu5XFw+5Ib2zt4PsvbiG/rI7mNgdffXyTJTvml85KpbnNwYd7agJdFRUEOj/DwXRFbL2fei4zAIcxpsRj21Zg8QDOPRco7LbtVyLya6AY+LExJq/7SSKyAlgBkJKSQl7eSYeEnMbGRq+1w+46BztrHWQlhjEtof/7zb/NP067w9WT3tru5LnVm2mYGjGo1+xwGvL2d/DqnjYa2lzbDNA2yPK82Q59aXcaosLgidVbsFVYp48J/NcGVmeldnj7oxYmxAhbNn3o99ceajtYNeDEAvXdttUDo/s6SUS+BiwAbvLYfCdQhOv23NXA6yIyzxizx/NcY8zDwMMAmZmZJjc3dzj1HxHy8vLwRjsUlNXxm1XrcTgNEXZHvwkGH35/D9trdmITcC/GyWW5p3Pm1OQBvdaG0hocTnj5owPsO9LGGVMS+dLpp3DPa4W0dzgJt9u45oKFA/5l6K12GIgLKj5iQ+kRzjl3MWE2awzbBv+2gZVZpR2cTsN/5a3k4lMnkpt7mt9ff6jtYNWA0wh0H+cXBzT0doKIXAr8GrjAGNN1T8IYs9HjsCdF5BrgYuAB71VX9eXNTw6fcLXywe6aXv/YP7q2lF/+eyefO20815+Zzov5B3ix4ABvbDvcb8ApKKvjmkc20OZeGyQtIZq/3bCQ3MwURITpY0ezofQIORlJlr0NsXT2ON7YdpiPy+t8lhtLBb/d1Y0ca+mw7Oe4N1YNOCWAXUSmG2N2ubfN5eRbZQCIyGeBR4DPGWM+6adsA1jnp2MIKKtp6vq3wXXlsWz2ODLHnXjB+ti6vdz35g4+d+p4/nTVPOxhNj4zJYnEmAj++n4pZ09L5qJTx/f6Ok+v39cVbAS4cuFElmSN7dqfHQQjwHIzUwgPE1YWVWrAUb3ydcJOX7HkoAFjTBPwMnCviMSIyCLgEuDp7seKyHm4BgpcZozZ1G3fGBFZJiJRImIXkeW4+nje9v27UACVx1pYu6uGZbNT+f6yTO66KIvG1g6+8H/reHRtKU73PbO/fbCXX7xRxEVzxvHHq13BptPtSzOZOzGeO/+5jQN1zT2+zpMf7uNfWw4hAmECkeG2Ad2Cs5q4qHDOnJrMysIKjDGBro6yqGBK2OnJqlc4ALcCjwNVwBHgFmNMoYik4eqTmWWMKQfuBuKBf3ukKllrjLkICAfuA7IAB7ATuNQYo3Nx/OTxD/bS4XTyo4tnkp7kWujs8uyJ/PDlT7jvzR288vEBEqIjWbfbFZT+fM18wsNO/B0UYbfxwDWnc/Gf1/Kdf2zh+RU5XQHJ6TT85u2d/PW9UpbOSuWGsybz8f6jlr5t1p+ls1L5yb+2s7uqkempfXZbqhBVUFbHgsnBkbDTk2UDjjGmFri0h+3luAYVdD7vdaUGY0w1sNAnFVT9OtbSzrMbyrno1PFdwQYgOTaSh6/L5rcri3lwzR6gAZvA1xdNOSnYdEpLiuZ/vjSH7/xjC39cvYs7lmXS1uHkBy9t5V9bDnFdTjr3fHE2YTbhrGnBd2Xj6UJ3wFlZVKkBR52kqqGFsiPNfOWM9EBXZdAsG3BU8Ht2YzkNrR3csnjqSftEhOgIe9dINMGVpuOMPtK6XDLvFD7YXcODebuJjgzj+U37Katt5vvLMrk1d2rQ/drrTWpcFHMnjWFlYQXfWjIt0NVRFlPg7r8JpgwDnSzZh6OCX2uHg8fX7eXsacnMOSW+x2NyMpKIsNsIEwi32waUQ+yeL85mQnwU979VTFltM+FhrtxjIyXYdFo6K5WtB+o5XH880FVRFpNfVkek3cacCT1/r6xMA47yiX99fJCqhla+sTij12Oy0xN45qYcvrc0s9+5OZ2iI+xcOOvTZaGdTjMisysvm50KwOqiygDXRFlNflkdcyeOIcIefH++g6/GyvKcTsNf3y9l9oQ4zu6nPyU7PYFvLZk2qA7+L8ydQFT44K6Mgs3UlFgykmNYqQFHeTje5qAwyBJ2etI+HOV1q3ZUUlrdxJ+vme+TW12dV0ZWn8Q5HCLChbNTeWztXuqPtxM/KjzQVVIWEIwJOz15/QpHRB4Qkdd72B4nIveIyEyPbd8VkW0ioldaI4Qxhr+8t4dJiaO4eM64/k8YoqFcGQWbpbPGufLBFVcFuirKIl7beggAu4XSHg2GV//Qi8hU4BvAz3vYvQD4Ga65MZ3+AowFrvdmPVTgbN5Xx8flR7n5nIwTJm+qwZs/aQzJsZF6W00Brrk3/9hUDsA3/l5gyYzn/fH2X4T/BrYaY/J72DcfaMU1aRMAY8xx4ClOXr9GBaGCsjp++PI2RkfZuSLbWoucBSObTbhwVirvFFXy53dKgvIPjPKeD3ZXdyWzDdalyAcUcERkmoi0i8jPu21/SEQaRGSBiEQCXwGe7eH8HcBvgUigXUSMiLzk3v0PYJaInDWsd6ICqqCsjmsf2cCe6iaOtzkoOnws0FUaETJSYmjpcPKHVbuGvS6QCm4xka4ud1sQD5YZUMAxxuwGHgW+KyLJACLyU+DrwJfcVzQ5wBhgbQ9FfBUoBV4HznQ/bnfv2wIcAz479LehAm1D6ZGuxJnGjMyhyoHQufqnAdqC9Fet8o69NU1EhNn4zvnTBzyNwGoGc0vt50AYcKeI3IirP+Y6Y8xq9/4cXN+LbT2cuxWYCLxrjNngfpQBGGOc7nNyhvgelAV4/toK1l9fVnTW1GSi3PMtnAYSoge3CJ0aGYwxrC6qYklWCt+5YEZQBhsYRMAxxlQAfwS+DfwV+C9jzAseh0wAjhlj2no4fTYQAXzUS/HV7vNVkDplzCgMsHhGStD++rKi7PQEnrk5h1tzpzIuLpLfvLWTkspel4VSI9T2g8eoONZywqTnYDTYQQO7cPXDrDfGPNhtXxSuQQE9OR3X1c+WXvYfB0YNsi7KQtbtdq15d+dnszTYeFl2egI/+GwWL37zLCLtNr762KZel2lQI9OqHZXYBM7zWN8pGA044LjXnfkrsB5YJCJzux1yBOjtL818YI8xpree5ESgppd9Kgis21VNcmwEWeM0u7GvTEqM5qkbP0NzWwdffWwTRxp7+32nRppVRZUsSE8kMSa4b6kOdJTa6cC/cA0cyAXKgV92O2wnEC4iE3soYhYew6F7MAXQNWqClNNpWLe7hrOnJWML0glpwSJrXByP37CQg0ePc8Vf1vOHVTpceqTbX9vMjsPHuHBWaqCrMmz9BhwRmQb8B1gJfNvdR/Nz4GIROdfj0Pfd//1MD8UcBea6V9/MEZGuHmURGQPM8DhfBZmdFQ3UNLZx9vSUQFclJCyYnMjtS2dQWtPEn97ZxVV/Xc9j60ppaGnvOqagrI4H1+zWYDQCvLPDNfH3ghEQcPrMpSYi43AFmh3AcveIMnBN1vwB8GvgLABjzD4R2QR8Adfy0J5+CjyG6yopCjgHWOfe9zmgDXhluG9GBcbaXdUA/SbqVN7T7jAIro7RDqfhF2/s4H/e3MHM8XGkJ0WzqqgSh9Ngt9n46ednMXPCaESEXZUNFB46xrxJYzhtYjx2m40wmxAeZqPwUD2Fh46xaFryCf1wBWV1vLGnjdFT6rR/LgBW7ahk2thYpiTH9H+wxfUZcNwj007KL2+McQAzTz6Dh4A/ici3jDHNHsdvB87o5WW+ArxojDlhgoGIJOIKUktx9e/80Bhz0qRS97HfBe7ENfDgn7iWo24dbDlqaNbtrmFGaizj4qMCXZWQkZORRGS4jfYOJ+FhNu66KIu65nbyy2pZVVRJu8M1Jb3N4eQnr24/6fyn1pf1WvbvV5WQnhTNlOQYwkR4r6Qah9Pwxr4NOgLRz+qPt7OxtJabz+19mY9g4u1s0U/juvK5FVdmgT6JyDxgCTCnh90P4rrySQXmAW+KyFZjTGG3MpYBdwHnAYdwXSn93L1twOWooWlpd7Bxb21QLncbzPrKmL1p7xG+8tgmOhxO7DYbP7woi6ljY3l160FeLjiIwTVb/eJTx3Ne1lg6nIbVRZWsKqrE4Fp91W4TjjS2UVrdSIc7n0rnxFMNOP6TV1xFh9Nwwczgv50GXg44xhiHiHwd1zDogRgHfM2dyaCLiMQAlwFzjDGNwDoReQ24jk8DSafrgcc6A4iI/AJ4BrhrkOWoIdi8r5a2DifnzNDbaf6WnZ7Q4x//z0xJ4rmbTw5GMZF23tx22HVVZLfxtUVTuvZNTYnl/V3VXfvuv3wu2ekJFJTVsfyRDbS4s0johF7/WlVUSXJsBPMnjQl0VbzC6+vhGGM2ABsGeOxbveyaATiMMSUe27YCi3s4djbwarfjUt0DE9IGWo6IrABWAERHR4+4JYt9ZUzu14jL/iLnn5qGaddhulYXMSGLqLRTaSn/hAX37RzQvogJWSScdyORqdP4TOYknK1N/q52aLLZmfRfz9C0cx1hYUsDXRuvsGr++Figvtu2eqCnSR7dj+389+jBlGOMedgYs8AYs2DixIkYY0L+sWbNmn6Pybn0a5w5IxVnW0vA6xvIdgiWR+vBHdSvf4HWgzsGvK/14A5+ffl8xB7OE3mFAX8PofJZeH/nIWyRMTz/ux8H/H0PtB36Y9WA0wjEddsWB/SU06P7sZ3/bhhkOWqQqhta2XH4GOfocOgRLz3ORta40byUvz/QVQkZq4oqiQq3cfb0kXO72qoBpwSwi8h0j21zgZ46+gvd+zyPqzSuUW+DKUcN0gfudDbnjKAvhOqZiHDFgklsPVCvudz8wBjXQI5zpqcQFR4W6Op4jSUDjjGmCddcnntFJEZEFgGX4BoF191TwI0iMktEEoCfAE8MoRw1SGt31ZAQHc7sCfGBroryg0vnTcBuE17UqxyfKzx0jEP1LSMiu4AnSwYct1txzaupAp7DNbemUETSRKRRRNKga+DB/cAaoMz9+Fl/5fjvbYxMxhjW7qrmrGnJhGk6m5CQFBvJeVljeeXjg7Q7nP2foIZs9Y5KROD8IE/W2Z3XR6l5izGmFri0h+3luAYDeG77PfD7wZSjhmdXVSNVDa2co9kFQsoVCyaxsqiSvOLqEffr20pWFVWSnZZAUmxkoKviVVa+wlEW9n6JO9+qTyQAAB8kSURBVJ2N9t+ElNzMFJJjI/S2mg+9XVhB4aFjzBzffbxT8NOAo4Zk3e4aMpJjmJgQHeiqKD8KD7Pxpfmn8O7OKl0ewQcKyur41jOudSpfyN8/4pKvasBRg9ba4WBjaa2OTgtRVyyYRIfT8K8thwJdlRFnQ+mRrlRCHQ5XKqGRRAOOGrSCsjqOtzt0OYIQNSN1NHMnxvNi/v4BTfZTA3eqe8SnAOF224hLJaQBRw3aul01hNmEnIzEQFdFBcjl2RPZWeFa6kB5z5Fm123K5TnpIzIztwYcNWhvba9gXFwkJZWNga6KCpAvzj2FCLtNBw942crCSlLjIrn3i7NHXLABDThqkN4rrqK0polDR1tY/uiGEdepqQYmPjqcpbNSeXXrIVo7HIGujtf0tVJq50J0vvrMt7Q7eK/ENdx8pC7Vbtl5OMqaXt3q6ig2QLuujxLSrlgwiTe2Heb7L27j+rMmB/3noKCsjuWPbqCtw0mYTfjO+dOZkuya8lda3cif391Fh8N3C9Gt21VDc5uDZbPHebVcK9GAowbF6Z5gHiYjs1NTDVx0hCvH12tbD7GyqCLo+xw2lB6htd2JAZwOw29XlvR4nK8WoltZVMHoKDtnTBm53ykNOGpQDtQ1MyM1lkvmnXLSSpMqtGzaW9v175FwtTs+PorOMXeRdhu//vJpzJoQhwgUHTrGD17aRps7pc8ZU7w7YMbhNKzeUcV5WWOJsI/cng4NOGrAWtodbDtQzw2LJvOtJdMCXR0VYDkZSUSECW0OQ5gtuK922zqcPLJ2L/Gj7HwlJ53zslJPCJ4zUkczKTGaX728kfxKB/lldSyY7L2gU1BWR21TG0tnjdzbaaCDBtQgbD9YT5vDGdS/YpX3ZKcn8MTXPkOYTVg6KzWoPxd/eqeEHYeP8dsr5vH9ZVk9vpfs9AS+NS+Si08dx+9WFvPJge5rOw7d24UVRNhtLM4c2XPbNOCoAdu8zzU6Z0EQ/2FR3nXWtGTOmprEjorgnY/zUXkdD+Xt4Yrsif0mJBURfvmlU0mKieQ7z39Mc1vHsF/fGMPKogrOnpZMbOTIvumkAUcNWP6+WjJSYkZcBls1PLmZY9lT3cT+2uZAV2XQmts6uP2FrYyPH8VPvzBrQOeMiY7g91fNZW9NE/e9uWPYddhZ0cD+2uMsDYHs2xpw1IA4nYb8sjoWpmt2AXWiXPdtoDx3BvFg8pv/7GRvTRP/e8VpjI4KH/B5Z01NZsW5GTy7sZyVhRXDqsPKQvfaNzM14CgFwO7qRuqPt7Ngst5OUyfKSI5hUuIo3iuuCnRVBuWD3TU8ub6MG86azFlTB5+I9vYLM5lzShx3/nMbVcdahlyPtwsryE5LIGX0yL9zoAFHDUi+u/9moRdH5qiRQUTInTGWD/ccCZqsA2t3VfPNpwuYEB/FnZ/NGlIZEXYbf7xqPsfbHdz8VD7/9+6uQWch2F/bTNHhYyydPfKvbsCCAUdEEkXkFRFpEpEyEbm2j2OvF5ECETkmIgdE5H4RsXvszxORFveS1I0iUuyfdzHy5O+rJTk2gvQkXf9GnWzxjBSa2xxdP0ysrKCsjhse30xDawc1TW0UHR76gIdpY2O5/szJbD1Qz29XlnDtI4NL97SqqBJgxA+H7mS5gAM8CLQBqcBy4CERmd3LsdHAfwPJwBnA+cAd3Y65zRgT635k+qjOI97msloWpCciMjJzPKnhOWtaEhFhNvKC4Lba2l3VONzLKji8sOZM3Cg7nd+K1g4nf/tg74CXbVhZVEFm6mgmJ8cMqw7BwlIBR0RigMuAu40xjcaYdcBrwHU9HW+MecgYs9YY02aMOQg8AyzyX41DQ0V9C/trj2v/jepVdISdz0xJJK/Y+gMHOmfy27yUniknI5nIcBs2ARF4Y9thbn6qoN9+ndqmNjbtrQ2Z22kAYqUFlERkPvChMWaUx7Y7gMXGmC8M4Px/ATuNMXe5n+cBs3GtZ1QM/NgYk9fLuSuAFQApKSnZL7zwwvDezADsrnOws9ZBVmIY0xLCfP56g9XY2EhsbCybDnfw/7a28tMzo8iIt149fa2zHULZQNrgrb3t/KO4jd8tHkXSKEv9lj3Bo5+0sqmig89PCWdW0uC+e721Q+d3eUaijT1HDS/vaiPcBhek2QkPE2b28B1fe6Cdx7a3cc+ZUUwOsu9Vb+2wZMmSAmPMgt7Os9oso1ig+/TdemB0fyeKyNeABcBNHpvvBIpw3aK7GnhdROYZY/Z0P98Y8zDwMEBmZqbJzc0dSv1PsmnvEV7dcojx8VGMjgqnqqGFqmOt7K5uZMv+JjAQGe6wZOLDvLw8cnNzyXutkFHh+7nu80sID7PuHxJf6WyHUDaQNpg4q4F/FL9Pa+I0cs9I80/FBsnhNHxv7Wo+O2cCv79m/qDP760dum/5RnUjtz7zEa+VNgBgk3Y+d+p4Zp8ST0psJMmjI9n8STGjI52cOnc+2UE2GGeo3wm/Bhz3FcfiXnZ/AHwbiOu2PQ5o6KfcS4FfAxcYY2o6txtjNnoc9qSIXANcDDwwuJoPzfo9NSx/dCNOj4vIMJuQHBvhrp9rW2u7kw2lNZYLOJ3yy2qZnzYmJIONGripKbGcMmYUecVVXGvRgPNRuStnWX8ZBYYrIyWWz582nuKKBlf2aQP/2V7B69sOn3Ts8sc2WvIHpy/4NeAYY3L72u/uw7GLyHRjzC735rlAYR/nfBZ4BPicMeaT/qoA+KXX2+E0/Oy1wq5gYxO4JXcqt1+Yic0mXWtvtLjToZdUNmKMsVynfGNrB0WHjnHbedMDXRVlcSLC4swUXv34IG0dTktmPV5VVEl4mHRNVvWlM6cmExm+m/YOJ+F2G8/ceAaZ4+OoaWjl4fdLeW5TecitK2WpT4Qxpgl4GbhXRGJEZBFwCfB0T8eLyHm4BgpcZozZ1G3fGBFZJiJRImIXkeXAucDbvn0XrtxIv3ijiJLKRuw2IUxcHZXnZX26kl92egLP3JTDHUtncNGccby65RA/eGkbHe7051bxcXkdTqP509TA5M5IoanNQX5Zbf8H+5kxhlVFleRkJA0qq8BQdX7Hv7c003UFMzmR2Eg7k5NjuCx7IpHhtpBbV8pqfTgAtwKPA1XAEeAWY0whgIik4eqTmWWMKQfuBuKBf3tcGaw1xlwEhAP3AVmAA9gJXGqM8flcnEfWlvLEh/u48ewpXHzqeDaUHulx7Zjs9ASy0xMwxvDH1bv40zu7OHq8nQeumU9UuDU6ETfvq8MmMD9tTKCrooLAWdOSCQ8T3iupHtLsfV/aU93I3pomvr5ost9es/M73tP2Z27K6fVvw0hluYBjjKkFLu1lXzmugQWdz5f0UU41sNDrFezHq1sO8st/7+Rzp43nxxfPxGaTfj9MIsJ3L5xBYkwEP3utkC//vw+4cFYq584YG/APYv6+WmaOj/PLL0IV/GIj7SycnMh7xdX88KKZga7OCVa6J1laJWdZb8FoJLPULbVg98HuGu54cSs5GYn8/sq5XbfPBur6sybz3xdMp+hwA396ZzfLHx3crGVv63Aatuw/quls1KAsnpHCzooGDtcfD3RVTrC6qJI5p8QxYcyo/g9WPqEBx0v+WXCAr/1tM+PjRvHX6xYQaR/aLbHwMFvXqIbOtdMDZX+Dk+Y2h074VIOSmzkWgPcsNAm0uqGVj/cf5cKZoZFCxqo04HjB6h2V3P7iVtocTiobWthd1TjksnIykogMt53wPFBK6lwDGBbokgRqEGakxjI+Por3LLRcwTs7KjEGnw+HVn3TgDNMTqc5YRGmjmHmZursTDxnWjJOAwnRges72VXnYGLCKMbFRwWsDir4iLiGHb9XXM0DQ8ig7Auriio5ZcwoZo7vdw658iENOMP05Pp97Ktp6hr+7I0hjtnpCfzuqrnYbcIzG8u9U9FBMsZQUufU/hs1JJMSomlud/CHVSUB74tsbutg3e4aLpyVarl5bqFGA04P6lvNgL4gxRUN/Oo/OzkvayzPr/AYb++FkSdjR0exbM44Xio4QEu7/9cYKa9t5lib0f4bNSTt7vlkTvPpxMZAWburhtYOp95OswANOD2oazVc08+6Fi3tDr7zj4+Ji7Jz/+WnkT05kW8tmebVYY5fOSOd+uPtvL71kNfKHKgXCw4AEBNhuZHzKgicPT0Fu3uUpj0ssBMbVxVVEhflymatAksDTi/aOpw8ura01/33v1XMzooG/veKuSTH+mZp2JyMRKaNjeXvfr6tVlBWx0NrXPlN73p5myXuwavgkp2ewF+uyyZMhPOyAjefzOE0vLuziiVZYzUXoAXo/4Fe2MSVbO+nr26nrePEdDPvlVTz+Ad7ueGsySxxDwH1BRFh+RlpbN1/lO0HuyfR9p31e2q6FqgK9O0QFbwumJnKl08/hbziao42twWkDgVl/knWqQZGA04PEiKFf6zIYcW5GTy1voxrH9nQtZjSkcZW7nhxKzNSY7nroqGthT4YXz59IqPCw/j7hjKfv1anzqwCQmjleVLed9M5GRxvdwRs8MvqHa5knYtn+D5Zp+qfBpwexEcKn5mSxI8unskD18yn8NAxPv/AOp7dWMZVD2+grqmNP13tn3xn8aPC+eLcCby65RDHWtp9/noAe2uaCA8TLpkaHjJp05VvZI4bzTnTk3niw320dvh38EvBvlpezN/PLE3NZBkacPrxhbkTeOVbZ2ET+NEr27smdTa3+e/L85WcdI63O3jZ3ZHvS06n4a3tFSzJHMul0yM02Khhu/mcDKobWnl968lrwfhKQVkd1z66kbrmdgoPHdN+SIvQgDMAWePiuHLBpK7nxhi/9mucOjGeuRPj+fvGcny9JPi2g/VUHGvhs3M0BYjyjnOmJ5OZOppH15b6/PPbaUPpka6+V6efv6+qdxpwBmhx5liiArh+xfKcdHZXNbJxr2/XGXlrewV2m3B+lnayKu8QEW48Zwo7Kxr4YLd//vDnZCR1LbUYof2QlqEBZ4BOWkzJz7eavnDaBOKi7D4dPGCM4a3thzlzahLxAUypo0aeS+ZNIDk2kkf6mGrgTRnJMWDgzIxE7Ye0EA04g5CdnuD1yZ0DNSoijMuzJ/Gf7Ye5/62dPrknXVzZwL4jzVw0Z7zXy1ahLdIexvVnpvNeSTUllQ0+f728kioMcOdFMzXYWIgGnCAyLy0ehxMeytvjk/xUb22vQEQz6irfWJ6TTlS4jcfW7vX5a72zo4rk2EhOOyXe56+lBs5yAUdEEkXkFRFpEpEyEbm2j2NvEBGHiDR6PHKHUlYw2F/rWtDK4JsJmW9tr2BheiIpo32TOUGFtsSYCC47fSL//OiAz67SwZXH7b2Sas7LShn0IojKtywXcIAHgTYgFVgOPCQis/s4fr0xJtbjkTeMsiwtJyOJSLv7f5mIVztC99U0sbOigWU6Ok35UE5GIh1O47OrdIDN+2ppaOmwzFLS6lOWCjgiEgNcBtxtjGk0xqwDXgOuC2RZVpGdnsCzN+dw2sR4MIbEmAivlf12YQUAy2brl1T5TrmPr9IB3t1RRUSYjbOnJXu9bDU8VksFPANwGGNKPLZtBRb3cc58EakBaoGngV8ZYzoGW5aIrABWAKSkpJCXlzfkN+FrX5/u5M7D8P2n13LbfO8sjvb8+uNMjrOxe+smdru3NTY2Wrod/EXbwXttEHnUgV2gw7hGLUceLSMvz7sTml//qJkZCTY2r1/n1XJBPwudhtoOVgs4sUD3LJX1QG/L9L0PzAHKgNnA80AH8KvBlmWMeRh4GCAzM9Pk5uYOvvZ+VG7fxe9XlRA7+TQWDHORtMP1xyl9612+vyyT3NxpXdvz8vKwejv4g7aD99ogF5g7v5av/W0z01NjuelLi4ZdpqfS6kYq33qPb12YRe6Zk71aNuhnodNQ28Gvt9REJE9ETC+PdUAjENfttDigx3GUxphSY8xeY4zTGPMJcC9wuXv3oMoKNjedM4XUuEjue3PHsGdvryysBNDsAsovFk5O5GuLJrN1/1EOHT3u1bLf2VEFwHlZvsvirobOrwHHGJNrjJFeHmcDJYBdRKZ7nDYXKBzoS9A1v3jYZVladISd25dmsmX/Ud78ZHg5qt7aXsH0sbFMTYn1Uu2U6tsV2ZNwGnjJy/kB39lZSda40UxMiPZquco7LDVowBjTBLwM3CsiMSKyCLgEV9/MSUTkIhFJdf87C7gbeHUoZQWjy06fSNa40fzmrZ1DzsRb29TGxr1H9OpG+VVaUjSLpiXxQv5+nE7v5FerP97O5n11enVjYZYKOG63AqOAKuA54BZjTCGAiKS559qkuY89H9gmIk3Av3EFmF8OpKyRIMwm/PhzM9lfe5yn1w8t5c2qogqcBpbN1oCj/OuqhWkcqDvOh3u8M1LtvZJqHE7D+TM14FiV1QYNYIypBS7tZV85rsEAnc/vAO4YSlkjxTnTU1g8I4U/v7OLy7MnMiZ6cEOln9+8n/hRdlrb/btWiVJLZ6USPyqcf2wu5+zpwx/C/O6OShJjIpg3SVPZWJUVr3DUIP3o4pk0tHRw05P5g5pI99qWg3xUfpRjxztY/thGXTNE+VVUeBhfmn8KKwsrqWsa3hLUHQ4na4qryc1MIUyzC1iWBpwRoLG1A5sI+WV1XPnX9axyjzrrTXVDKz/51yd85/ktgG8n4SnVl6sWTqLN4eSVjw8Oq5yPyo9Sf7xdl9WwOA04I8CG0iMYXB2vDqfhG3/P53vPb6Ho0LETjjve5uCBd3aR+79reG7TfpbNGkekPXBr/Cg1c3wccyfG8/zm/cMa3v/OzkrsNuHcGZpdwMos14ejBi8nI4kIu432Dif2MBsXzBzLW4UVvPzxQRZNSyI3cywF++rYuPcIdc3tLJudyg8+m8XUlFgKyurYUHqEnIwkTeOuAuKqhWn86JVP2HqgnnmTxgypjHd2VHFGRiKjo3QdJyvTgDMCdC4O5xk46o+389ymch5+r7RrlUURuO/SOXwlJ/2EczXQqED6wtzx/OKNIp7fXD6kgFN2pIndVY1c+5m0/g9WAaW31EaI7ovDxY8K55uLp3LDosldM2FtuOYqKGUlo6PC+dxp43ltyyGaWjsGff4TH+4DIDVOl9WwOg04I9yiaclEhms/jbK2qxZOoqnNMeisGQVldTzpDji3v7hVR1panN5SG+F6ut2mlNUsSE8gIyWGx9btpbqhdcCf1afX76MzUUHnSEv9jFuXBpwQoP00yupEhEVTk3l6Qxm/qywmwm7jmZty+vzcbj9Yz78/cS2LbkOv4IOBBhyllCWMjnL9OXIaaOvnaqXqWAs3P5VPcmwE9146h+KKBr2CDwIacJRSlnD+zFQeXbeXtg4nTgMpsT2naWppd3Dz0wUcbW7npVvOZPaEeC7Q5aSDgg4aUEpZQnZ6As/dnMONZ08hZXQE97xexJriqhOOMcbw/Ze2se3AUf549TxmT4gPUG3VUGjAUUpZRnZ6And/fhZvfvscJifFcNOT+byYv79r/wPv7ub1rYf4wbIszXAehPSWmlLKcsbGRfH8N3K45e8f8f2XtvHx/qM0tnTw2tZDfPn0U/jm4oxAV1ENgQYcpZQljY4K5/EbFnLjk5t5dmM54MqWceWCiYhoRuhgpLfUlFKWFWG3ccaUxBOyZRSUHQ1kldQwaMBRSlnamVM1W8ZIYbmAIyKJIvKKiDSJSJmIXNvHsX9xLznd+WgVkQaP/Xki0uKxv9g/70Ip5S2d2TK+tzSz38mgytqs2IfzINAGpALzgDdFZKsxprD7gcaYbwLf7HwuIk8Azm6H3WaMedR31VVK+ZpmyxgZLHWFIyIxwGXA3caYRmPMOuA14LpBnPukb2uplFJqKCwVcIAZgMMYU+KxbSswewDnXgZUA+932/4rEakRkQ9EJNc71VRKKTVYVrulFgvUd9tWD4wewLnXA0+ZE9epvRMownWL7mrgdRGZZ4zZ0/1kEVkBrABISUkhLy9v8LUfYRobG7Ud0HYAbYNO2g4uQ24HY4zfHkAeYHp5rAPmA83dzrkdeL2fcicBHUBGP8e9BXy7v3rOmDHDKGPWrFkT6CpYgraDtkEnbQeX3toByDd9/G316xWOMSa3r/3ufhi7iEw3xuxyb54LnDRgoJuvAh8aY0r7qwKgM8aUUioALNWHY4xpAl4G7hWRGBFZBFwCPN3PqV8FnvDcICJjRGSZiESJiF1ElgPnAm/7oOpKKaX6YamA43YrMAqoAp4DbjHuIdEikuaeT5PWebCInAlMBF7sVk44cB+ugQQ1wLeBS40xOhdHKaUCwGqDBjDG1AKX9rKvHNfAAs9t64GYHo6tBhb6oo5KKaUGz4pXOEoppUYgDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8QgOOUkopv9CAo5RSyi804CillPILDThKKaX8wnIBR0RuE5F8EWkVkScGcPx3RaRCROpF5HERifTYlygir4hIk4iUici1Pq28UkqpXlku4ACHgPuAx/s7UESWAXcB5wOTgQzg5x6HPAi0AanAcuAhEZnt5foqpZQaAMsFHGPMy8aYfwFHBnD49cBjxphCY0wd8AvgBgARiQEuA+42xjQaY9YBrwHX+abmSiml+mIPdAWGaTbwqsfzrUCqiCQBaYDDGFPSbf/ingoSkRXACvfTVhHZ7oP6BptkoCbQlbAAbQdtg07aDi69tUN6XycFe8CJBeo9nnf+e3QP+zr3j+6pIGPMw8DDACKSb4xZ4N2qBh9tBxdtB22DTtoOLkNtB7/eUhORPBExvTzWDaHIRiDO43nnvxt62Ne5v2EIr6OUUmqY/BpwjDG5xhjp5XH2EIosBOZ6PJ8LVBpjjgAlgF1EpnfbXzj0d6CUUmqoLDdoQETsIhIFhAFhIhIlIr3d+nsKuFFEZolIAvAT4AkAY0wT8DJwr4jEiMgi4BLg6QFU4+Hhvo8RQtvBRdtB26CTtoPLkNpBjDHersiwiMg9wM+6bf65MeYeEUkDioBZxphy9/HfA+4ERgH/BL5pjGl170vENbz6Qlyj3u4yxjzrlzeilFLqBJYLOEoppUYmy91SU0opNTJpwFFKKeUXGnA8hGrutb7y14nI+SKyU0SaRWSNiPQ5sStYiUikiDzm/v/eICIfi8hFHvtDoh0AROTvInJYRI6JSImI3OSxL2TaAUBEpotIi4j83WNbyLSBeypLi4g0uh/FHvsG3Q4acE4UqrnXesxfJyLJuEb63Q0kAvnA836vnX/Ygf24MlHE43rPL4jI5BBrB4BfAZONMXHAF4H7RCQ7BNsBXH8TNnc+CdE2uM0YE+t+ZMLQ20EHDbi5c6/VAXM60+GIyNPAQWPMXQGtnJ+IyH3ARGPMDe7nK4AbjDFnuZ/H4EpnMd8YszNgFfUTEdmGKxlsEiHaDiKSCeQB3wHGEELtICJXA1/GNTJ2mjHmK6H2nRCRPODvxphHu20fUjvoFc6nZtBz7rVQuMLpzWxcbQB0zW3aQwi0iYik4vpMFBKC7SAi/09EmoGdwGHg34RQO4hIHHAvcHu3XSHTBh5+JSI1IvKBiOS6tw2pHTTgfGpQuddCREi2iYiEA88AT7p/rYVcOxhjbsX1/s7BdeukldBqh1/gykS/v9v2UGoDcM1xzABOwTXZ83URmcoQ20EDzqc099rJQq5NRMSGKxtFG3Cbe3PItQOAMcbhXtZjInALIdIOIjIPuAD4Qw+7Q6INOhljNhpjGowxrcaYJ4EPgIsZYjtowPmU5l472Qm56tz3aacyQttERAR4DNegkcuMMe3uXSHVDj2w8+n7DYV2yMW1oGO5iFQAdwCXichHhE4b9MYAwlDbwRijD/cD+AfwHBADLMJ1iTg70PXyw/u2A1G4Ric97f63HUhxt8Fl7m2/ATYEur4+bIe/ABuA2G7bQ6YdgLHA1bhumYQBy4AmXHkIQ6IdgGhgnMfjt8BL7vcfEm3gbocx7v//nX8Plrs/C5lDbYeAvykrPXAN7/uXu1HLgWsDXSc/ve973L9cPB/3uPddgKvj+Diu0UqTA11fH7VBuvt9t+C6XdD5WB5i7ZACvAccBY4BnwA3e+wPiXbo1ib34BqpFVJt4P4sbMZ1m+yo+8fYhcNpBx0WrZRSyi+0D0cppZRfaMBRSinlFxpwlFJK+YUGHKWUUn6hAUcppZRfaMBRSinlFxpwlFJK+YUGHKWChIjEicg9IjIz0HVRaig04CgVPBYAPwPCA10RpYZCA45SwWM+rmUCigJdEaWGQlPbKBUERGQHkNVt8z+NMZcHoj5KDYUGHKWCgIgsxJXNvBD4pXvzYWNMWeBqpdTg2ANdAaXUgGzFtRDaA8aYDYGujFJDoX04SgWH2UAE8FGgK6LUUGnAUSo4nI5rvZ4tga6IUkOlAUep4DAf2GOMORboiig1VBpwlAoOs9Dh0CrI6aABpYLDUeB0EVmGay35XcaYIwGuk1KDosOilQoCIjIHeAw4DYgCzjHGrAtsrZQaHA04Siml/EL7cJRSSvmFBhyllFJ+oQFHKaWUX2jAUUop5RcacJRSSvmFBhyllFJ+oQFHKaWUX2jAUUop5Rf/H2LR9o1zeQA0AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = model.predict(X_valid)\n",
    "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_oMiC2wAFpad",
    "outputId": "1f5e46f4-c29e-4c41-a8f3-ed96e59141b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 4s 614us/sample - loss: 0.0221 - val_loss: 0.0053\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 3s 471us/sample - loss: 0.0043 - val_loss: 0.0037\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 3s 475us/sample - loss: 0.0035 - val_loss: 0.0032\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 3s 484us/sample - loss: 0.0033 - val_loss: 0.0032\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 3s 465us/sample - loss: 0.0032 - val_loss: 0.0030\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 3s 468us/sample - loss: 0.0031 - val_loss: 0.0030\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 3s 468us/sample - loss: 0.0031 - val_loss: 0.0028\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 3s 467us/sample - loss: 0.0031 - val_loss: 0.0028\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 3s 464us/sample - loss: 0.0030 - val_loss: 0.0029\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 3s 479us/sample - loss: 0.0031 - val_loss: 0.0030\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 3s 461us/sample - loss: 0.0030 - val_loss: 0.0029\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 3s 454us/sample - loss: 0.0029 - val_loss: 0.0027\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 3s 452us/sample - loss: 0.0030 - val_loss: 0.0030\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 3s 464us/sample - loss: 0.0030 - val_loss: 0.0030\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 3s 471us/sample - loss: 0.0029 - val_loss: 0.0027\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 3s 468us/sample - loss: 0.0030 - val_loss: 0.0028\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 3s 462us/sample - loss: 0.0029 - val_loss: 0.0031\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 3s 469us/sample - loss: 0.0030 - val_loss: 0.0027\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 3s 456us/sample - loss: 0.0029 - val_loss: 0.0030\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 3s 477us/sample - loss: 0.0029 - val_loss: 0.0026\n"
     ]
    }
   ],
   "source": [
    "# Make the second SimpleRNN layer return only the last output:\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(20),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\")\n",
    "history = model.fit(X_train, y_train, epochs=20,\n",
    "                    validation_data=(X_valid, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q4u2lBOiFpad",
    "outputId": "8ccf8f02-1b11-4ef4-d8ab-1f18b82de79d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 137us/sample - loss: 0.0026\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.0025768042597919703"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0w0CERKyFpae",
    "outputId": "6638a5b3-b422-49a0-8c9a-453a115eb6a3"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXhU5fn/8fedlZCQEEACiSyyKWAFQcrmQn/gvn+x1boUqkhbv9Ziq1+xRUWtWrzAWhWtrSJqFax1r1arCG0RXLCiFqlUqqBE9i0Bss3cvz/OEJM4hEyYmQTyeV3XuTLznGeec5+ZybnnPGd5zN0RERGJt5SmDkBERA5MSjAiIpIQSjAiIpIQSjAiIpIQSjAiIpIQSjAiIpIQSjAiIpIQSU0wZtbOzJ4xsx1mtsrMzq+n7pVmttbMtpnZLDPLrDFvgZmVmVlpZPo4OWsgIiINlew9mJlABVAAXADcZ2b961YysxOBycBooDvQA7ixTrXL3T0nMh2a0KhFRCRmSUswZpYNjAWuc/dSd18IPA9cFKX6OOBBd1/m7luAm4HxyYpVRET2XVoSl9UHCLn7ihpl7wPHRanbH3iuTr0CM2vv7psiZbeZ2a+Aj4FfuPuCaAs1s4nARICsrKzBXbp02aeVCIfDpKQ0n0NXzSkexRJdc4oFmlc8iiW65hQLwIoVKza6+0Exv9DdkzIBxwBr65RdCiyIUnclcFKN5+mAA90jz4cCbYBMgr2dEqDn3mIYPHiw76v58+fvcxvx1JziUSzRNadY3JtXPIoluuYUi7s7sMQbsd1PZoosBXLrlOVGksPe6u5+XALg7m+5e4m7l7v7w8AbwClxjldERPZBMhPMCiDNzHrXKBsALItSd1lkXs166/yr7rG6HLC4RCkiInGRtATj7juAp4GbzCzbzEYCZwKPRqn+CHCJmfUzs3xgCjAbwMzamtmJZtbKzNLM7ALgWOCVpKyIiIg0SLKPIl0GZAHrgTnAj9x9mZl1jVzP0hXA3V8GbgfmA6si0w2RNtKBXwIbgI3Aj4Gz3F3XwoiINCPJPIsMd98MnBWlfDWQU6fsDuCOKHU3AEMSFaOIiMRHUhOMiDTe9u3bWb9+PZWVlfvUTl5eHsuXL49TVPtGsUSXzFjS09Pp2LEjubl1z8Had0owIvuB7du3s27dOoqKisjKysKs8ee0lJSU0KZNmzhG13iKJbpkxeLu7Nq1izVr1gDEPck0nyt5RGSP1q9fT1FREa1bt96n5CJSk5nRunVrioqKWL9+fdzbV4IR2Q9UVlaSlZXV1GHIASorK2ufu16jUYIR2U9oz0USJVHfLSUYERFJCCUYERFJCCUYEdkvnXfeeZxzzjkxvWbYsGFcddVVCYroK5MnT+aoo45K+HKaO52mLCIJsbd+/XHjxnH33Xc3uv37779/993WG+yll14iPT290cuU2CjBiEhCfPnll9WP//znP3PppZfWKtvTWXGVlZUNSgJ5eXkxx9SuXbuYXyONpy4ykRZk8WKYMSODxYsTv6xOnTpVT23btv1aWV5eHitWrMDMePLJJznuuONo1aoVDz/8MOvWrePcc8+tvvbn8MMP57HHHqvVft0usmHDhnHllVdy9dVX065dOzp16sTPf/7zWns5dbvIOnXqxLRp07j44ospLCykS5cu3HXXXbWW89FHHzFy5EhatWpFv379ePXVV0lLS2Pu3LkNfi9CoRDXX389Bx98MJmZmQwcOJCXXnqper67c91119G1a1cyMzPp06cPEyZMqJ4/b948vvnNb5KdnU3btm0ZNmwYH3/c/G+/qD0Ykf3QpEmwdGlsr9m2DT74AMLhDG6+GY44AmLZCRg4EO68M7ZlNtTkyZOZMWMGAwYMIDMzk127djFs2DCuvfZacnNz+ctf/sK4cePo1q0bRx999B7bmTVrFldffTVvvfUWb7/9Nt/73vcYMmQIZ5999h5fM336dG6++WauuOIKXnvtNX7yk59w9NFHM2jQIKqqqjjzzDPp2bMnb7/9Ntu3b+fKK68kHA7HtH633347d911F/fffz8DBw5k1qxZnHnmmXzwwQf07duXxx9/nJkzZzJnzhz69u3Lf//73+pbxZSXl3PWWWfx4x//mLlz51JeXs6SJUua1YiXe6IEI9JCbNsGwXbRCIeD543oZUqIn/70p5x1Vu374F555ZXVj//3f/+XV199lblz59abYAYNGsSUKVMA6N27N7/97W+ZN29evQnmtNNO44c//CElJSUMHDiQ3/zmN7z++usMGjSIF198kVWrVvHGG2/QsWNHAKZNm8bo0aNjWr/p06fz85//nHPPPbe6jb/97W/MmDGDBx54gFWrVlFUVMSYMWNITU0lPz+fUaNGAbBp0yZKS0s588wz6dGjBwB9+/aNaflNRQlGZD/UmD2JxYth9GioqHAyMozHHoPhw+MfW2PUPeOqqqqKW265hT/96U+sWbOGiooKysvLOfnkk+tt54gjjqj1vLCwcK+3QKnvNf/+97/p3r17dXIBGDp06F7Xp6b169ezefNmRo4cWav86KOPZtGiRUDQ3Tdz5kwOOeQQTjzxRI477jjOPfdc0tPTKSws5LzzzmPUqFGMGTOG0aNH8+1vf5uioqKY4mgKzX8fS0TiYvhwmDcPpkypYN685pNcALKzs2s9v+WWW5g5cybXXnst8+fPZ+nSpZxyyilUVFTU207dkwPMbK/dWfW9xt33+Sr33ceAorWzu6xHjx588skn3HPPPbRu3Zr/+7//Y+jQoZSVlQEwZ84c3njjDUaMGMFTTz1F7969mT9//j7FlQxKMCItyPDh8LOfVTSr5BLNwoULOfvsszn//PMZMGAAPXr0YMWKFUmPo2/fvnz22Wds2LChuuztt9+OqY2CggLat2/PwoULa5UvXLiQfv36VT/PysrijDPO4De/+Q2vvvoq7733Xq1lDRo0iGuvvZZ//OMfDB06lEcfjTYYcPOiLjIRaXb69OnDiy++yOLFi2nbti133HEHxcXFdOvWLalxnHrqqXTt2pVx48bxq1/9ipKSEiZPnoyZxbRnc9VVV3HrrbdyyCGHMGDAAGbNmsW7777LQw89BMDvf/970tLSGDJkCNnZ2Tz++ONkZGTQs2dPPv74Yx555BFOO+00CgsL+c9//sNHH33EqaeemqjVjhslGBFpdm688UY+//xzjj/+eLKzs5kwYQLnnHNO9bglyZKWlsZzzz3HhAkTGDJkCD169GD69OmcfvrptGrVqsHtXH311ezcuZNJkyaxYcMG+vbty7PPPlt9sL5t27ZMnz6dSZMmEQqFOOyww3juuecoKipizZo1LFu2jIceeohNmzbRuXNnJkyYUOskiGbL3VvMNHjwYN9X8+fP3+c24qk5xaNYootHLB999NG+BxKxffv2uLW1r/bHWN58800H/F//+leTxxJP9X3HgCXeiG2u9mBEROrx5JNPkp+fT69evVi5ciWTJk3im9/8Jv3792/q0Jo9JRgRkXps27aNa6+9li+++IL27dszevRoZsyY0dRh7ReUYERE6jFhwoRat22RhtNpyiIikhBKMCIikhBKMCIikhBKMCIikhBKMCIikhBKMCIikhBKMCLS7D3wwAPVo2JGex7Nr371K3r16hX3ZSfKlClTGDhwYMKXk0xKMCKSEKeffjpjxoyJOm/58uWYGa+//nqj2r7gggvifnflqqoqzIxnn3024ctqKZRgRCQhJkyYwOuvv85nn332tXkPPvgg3bp1qx61MVZZWVm1BgFLpGQu60CjBCPSkixeTMaMGcHwlgl26qmnUlBQUH1L+t0qKyt59NFHufjii6vHlb/qqqvo06cPWVlZHHLIIUyePJny8vI9th2t2+q2226joKCANm3aMH78eHbu3Flr/ltvvcXxxx9Phw4dyMvL45hjjqk13srhhx8OwNlnn42ZVXevRVvWvffeS8+ePcnIyKB3797MmjWret7uPaEHHniAsWPHkp2dTc+ePZkzZ05D3zoAwuEwN954IwcffDCZmZkcccQRvPDCC9Xz3Z2pU6fSrVs3MjMz6dy5M9///ver5y9YsIChQ4eSnZ1N27ZtGTp0KMuXL48phn2lW8WI7I8mTYKlS2N7zbZt8MEHZITDcPPNcMQRkJfX8NcPHBjTWM1paWmMGzeO2bNnc8MNN1QnkxdeeIGNGzfW2hjm5uYye/ZsCgsLWbZsGT/4wQ/IysrihhtuaNCyHn/8caZOnco999zDcccdx9y5c5k+fXqtPY+SkhLGjRvHXXfdBcDdd9/NySefzCeffEJ+fj4LFiygT58+PPTQQ5x00kmkpUXfPD755JNMmjSJO++8kzFjxvDSSy8xceJEOnfuXGtI5xtvvJFp06Yxbdo07r//fsaPH88xxxzDwQcf3KB1mjFjBnfccQe//e1vGTRoEI888ghnn302S5cu5fDDD+ePf/wjd955J3PmzKF///6sW7eOd955BwiS+JlnnskPf/hDHn/8cSorK1myZEn1Z5As2oMRaSm2bYNwGAMIh4PnCXbJJZewevVqXnvtteqyBx98kBNOOIEuXbpUl11//fWMGDGC7t27c+qppzJ58uSYfvHfeeedXHzxxVx66aX06dOH66+/nkGDBtWqM2bMGC688EL69u1L3759mTlzJikpKbzyyisAdOjQAQjGZunUqVP187qmT5/O+PHjueyyy+jTpw+TJk3ivPPOY9q0abXqjR8/nvPPP59evXpxyy23AHxtVMv6TJ8+nWuuuYbvfve7HHroodxyyy0MGzaM6dOnA7Bq1SoKCws5/vjj6dq1K0OGDOGyyy4DYMuWLWzfvp0zzjiDnj17cthhh3HhhRdy6KGHNnj58ZDUPRgzawc8CJwAbASudffH91D3SuAaIAt4CviRu5fXqdMb+BD4k7tfmMjYRZqVGPYkqi1eDKNH4xUVWEYGPPYYiR47uXfv3hx77LHMmjWLE044geLiYl555RWeeOKJWvWeeOIJ7rrrLlauXElpaSlVVVUx/dpevnw5l19+ea2y4cOH8+STT1Y/X7duHddddx0LFixg3bp1hEIhdu7cyerVq2Nap+XLl1dvyHc7+uijuf7662uVHXHEEdWPMzIy6NChA+vXr2/QMjZv3sz69esZOXLk15az+8SIc889l7vvvptDDjmEE088kZNOOokzzjiDjIwMOnbsyIUXXsiYMWMYPXo0o0eP5tvf/naD957iJdl7MDOBCqAAuAC4z8y+NqiCmZ0ITAZGA92BHsCNe2jvnUQFK3JAGT4c5s2jYsoUmDcv4clltwkTJvDss8+yefNmZs+eTbt27TjjjDOq5y9cuJALLriAU045hRdeeIH33nuPm266iYqKirjGceGFF/Lee+9x5513smjRIpYuXUphYWGjlhNtuOS6Zenp6V+bHw6HG9R+MMZX/cvp1q0bK1as4N577yUnJ4crr7ySIUOGVB97evTRR1m8eDFHH300zzzzDH369Km1J5kMSUswZpYNjAWuc/dSd18IPA9cFKX6OOBBd1/m7luAm4Hxddo7D9gKzEto4CIHkuHDqfjZz5KWXADOOeccWrVqxR/+8AdmzZrF9773vVob3zfeeINu3brxi1/8giFDhtC7d++oZ57Vp2/fvrz55pu1yuo+X7hwIVdccQWnnHIK/fv3Jzs7m7Vr11bPT01NJTU1lVAotNdl1e3qWrhwIf369Ysp5vq0b9+ejh077nU5WVlZnH766dx55528+eabfPDBB7XWe+DAgUyePJm///3vjBw5kkceeSRuMTZEMrvI+gAhd695Qvn7wHFR6vYHnqtTr8DM2rv7JjPLBW4i2MO5pL6FmtlEYCJAQUEBCxYsaPwaAKWlpfvcRjw1p3gUS3TxiCUvL4+SkpK4xBMKheLWVkOdc8453HDDDWzdupXzzjuvevmhUIguXbqwevVqZs2axeDBg3n11Veru9B21ysrK6v3+cSJE7n88svp378/w4cP55lnnuHdd9+lQ4cO1XV69erFQw89RN++fSkpKeG6664jMzOT8vJySkpKCIfDdOnShb/85S8MGDCAjIwM8vPzv7asyy+/nEsuuYS+ffsyatQoXnnlFebOncsTTzxBSUkJVVVVAOzatavW++zulJWV7fG9Ly8vJxwOU1JSQigU4oorrmDatGkUFhZyxBFHMGfOHN58801mzJhBSUkJjz76KACDBw+mdevWPPnkk6Snp9OpUyeWLl3Ko48+ysknn0xhYSErV67kww8/5Oijj97j8svKyuL/P9OYcZYbMwHHAGvrlF0KLIhSdyVwUo3n6YAD3SPPfwNcE3k8FfhDQ2IYPHhwQ4amrldzGuvdvXnFo1iii0cs9Y2XHqumGO/93XffdcBHjBgRNZarr77aO3To4Dk5OT527Fi/5557PDU1tbre73//e8/Ly9vjc3f3m2++2Q866CDPzs72Cy64wKdMmeI9e/asnv/Pf/7ThwwZ4pmZmd6zZ09/7LHH/NBDD/Wbb765OpZnnnnGe/Xq5WlpadWvjbasmTNneo8ePTw9Pd179erlDzzwQPW8yspKB/yZZ56p9ZqioiL/9a9/vcf36Be/+IUPGDCgOpZQKORTp071oqIiT09P92984xv+/PPPV9d/6qmnfOjQoZ6Xl+fZ2dk+ZMgQf/HFF93dvbi42M866yzv3LmzZ2RkeNeuXX3y5MleWVm5x+XX9x0DlnhjtvuNeVGjFgRHAjvrlP0MeCFK3feB79R43j6SYNoDA4FlQIYrwTSreBRLdEowe6ZYomuKWBKRYJLZRbYCSDOz3u7+n0jZgEiyqGtZZN4fa9Rb50H32EUEB/5XRw525QCpZtbP3QdFaUtERJpA0g7yu/sO4GngJjPLNrORwJnAo1GqPwJcYmb9zCwfmALMjsz7HdCTYE9mIPBb4EXgxMSugYiIxCLZpylfRnBdy3pgDsG1LcvMrKuZlZpZVwB3fxm4HZgPrIpMN0Tm7XT3tbsnoBQoc/cNSV4XERGpR1IvtHT3zcBZUcpXE3R11Sy7A7ijAW1OjVd8IiISP7pVjMh+wiMX34nEW6K+W0owIvuB9PR0du3a1dRhyAFq165dX7vzQDwowYjsBzp27MiaNWvYuXOn9mQkbtydnTt3smbNmoSMeaPb9YvsB3JzcwEoLi6msrJyn9oqKyujVatW8QhrnymW6JIZS3p6OgUFBdXfsXhSghHZT+Tm5sZlI7BgwQKOPPLIOES07xRLdM0pln2hLjIREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUkIJRgREUmIpCYYM2tnZs+Y2Q4zW2Vm59dT90ozW2tm28xslpll1pj3BzP70sy2m9kKM5uQnDUQEZGGSvYezEygAigALgDuM7P+dSuZ2YnAZGA00B3oAdxYo8ptQHd3zwXOAH5pZoMTG7qIiMQiaQnGzLKBscB17l7q7guB54GLolQfBzzo7svcfQtwMzB+98xIefnup5GpZyLjFxGR2Ji7J2dBZkcCi9w9q0bZVcBx7n56nbrvA7e6+xOR5x2ADUAHd98UKbuXIOlkAe8Bx7p7aZTlTgQmAhQUFAyeO3fuPq1HaWkpOTk5+9RGPDWneBRLdM0pFmhe8SiW6JpTLADf+ta33nX3o2J+obsnZQKOAdbWKbsUWBCl7krgpBrP0wn2UrrXqZcKHA1MAdL3FsPgwYN9X82fP3+f24in5hSPYomuOcXi3rziUSzRNadY3N2BJd6I7X4yj8GUArl1ynKBkgbU3f24Vl13D3nQ1XYw8KM4xSkiInGQzASzAkgzs941ygYAy6LUXRaZV7PeOo90j0WRho7BiIg0K0lLMO6+A3gauMnMss1sJHAm8GiU6o8Al5hZPzPLJ+gCmw1gZh3N7DwzyzGz1MgZZ98FXk/KioiISIMk+zTlywgOyq8H5gA/cvdlZtbVzErNrCuAu78M3A7MB1ZFphsibThBd9gXwBZgOjDJ3Z9L6pqIiEi90pK5MHffDJwVpXw1kFOn7A7gjih1NwDHJSpGERGJD90qRkREEkIJRkREEkIJRkREEmKfEoyZZZnZGDPrFq+ARETkwBBTgjGz2WZ2WeRxBvA28FfgYzM7OQHxiYjIfirWPZgTgTcjj88A2gCdgKmRSUREBIg9weQTXMMCcBLwlLuvB+YC/eIZmIiI7N9iTTBrgcPNLJVgb+a1SHkOUBnPwEREZP8W64WWs4AngGIgBMyLlA8F/h3HuEREZD8XU4Jx95vMbBnQFXjS3Ssis6qAafEOTkRE9l8x3yrG3Z+KUvZwfMIREZEDRaynKX/HzE6o8fx6M/vCzF4xs87xD09ERPZXsR7kn7r7gZkNAn4O3EUw4uSM+IUlIiL7u1i7yLoBH0cenw086+63m9lfgVfiGpmIiOzXYt2DKSO4uBJgNF+dprytRrmIiEjMezD/AGaY2ULgKOCcSHkf4PN4BiYiIvu3WPdgLgcqCBLLD929OFJ+MuoiExGRGmK9DuYL4PQo5ZPiFpGIiBwQGjVkspn9P4J7jznwkbvPj2tUIiKy34spwZhZEfAMMJjgdjEAhWa2BDi7RpeZiIi0cLEeg7mL4B5kvdy9i7t3AXpHyu6Kd3AiIrL/irWL7HhglLt/urvA3f9rZlfw1Y0vRURE9m3I5BrCcWpHREQOELEmmHnAXWbWZXeBmXUFfgO8Hs/ARERk/xZrgrkCaA3818xWmdlnwEogC/hxnGMTEZH9WKzXwXwODDKz44HDAAM+Aj4B7gC+E/cIRURkv9So62Dc/VXg1d3PzWwAMDZeQYmIyP4vXgf5RUREalGCERGRhFCCERGRhGjQMRgze34vVXLjEIuIiBxAGnqQf1MD5n+6lzoiItKCNCjBuPv3Ex2IiIgcWHQMRkREEiKpCcbM2pnZM2a2I3IngPPrqXulma01s21mNsvMMiPlmWb2YOT1JWb2npmdnLy1EBGRhkj2HsxMgiGXC4ALgPvMrH/dSmZ2IjAZGA10B3oAN0ZmpwGfA8cBecB1wB/NrHtiQxcRkVgkLcGYWTbB1f7XuXupuy8EngcuilJ9HPCguy9z9y3AzcB4AHff4e5T3f0zdw+7+58JTjAYnJQVERGRBjF3T86CzI4EFrl7Vo2yq4Dj3P30OnXfB2519ycizzsAG4AO7r6pTt0CYBUw0N3/HWW5E4GJAAUFBYPnzp27T+tRWlpKTk7OPrURT80pHsUSXXOKBZpXPIoluuYUC8C3vvWtd939qJhf6O5JmYBjgLV1yi4FFkSpuxI4qcbzdMCB7nXqpQOvAfc3JIbBgwf7vpo/f/4+txFPzSkexRJdc4rFvXnFo1iia06xuLsDS7wR2/1kHoMp5esXZOYCJQ2ou/txdV0zSwEeJTimc3n8whQRkXhIZoJZAaSZWe8aZQOAZVHqLovMq1lvnUe6x8zMgAcJThYY6+6ViQlZREQaK2kJxt13AE8DN5lZtpmNBM4k2Aup6xHgEjPrZ2b5wBRgdo359wF9gdPdfVdiIxcRkcZI9mnKlxGMfrkemAP8yN2XmVlXMyuNDL+Mu78M3A7MJziAvwq4AcDMugE/AAYCayOvKzWzC5K8LiIiUo9GDTjWWO6+GTgrSvlqIKdO2R0Eo2TWrbuKYCRNERFpxnSrGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSQglGBERSYgWlWDKypo6AhGRlqNFJZgdO5o6AhGRlqNFJZjS0qaOQESk5VCCERGRhGhRCaasDLZsaeooRERahhaVYAAWL27qCEREWoYWl2AWLWrqCEREWoYWlWBat4Y33mjqKEREWoYWlWBycuDtt6GysqkjERE58LW4BLNzJ3zwQVNHIiJy4GtRCSY7O/irbjIRkcRrUQkmIwO6dNGBfhGRZGhRCQZg5EjtwYiIJENSE4yZtTOzZ8xsh5mtMrPz66l7pZmtNbNtZjbLzDJrzLvczJaYWbmZzY4lhhEj4Isv4PPP92FFRERkr5K9BzMTqAAKgAuA+8ysf91KZnYiMBkYDXQHegA31qhSDPwSmBVrACNGBH/VTSYiklhJSzBmlg2MBa5z91J3Xwg8D1wUpfo44EF3X+buW4CbgfG7Z7r70+7+LLAp1jgGDND1MCIiyWDunpwFmR0JLHL3rBplVwHHufvpdeq+D9zq7k9EnncANgAd3H1TjXq/BA529/H1LHciMBGgoKBg8Ny5c/npTwewY0ca99//bszrUVpaSk5OTsyvS5TmFI9iia45xQLNKx7FEl1zigXgW9/61rvuflTML3T3pEzAMcDaOmWXAgui1F0JnFTjeTrgQPc69X4JzG5oDIMHD3Z39ylT3FNT3UtKPGbz58+P/UUJ1JziUSzRNadY3JtXPIoluuYUi7s7sMQbsd1P5jGYUiC3TlkuUNKAursfR6sbsxEjIBSCd96JR2siIhJNMhPMCiDNzHrXKBsALItSd1lkXs1667xG99i+GDYs+KvjMCIiiZO0BOPuO4CngZvMLNvMRgJnAo9Gqf4IcImZ9TOzfGAKMHv3TDNLM7NWQCqQamatzCytobHk50P//jqTTEQkkZJ9mvJlQBawHpgD/Mjdl5lZVzMrNbOuAO7+MnA7MB9YFZluqNHOFGAXwanMF0YeT4klkBEjgrFhwuF9XCMREYkqqQnG3Te7+1nunu3uXd398Uj5anfPcffVNere4e4F7p7r7t939/Ia86a6u9WZpsYSy8iRsHUrLF8et9UTEZEaWtytYnbTBZciIonVYhNMr15w0EE60C8ikigtNsGYBXsx2oMREUmMFptgIEgw//kPbNjQ1DMQxJcAABWPSURBVJGIiBx4WnSCGTky+Ku9GBGR+GvRCWbw4GAQMiUYEZH4a9EJplWrIMnoQL+ISPy16AQDwXGYJUugvHzvdUVEpOGUYEYEyeW995o6EhGRA4sSTOSCS3WTiYjEV4tPMJ06QY8eOtAvIhJvLT7BQHC68htvQJIG9xQRaRFaVoL58svgFsp1jBgB69bBp582QUwiIgeolpVgiovhuOPg5ZdrFeuCSxGR+GtZCQagshJOOw3Gj4e33gJ3+vWD3Fwd6BcRiaeWl2AyM+GMM+Cpp4KxkwcPJvWhBzjuqB3agxERiaOWlWCKimD+fHj66aC77N57oaoKLr2UPy4q4pIPruDuy5ZHO0wjIiIxalkJplMnGD48eNymDfzoR/D++7BwIZ/1P5UfcD8/vq8fFSNHsWjSHwmXVTRtvCIi+7GWlWCiMYORI3ni9Mfowudcw6/o6qsY8Ztz2dC6Kw93mcL141fz0EOwdClUVlpTRywisl9Qgok44QQozerIjNRr6J/xCbO//SLruw3hoi9u5YaHD6HdxWfy8JG/5m8nLmT8oYu55BK4557gxIDS0q/aWbwYbrst6tnQIiItSlpTB9BcDB8O8+bBggUwalQqw4efApwCn32G3/87Tp15H2eWPI87+IoUXvv0JN6ZNZDHKGI6haR2KSS1axHPv1VARTiN9HT43e/gmGOC3rjc3GBogIZavHh3LF/16omI7E+UYGoYPjzKxrx7d+y2W0lrnQVTp2LhMEaY47MWcnz4FSwUCup9HkwhUlhHAcXlhawZV8TLFFJMIWsoYkN6Idtzijgkcw2D+SfLO32L1UXDyc2lemrTBjZtCs4/CIUgLQ2mTYNBgyArKxhioFWr4HHOh4vp/MQcQmmZpB799SwUrySlZLdnem+kFn0halGCaagxY+C22wiXl5OSmYm9/DJ885vBeMvFxbBmDSsXFjN3RjGdwmsosmKGFawir2QRrUo3Bm1UAltqtLkWQkuNEGmESCVMCiFSCZHKdZHH4YoUQlem1pofJoUqKsjlc/Jwwg8/wHyO5Yu0Q9iRnkdZei7bLY/Pt+WylTz+brm075FHSn4eZRm57EzPozIjm5RUo//2xRyxZQEfHTSKTw4aTkoKpKYGU0pKkOxeeQXC4aBs7Fjo1g3S04MpIyP42+WLxWS/8wLPH5vJlsOG15q3+/GKFcE5FQMHwuGHf/U2WJ3DWvU9/9e/4J//DBLugAHBvJpTSkrw95NPsmnXrnZZzen994PLoIYMCdoJh78+ZS1dTJt3F7B14Ci29x9ea557EMv//V9waVV6OsycGYwvtHt9MzIg+4PFdPzjXHZUBT8C0tOD97GmBm2TGlApHtu2hMbiHrx5oVAwvfEGXefMCS4daOp1amw7oRBs3QobN8Lf/gY//nFwZmpGBrzwQrDdSEA8zek7Ux/zFnQDrqOOOsqXLFnS+AYWL+a/s2bR4+KLY/vAysuD29QUFwdboTlzgn82s6DisGHBFzUcJlQZ4ovPQrz05zCEQ6RbiJHDw+TlhAhXhghVhglXhchb/S/y13yIAQ6Utu5IKCWdzPLtZFWW7HVVQqRQltKarPAODMcxvkzrQqVlYoQxd4wwhMLgYQwnhTApfP1xOhXksAMAx/iMbmzkIErJYQfZMf3tzX84ind4i6G8xyAci3kazLuM5A3eZRD/pi9Z7KIVZWSxK+oUbd7BfM7/43VSCOOk8C6D2EEOqYRIoyryM+Crx9H+tmIX7Wr8othAB0ppQyXpVFk6VZZBJemUhdOpIHicmplOOC2DqpR0QpZOVUoGeaHNHFvyIimECJPKUwU/4tPsb1CWlkNZehvK0tuwsSyHJR+3YZu3YYe1oc/A1uTkplQnw92Jsf/2xfTb8DIftjuJD7KDpEkoRHblVtJKt7Ljiy20ZQv5bKVnuy10SNtKXngLuaEttAlvpajiUw7ftQQjjGNsSOtMKCWdFA+R5lWkECLFgwSy+z1KtypSPPhpFI0DGzM6symjiO3p7die3p7t6e3ZGGrHsnXt2ejt2WrtyO/VHs9vx9bU9pSm5uGWgjv027aYI7ct4J95o/hXzvDqewq6w/bt8PHHwbqnpMBhhwU9BXV/dBxesph+a1/ki4OGsy7rEPIqN5JXtYm8qo3kVW4is3QTVWs30o5NdGAjRZmbyA9tJKdqCynseRu6MyWbDemFrE8rZENaZ4q9kI+3d2YNhXxJIZndO7Mrv5DyjDakpQU/Pr5RupjD1r7Eyi6nsKL98OoffGlpkOU7Ca3byIpFG8kPb6SjbWTkoRvplLaRnPKNtCkL/rbfsYpOO/5b/T+5vN3RrM49nE2tCtmUUcjmzM58Wl7IKx8WstHbk5mVwrx5e04yZvauux+11w1L3dcpwcRmwYIFjBo1qvENLF4Mo0dDRUXwK2cPn+pef1lE2tm9R1WrnVCIt18v5cLTt5FVuZ32aduYedt2+hZuC/7jtkX+zptX+2yEb3wj2LXY/d+XksL6jSm8/IpRGU7BUlI49TSjoHMKpKQQthTCYYN/vkvq24sxDxJVRe9+VHbqgpWWYjt3YDtLKdu0g5SdpWSzgzRCjX//EsTNCGVkEcrIIpzRipSKXWSUbK5O4GUdu1JW0A1SU/HUYEtQsjONDz5KpTKcRjgllf4D0sjJTaUqskeau/pDDvr0neoEvrZoEOs79MMqK7GqCqyqkpJNlezcVkE6laRTSW5WJW0yKkgNV0amCrIrtpAVKiWW8xfDGGWp2exMbUNZag4709pghOlR8kEkaRpbMzrSKryT1lX1/yCpsjR2ZORTmp5PRtVOOpR9gUWW8VnuEXyaO4AwqYQsWO8t21PZtC1IL1Wkkds2ldz8yF64pVJlaXxjy985avNfScEJY6zMPoLNGZ1pU7WZ3MpN5FVtok3V1j3GFCKF0vR8yqw1B1WsqU54xVm9qEhtRaoHiS5UGfww2530M1JDpKcE84KkFyI1XEm6V9T7/u6yLDZ4BzbRno10YFfr9lTltWd7ege2p7enJKM9+VXrmbDyWlLDlYQtjZe7Xko4JZ125V/SrryYdmXFtN1ZTJbv+lr7O1Nz2JzRmZ0p2fTc8WFkfVJY3noQqV5F26qNtK3aGPW1uz/vrSnt2ZLagS1pHWgb2kyPiuWR9xc2pRWQ5lXkhzZ97bUVpPMlnUnrUkjRUZ2hsLD2tH49B1900Zov3A+u94sShRJMjPY5wUBc99nr26NqaJJKaLKru6hyp01GOX99ZgdD+pbCjh3BaXg7dsBDD8Fjj331c3PsWDj99OCnqDuf/MeZfnuYUJWTnub87KdOzx5ePR/34D5zL7wQPE5Jge98B7773a8OXGVlsfTjLM6/JIvtlUFCeeblLIYdm1G7Ly4e700s70t9i6lb6bnnoF8/KCkJptJS/v1OCdOmlJBVVUJeaimXfa+ELnnBvOp6y5bBZ5991e6gQcFZKPn50LYt/9mUz7XT8tlY1ZbS9Hzuf6Itg8fkQ+vWX703DQg4lnWq971ZGOI7x2+hTcUmOqZt5r5bNtH3oE2weXPQd7tpU3Aa54cffvWiww4LpsjP/o1bUvnr66lUhNMgJZWTT0uloDCVWrsF77wD//hH7e/MhAnQvj106ADt27N4aVZDvg57/WdZvMg5e/R2OlQU0yXtS+6aXEzvnEjvRnExvP127c+oS5egD7dDh+pp5bYOXDujA2urOrAtvQMPPNuBIce3rd33uqcPoawM1q6F4mI+XvAl908tpqCqmKKUYk4bVEzbHcVBb8uWLbXiPgpY4h77NRru3mKmwYMH+76aP3/+PrcRT/scz6JF7rfeGvzdx3ZWTphQbzt7XdSiRe5ZWe6pqcHfKBUb2kYoJWWPbTSonQZX2ot4vC8NrBSP9yZpsUQqHXDfmQaot50kf3/3WGXnTveVK90nTnRPSfHB4N6IbW6Tb/STOSnBJFZcYknSRj2ZmtNndEC+Nwfad6aZxeJZWT4Iwt6Iba7OIpPmJeq54rG3sbq8nB46TfTrDsT35kD7zjSzWJg3j3UjRhQ35uW6kl9ERPZs+HDWwNrGvFQJRkREEkIJRkREEkIJRkREEkIJRkREEiKpCcbM2pnZM2a2w8xWmdn59dS90szWmtk2M5tlZpmNaUdERJpGsvdgZgIVQAFwAXCfmfWvW8nMTgQmA6OB7kAP4MZY2xERkaaTtARjZtnAWOA6dy9194XA88BFUaqPAx5092XuvgW4GRjfiHZERKSJJPNCyz5AyN1X1Ch7HzguSt3+wHN16hWYWXugawztYGYTgYmRp6Vm9nEj49+tA7BxH9uIp+YUj2KJrjnFAs0rHsUSXXOKBeDQxrwomQkmB9hWp2wb0KYBdXc/bhNjO7j774DfxRrsnpjZEm/EbasTpTnFo1iia06xQPOKR7FE15xigSCexrwumcdgSoHcOmW5QLR7hdetu/txSYztiIhIE0lmglkBpJlZ7xplA4BlUeoui8yrWW+du2+KsR0REWkiSUsw7r4DeBq4ycyyzWwkcCbwaJTqjwCXmFk/M8sHpgCzG9FOIsStuy1OmlM8iiW65hQLNK94FEt0zSkWaGQ8SR1wzMzaAbOA44FNwGR3f9zMugIfAf3cfXWk7k+Ba4As4Cngh+5eXl87SVsRERHZqxY1oqWIiCSPbhUjIiIJoQQjIiIJoQTTQGaWaWYPRu59VmJm75nZyU0cU28zKzOzPzRlHJFYzjOz5ZH7w600s2OaKI7uZvaSmW2J3MvuHjNLyvVeZna5mS0xs3Izm11n3mgz+7eZ7TSz+WbWrSliMbNhZvaqmW02sw1m9qSZdU5kLPXFU6fODWbmZjamqWIxs9Zmdq+ZbYzcB/HvTRjLdyL/UyVm9pGZnZXgWOrdxjXmO6wE03BpwOcEdwzIA64D/mhm3ZswppnAO024fADM7HhgGvB9ggtejwX+20Th3AusBzoDAwk+r8uStOxi4JcEJ6BUM7MOBGc+Xge0A5YATzRFLEA+wRlB3YFuBNePPZTgWOqLBwAz6wmcA3zZxLH8juAz6hv5e2VTxGJmRcAfgJ8SXOd3NfC4mXVMYCx73MY19juczCv592uR06On1ij6s5l9CgwGPkt2PGZ2HrAVWAT0Svby67gRuMnd34w8X9OEsRwC3OPuZcBaM3uZ4NZDCefuTwOY2VHAwTVm/Q+wzN2fjMyfCmw0s8Pc/d/JjMXd/1KznpndA/wtETE0JJ4a7iE4a/TeporFzA4FzgAOdvftkeJ3myKWyOOtNT6vF81sB9CT4AdUImKpbxvXnkZ8h7UH00hmVkBwf7WkX+BpZrnATcDPkr3sKLGkAkcBB5nZJ2b2RaRbKquJQvoNcF6kq6MIOBl4uYli2a0/wf3ygOp/5JUkKfHtxbE08UXKZvZtoMLdX2rKOIChwCrgxkgX2YdmNraJYlkCLDezM8wsNdI9Vg58kKwA6mzjGvUdVoJpBDNLBx4DHk7UL9C9uJngbtOfN8Gy6yoA0gm6N44h6JY6kuDi2KbwN4Iv/XbgC4J/1GebKJbdYrp/XrKY2RHA9QTdL00VQw5wKzCpqWKo4WDgcILPphC4HHjYzPomOxB3DxFccP44QWJ5HPhBZMOecFG2cY36DivBxMjMUgjuGlBB8AVM9vIHAmOAXyd72XuwK/L3bnf/0t03AncApyQ7kMhn8wpBX3E2wR1p8wmODzWlZnf/PDPrBfwF+Im7/6Op4iDoXn3U3T9twhh22wVUAr909wp3/xswHzgh2YFETnS4HRgFZBAcF3kg8v+f6GVH28Y16jusBBMDMzPgQYJf7WPdvbIJwhhFcIB2tZmtBa4CxprZP5sgFiLj9XwBNIcrdtsBXQiOwZRH7l33EE2Q7OqodW89C8Y06kkTdU1Fzv55DbjZ3ZN1i6U9GQ1cETnjby3B5/dHM7umCWJJWvdTAwwE/u7uS9w97O7vAG8R/LhMmHq2cY36DivBxOY+grNLTnf3XXurnCC/I/hgB0am3wIvAic2UTwQbMR/bGYdLbh33CTgz8kOIrL39CnwIzNLM7O2BIPXvV//K+MjssxWQCqQamatLDhF+hngcDMbG5l/PfBBIrtX9xRL5LjU68BMd/9topbf0HgIEszhfPV9LgZ+QHCGZLJj+TuwGrg2UmckwQ+6V5oglneAY3bvsZjZkQRd0IlOgnvaxjXuO+zumhowEZzS6UAZwe7i7umCJo5rKvCHJo4hneDsn63AWuAuoFUTxTIQWABsIRiw6UmgYxI/C68zTY3MGwP8m6AbZgHQvSliAW6IPK75HS5tyvemTr3PgDFN+Dn1BxYDOwjuj3h2E8ZyOfAJQTfUf4GfJTiWerdxjfkO615kIiKSEOoiExGRhFCCERGRhFCCERGRhFCCERGRhFCCERGRhFCCERGRhFCCEdmPWDBeyjlNHYdIQyjBiDSQmc2ObODrTm/u/dUiLY/GgxGJzWvARXXKKpoiEJHmTnswIrEpd/e1dabNUN19dbmZvRgZVnaVmV1Y88Vm9g0ze83MdlkwbPFsM8urU2dcZCyScjNbZ18fYridBUMd7zCz/0ZZxvWRZZdHbiL5SCLeCJG9UYIRia8bgecJ7on2O+CRyGiFmFlrgsHPSoFvAmcDI6gxXK6Z/QC4n+AGokcQ3Am67h1rrweeI7i77RPArMgdkokMkHUVwTDRvYHTgLcTsJ4ie6V7kYk0UGRP4kKCmwHWNNPdrzEzBx5w90trvOY1YK27X2hmlwLTCYbkLYnMH0Uw5khvd//EzL4guHnp5D3E4MCv3P3ayPM0gsHVJrr7H8zspwR3Iz7cm2Y4CZFqOgYjEpu/AxPrlG2t8XhxnXmLgVMjj/sS3OK85iBNi4Aw0M/MtgNFwLy9xFB9y3Z3rzKzDUDHSNGTwE+AT83sFYI9pufdvXwvbYrEnbrIRGKz090/qTNtbOBrjT0PzOaR+Q1Rd8/EifwvezCM9qEEezHbgRnAu5EBokSSSglGJL6GRXm+PPL4I2CAmdUcx3wEwf/hcndfB6whGISr0dy9zN1fdPcrgSEEY5yM3Jc2RRpDXWQisck0s051ykLuviHy+H/M7B2CAZnOIUgWQyPzHiM4CeARM7seyCc4oP+0u38SqXML8GszW0cwUmlrYLS7z2hIcGY2nuD/+i2CkwnOJdjj+U+M6ymyz5RgRGIzBviyTtka4ODI46nAWIJRPTcA3/dgPHXcfaeZnQjcSXBmVxnB2WA/2d2Qu99nZhXAz4BpwGbgpRji2wpcQ3AyQTrBXtP/uPunMbQhEhc6i0wkTiJneH3b3f/U1LGINAc6BiMiIgmhBCMiIgmhLjIREUkI7cGIiEhCKMGIiEhCKMGIiEhCKMGIiEhCKMGIiEhC/H9+prBE7Rui3wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4XnwpdFSFpae",
    "outputId": "230faef9-4082-4eee-c865-0a99298351e9"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEUCAYAAAAfooCMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deXzU1bn48c8zmSyQkJCNsCYhLAmLAgZqEJWACtUu0qp1QatWpdXa29tqq+2trbXeLt7uXn+2bnWpu9W69SqgREHZEiVIAgkQSNiykRCSQLaZ8/tjJmkI2TPLdzLP+/Wal8x3OXPmOJNnvt9zznPEGINSSinlbTZ/V0AppVRw0ICjlFLKJzTgKKWU8gkNOEoppXxCA45SSimf0ICjlFLKJzTgKKWU8glLBhwRuV1EckWkWUSe7OPY74lIuYjUicgTIhLeaV+ciLwmIo0iUioi13i98koppbplyYADHAbuB57o7SARWQ7cDVwApAJpwM87HfIQ0AIkASuBh0Vklhfqq5RSqg9i5UwDInI/MNEYc0MP+58D9htjfux+fgHwrDFmrIhEArXAbGNMsXv/M8AhY8zdPnkDSimlOtj9XYEhmgW83ul5PpAkIvFAMuBoDzad9i/uriARWQWsAoiIiMhMTk72To0DiNPpxGaz6kWw72g7aBu003Zw6akdiouLq40xiT2dF+gBJwqo6/S8/d+jutnXvn9UdwUZYx4BHgFIT083RUVFnq1pAMrJySE7O9vf1fA7bQdtg3baDi49tYOIlPZ2XqCH6gYgutPz9n/Xd7OvfX+9D+qllFKqi0APOAXAnE7P5wAVxpijQDFgF5FpXfYX+LB+Siml3CwZcETELiIRQAgQIiIRItLd7b+ngZtEZKaIxAI/AZ4EMMY0Aq8C94lIpIgsAi4FnvHJm1BKKXUKSwYcXIHjJK4hz9e6//0TEUkWkQYRSQYwxrwDPACsA0rdj591Kuc2YARQCTwP3GqM0SscpZTyA0sOGjDG3Avc28PuqC7H/h74fQ/l1AArPFk3pZRSg2PVKxyllFLDjAYcpZRSPqEBRymllE9owFFKKeUTGnCUUkr5hAYcpZRSPqEBRymllE9owFFKKeUTGnCUUkr5hAYcpZRSPqEBRymllE9owFFKKeUTGnCUUkr5hAYcpZRSPqEBRymllE9owFFKKeUTGnCUUkr5hGUDjojEichrItIoIqUick0Px/3Fvex0+6NZROo77c8RkaZO+4t89y6UUkq1s+QS024PAS1AEjAXeFtE8o0xBZ0PMsZ8C/hW+3MReRJwdinrdmPMY96trlJKqd5Y8gpHRCKBy4B7jDENxpgNwBvAdf087ynv11IppdRAWDLgANMBhzGmuNO2fGBWH+ddBlQBH3bZ/isRqRaRj0Qk23PVVEop1V9WvaUWBdR12VYHjOrjvOuBp40xptO2u4BCXLfnrgLeFJG5xpi9nU8UkVXAKoDExERycnIGX/thoqGhQduB7tthT62DwqMOZsaHMDU2xD8V8yH9LLhoO7gMth3k1L/N1iAi84CPjDEjO227A8g2xnyph3MmAfuA6caYkl7Kfgd42xjzYE/HpKenm6IiHVuQk5NDdna2v6vhd13bIa+0lqse2UirwxBut/HcLVlkpsT6r4I+oJ8FF20Hl57aQUTyjDHzezrPqrfUigG7iEzrtG0OUNDD8QBfBz7uLdi4GUCGWD8VxD4srqLV4fqh1tzmZFPJUT/XSKnAYMmAY4xpBF4F7hORSBFZBFwKPNPLaV8Hnuy8QURGi8hyEYkQEbuIrATOB971UtVVEKg83nTK84hQS36NlLIcK39TbgNGAJXA88CtxpgCEUl2z6dJbj9QRBYCE4GXu5QRCtyPayBBNfAdYIUxRu+XqUE5dqKFt7YfYUFqLN+/aBpjo8N5fssBWh1dR+Irpbqy6qABjDE1wIputpfhGlTQedtGILKbY6uABd6qowo+f/mghIaWNn6xYjYZY6OZOS6Gm5/O5ZmNpXzj3Mn+rp5SlmblKxylLKXyeBNPfryPS+eMJ2NsNAAXzBjD+dMT+cPaYo42NPu5hkpZmwYcpfrpf9ftoc1h+M8Lp3dsExF++sUZnGxx8NvVxb2crZTSgKNUPxyoOcHzW8r42oJJpCacevd26phRfH1hKi9sLaPgcNfpY0qpdhpwlOqHP67djYjwH0undbv/uxdOI3ZkGD9/oxArzm1Tygo04CjVh0MNTl779CDXL0xhbExEt8fEjAjlzmXpbNlfw1vbj/i4hkoFBg04SvXhtd0tjAgN4dbsqb0ed+WCScwaH83P3yjgT2uLySut9VENlQoMGnCU6sVLW8vIrXDwhTPHERcZ1uuxITbh6gXJVDe28Ie1u1n52CYNOkp1ogFHqR7kldZy96ufAfDGtsP9Ch51Ta0deZOaWp1s2F3lxRoqFVg04CjVgzWF5Tjd/f+tjv7lTMtKiyc81NYRdN7MP3xaKhylgpUGHKV64g42AoTabWSlxfd5SmZKLM/enMWdy9O5++IMDtc1seKhjyg8fNy7dVVqELbsO8pD6/b47NavZVPbKOVve6oaSYgKY/E4wzUXLuj3EgSZKbEdx543LYGbnszl8r98zJ+vmseFM5O8WWWlTpNXWsumkqNkpcUzdUwUW/bV8PHeat4rrKSs9gQAYSE2nr/lbDJT47xaFw04SnWjqdXBR3uquTxzIheMrh70ejezxsfw+u2LuPmpXG5+Opcl6YncvnTasF8/R1lDXmkt1zy6iZY2p+tS3bgu3CNCbYyNjmjfRIvDyW3PfcJPvziLz88eS4jNOyu4aMBRqhub99VwstXB0owxUF49pLKSoiP40cUZXPv4ZtYVVfHR3qM8HwSLtin/21RylOY2dyZzAwunxPPdC6YxL3k0Ow4dZ+Vjm2htc2KzCQJ8+7lPSI0fyfJZYxkZFsK50xI9+jnVgKNUN9btqiQi1MbCKfFsKh96eZ8eONbx71b3om0acJS3TUl0pWESIDzUxp3L0js+d+39je232+ZOGs27BeX8bnURf/3QtY7lwx/s5dmbPffjSAOOUl0YY3h/VyXnTEkgIjTEI2VmpcUTZrfR1OpEhH4NQFBqqCqOuzKY33TuZC4+Y9xpgaNzfyPAJWeMY191A79bXYzTeP7HkY5SU6qLvVWNlNWcYEnGGI+V2f5rMn3sKOIiw/TqRvnEmsIK0hIj+ckXZ/b7M5eVlkCY3UaI9H90Zn9pwFGqi3W7KgFc/TcelJkSyxWZE6luaNG5Ocrrjje1sqnkKBcNcGRk+4+j7y9L9+jtNNBbakqdZl1RJelJo5gweoTHy56XPBpw9eksnzXW4+Ur1e6DoiranIaLZgx8KH7XW22eYtkrHBGJE5HXRKRRREpF5JoejrtBRBwi0tDpkT3QcpQCqG9qZcu+GrIzEr1S/qzxMYSGCNs6DSJQyhvW7qwgPjKMecnWuX1r5Such4AWIAmYC7wtIvnGmIJujt1ojDnXA+WoILdhdzVtTsPSdM/eTmsXERrCjHHRfFqmST2V97Q6nKzbVcnyWd6bUzMYlrzCEZFI4DLgHmNMgzFmA/AGcJ0/ylHB4/1dlURH2L3aqT9v0mi2H6zD4dSF2pR3bN1Xw/GmNs9ntnj2WUhNZfHSpZCa6no+AFa9wpkOOIwxnReJzwcW93D8PBGpBmqAZ4BfGWPaBlKOiKwCVgEkJiaSk5Mz5DcR6BoaGoKqHZzG8O5nJ8mIs7Fh/Ycd2z3dDuGNbZxocfDc2+uYNMqSv/lOE2yfhZ4ESjs8u7MZuw04spOcql0eKXPM2rWk//a3hDQ3u5LTlpbiuOkminbupPLCC/tXiDHGcg/gPKC8y7ZbgJxujk0DJuO6WjsDKAR+NNByOj+mT59ulDHr1q3zdxV8Kv9ArUm56y3zj7wDp2z3dDvsq2owKXe9ZZ7dVOrRcr0p2D4LPQmEdnA6nebc37xnbvzbFs8WnJJiDJz+SEnpOATINb38bbXqz6sGILrLtmigvuuBxpgSY8w+Y4zTGPMZcB9w+UDLUer9XZWIwOLp3hkw0C4lfiSxI0PZdkD7cZTnFVXUc6Dm5ICHQ/eprGxg27th1YBTDNhFZFqnbXOA/nT0G+hYjmQo5aggs25XJXMnjSY+KtyrryMizJ00mk/LdKSa8ry1hRUAXODheWQkJw9sezcsGXCMMY3Aq8B9IhIpIouAS3H1z5xCRC4WkST3vzOAe4DXB1qOCm5V9c3kH6zz2ui0ruYlx7KnqoHjTa0+eT0VPNbsrGTOpNGMiY7wbMH//d8wcuSp20aOdG3vJ0sGHLfbgBFAJfA8cKsxpkBEkt1zbdrD6gXAdhFpBP6FK8D8sq9yfPUmVGDIKXJlF/BkOpvezJ00GmNg+4E6n7yeCg6Vx5vIP3CMi2Z44XO8ciU88gikpGBEICXF9Xzlyn4XYdVRahhjaoAV3WwvA6I6Pb8TuHOg5SjVWU5RFWNGhTNrfNcuP++YM8mdcaCslnOnJfjkNdXwt3an64fTRTM9l8XigQdgwQJYsgRXcFm5kg9ycsjOzu44Zt062Lq177KsfIWjlE+0Opx8WFzFkvQxiPhmklzMiFCmJEZqxgHlUWt3VjApbgTTk6L6PrifFiyAr33NFVS6s26da/+CBX2XpQFHBb3nNpdR39xGSvzIvg/2oHnJsXx64Fj7cH2lhuRESxsb9lRz4Ywkj/5wWrIEXnqp+6DTHmxeesl9BdQHDTgqqOWV1nLfW4UA/Pm93eSV+m6o8rzk0dQ0tnCg5qTPXlMNX+t3V9PS5vT8cGi6DzoDDTagAUcFuY/2VHekmGl1uBab8pW57f04Oh9HecCawgqiI+wsSI3zSvmdg84TT6QOONiABhwV5Ozub4DNC4tN9SU9aRQjQkN0Po4asq37a3h7+xHmTBpNaIj3/qwvWQK33grPPJPKrbcOLNiABhwV5HJLjxE3MozvXzTd44tN9cUeYuOMiTF8qgMH1BDklday8rHNnGx1sKnkqFdvC69bBw8/DNddt5+HH+55IEFPNOCooFVe10ROUSVXnz2J25dO88uyz/OSR1N4uI6mVofPX1sND5tKjtLa5gTA6TReuy3cuc/mG9/Y3+NAgt5owFFB6x+fHMRp4IrMSX6rw7xJo2l1GAqPHPdbHVRgy0qLp31QmrduC3c3QKC30Ws90YCjgpIxhpdzD/C5yXGkJkT6rR7tqzFqP44arHmTRhMRGsKZE2K8clu4t9FoAw06GnBUUNq6v5b9R09w5Xz/Xd0AJEVHMD4mQlcAVYO2p6qBEy0OrluY4pXbwlu39j4arT3oaKYBpXrw4tYDRIXbufgMz6UAGay5yaM144AatNz9rh8r8700HPqHP+x7NNqSJa7j+qIBRwWd+qZW/vXZEb40Zxwjw/yfTnDepFgO1p6kqr7Z31VRASi3tIb4yDBSfZwpYzA04Kig8/b2I5xsdXCFn2+ntZub7JoAqlc5ajDySmvJTIn1WR7AodCAo4LOi7kHmDominnumf7+Nnt8DCECj60v8WlqHRX4quqbKT16gvmpvh/SPxgacFRQ2V1Rz6dlx7hy/iTL/CIsPHIcJ7B5Xw0rH9ukQUf1W15pDQCZKd7pv/E0DTgqqLycdxC7TVgxb4K/q9JhU8lR18LoQGubb/O5qcCWu7+WMLuN2RN8s47TUGnAUUGj1eHk1U8OsjRjDImjwv1dnQ5ZafGEupO62Wzi03xuKrDlltYyd+Jowu0h/q5Kv1g24IhInIi8JiKNIlIqItf0cNz1IpInIsdF5KCIPCAi9k77c0Skyb0sdYOIFPnuXSgreX9XJdUNLXzNIoMF2mWmxPLczWcTGRbC/JRYv6TYUYHnZIuDHYfqyAyQ/huwcMABHgJagCRgJfCwiMzq5riRwH8CCcDZwAWcvuT07caYKPcj3Yt1Vh6UV1rLQ+v2eKxP49EPS4gMCyF6hP+HQnc1PzWOS+dNIP+g5lVT/ZN/8BhtTsP8APqBYsmAIyKRwGXAPcaYBmPMBuAN4LquxxpjHjbGrDfGtBhjDgHPAot8W2PlaXmltVz1yEZ+t7poyB3pDc1t/ODlbeSW1nKixcHXn9hiyY75ZTOTONHi4OO91f6uigoA7Z/hQLoitt5PPZfpgMMYU9xpWz6wuB/nng8UdNn2KxH5NVAE/JcxJqfrSSKyClgFkJiYSE7OaYcEnYaGBo+1w55aB7tqHGTEhTA1tu/7zb/NPUmrw9WT3tzq5Pm1W6mfEjag12xzGnIOtPH63hbqW1zbDNAywPI82Q69aXUaIkLgybXbsJVbp48JfNcGVmeldnj3kybGRwrbtnzs89cebDtYNeBEAXVdttUBo3o7SURuBOYDN3fafBdQiOv23FXAmyIy1xizt/O5xphHgEcA0tPTTXZ29lDqPyzk5OTgiXbIK63lN2s24nAawuyOPhMMPvLhXnZU78Im4F6Mk8uyz2LhlIR+vdamkmocTnj1k4PsP9rC2ZPj+MpZE7j3jQJa25yE2m1cfeGCfv8y9FQ79MeF5Z+wqeQo552/mBCbNYZtg2/bwMqs0g5Op+E/clZzyRkTyc4+0+evP9h2sGrAaQC6jvOLBup7OkFEVgC/Bi40xnTckzDGbO502FMicjVwCfCg56qrevP2Z0dOuVr5aE91j3/sH1tfwi//tYsvnDmO6xem8HLuQV7OO8hb24/0GXDySmu5+tFNtLjXBkmOHcnfblhAdnoiIsK0MaPYVHKUrLR4y96GWDZrLG9tP8KnZbVey42lAt+eqgaON7VZ9nPcE6sGnGLALiLTjDG73dvmcPqtMgBE5PPAo8AXjDGf9VG2Aazz0zEIlFY3dvzb4LryWD5rLOljT71gfXzDPu5/eydfOGMcf7pyLvYQG5+bHE9cZBh//bCEc6cmcPEZ43p8nWc27u8INgJ8bcFElmSM6difGQAjwLLTEwkNEVYXVmjAUT3ydsJOb7HkoAFjTCPwKnCfiESKyCLgUuCZrseKyFJcAwUuM8Zs6bJvtIgsF5EIEbGLyEpcfTzvev9dKICK402s313N8llJ/GB5OndfnEFDcxtf+t8NPLa+BKf7ntnfPtrHL94q5OLZY/njVa5g0+6OZenMmRjDXf/YzsHaE92+zlMf7+ef2w4jAiEC4aG2ft2Cs5roiFAWTklgdUE5xhh/V0dZVCAl7OzMqlc4ALcBTwCVwFHgVmNMgYgk4+qTmWmMKQPuAWKAf3VKVbLeGHMxEArcD2QADmAXsMIYo3NxfOSJj/bR5nTy40tmkBLvWujs8syJ/OjVz7j/7Z289ulBYkeGs2GPKyj9+ep5hIac+jsozG7jwavP4pI/r+e7L2zjxVVZHQHJ6TT85t1d/PWDEpbNTOKGc1L59MAxS98268uymUn85J872FPZwLSkXrstVZDKK61lfmpgJOzszLIBxxhTA6zoZnsZrkEF7c97XKnBGFMFLPBKBVWfjje18tymMi4+Y1xHsAFIiArnkesy+e3qIh5atxeoxybwjUWTTws27ZLjR/LfX5nNd1/Yxh/X7ubO5em0tDn54Sv5/HPbYa7LSuHeL88ixCacMzXwrmw6u8gdcFYXVmjAUaeprG+i9OgJrj07xd9VGTDLBhwV+J7bXEZ9cxu3Lp5y2j4RYWSYvWMkmuBK03F2L2ldLp07gY/2VPNQzh5Ghofw4pYDlNac4AfL07kte0rA/drrSVJ0BHMmjWZ1QTnfXjLV39VRFpPn7r8JpAwD7SzZh6MCX3Obgyc27OPcqQnMnhDT7TFZafGE2W2ECITabf3KIXbvl2cxPiaCB94porTmBKEhrtxjwyXYtFs2M4n8g3UcqTvp76ooi8ktrSXcbmP2+O6/V1amAUd5xT8/PURlfTPfXJzW4zGZKbE8e3MW31+W3ufcnHYjw+xcNPPfy0I7nWZYZldePisJgLWFFX6uibKa3NJa5kwcTZg98P58B16NleU5nYa/fljCrPHRnNtHf0pmSizfXjJ1QB38X5oznojQgV0ZBZopiVGkJUSyWgOO6uRki4OCAEvY2Zn24SiPW7OzgpKqRv589Tyv3OpqvzKy+iTOoRARLpqVxOPr91F3spWYEaH+rpKygEBM2NmZx69wRORBEXmzm+3RInKviMzotO17IrJdRPRKa5gwxvCXD/YyKW4El8we2/cJgzSYK6NAs2zmWFc+uKJKf1dFWcQb+YcBsFso7dFAePQPvYhMAb4J/Lyb3fOBn+GaG9PuL8AY4HpP1kP5z9b9tXxadoxbzks7ZfKmGrh5k0aTEBWut9UU4Jp788KWMgC++fc8S2Y874un/yL8J5BvjMntZt88oBnXpE0AjDEngac5ff0aFYDySmv50avbGRVh54pMay1yFohsNuGimUm8V1jBn98rDsg/MMpzPtpT1ZHMNlCXIu9XwBGRqSLSKiI/77L9YRGpF5H5IhIOXAs81835O4HfAuFAq4gYEXnFvfsFYKaInDOkd6L8Kq+0lmse3cTeqkZOtjgoPHLc31UaFtISI2lqc/KHNbuHvC6QCmyR4a4ud1sAD5bpV8AxxuwBHgO+JyIJACLyU+AbwFfcVzRZwGhgfTdFfB0oAd4EFrofd7j3bQOOA58f/NtQ/rap5GhH4kxjhudQZX9oX/3TAC0B+qtWeca+6kbCQmx894Jp/Z5GYDUDuaX2cyAEuEtEbsLVH3OdMWate38Wru/F9m7OzQcmAu8bYza5H6UAxhin+5ysQb4HZQGdf20F6q8vKzpnSgIR7vkWTgOxIwe2CJ0aHowxrC2sZElGIt+9cHpABhsYQMAxxpQDfwS+A/wV+A9jzEudDhkPHDfGtHRz+iwgDPikh+Kr3OerADVh9AgMsHh6YsD++rKizJRYnr0li9uypzA2OpzfvLOL4ooel4VSw9SOQ8cpP950yqTnQDTQQQO7cfXDbDTGPNRlXwSuQQHdOQvX1c+2HvafBEYMsC7KQjbsca15d9fnMzTYeFhmSiw//HwGL3/rHMLtNr7++JYel2lQw9OanRXYBJZ2Wt8pEPU74LjXnfkrsBFYJCJzuhxyFOjpL808YK8xpqee5Diguod9KgBs2F1FQlQYGWM1u7G3TIobydM3fY4TLW18/fEtHG3o6fedGm7WFFYwPyWOuMjAvqXa31FqZwH/xDVwIBsoA37Z5bBdQKiITOymiJl0Gg7djcmArlEToJxOw4Y91Zw7NQFbgE5ICxQZY6N54oYFHDp2kiv+spE/rNHh0sPdgZoT7DxynItmJvm7KkPWZ8ARkanA/wGrge+4+2h+DlwiIud3OvRD938/100xx4A57tU3s0Sko0dZREYD0zudrwLMrvJ6qhtaOHdaor+rEhTmp8Zxx7LplFQ38qf3dnPlXzfy+IYS6ptaO47JK63loXV7NBgNA+/tdE38vXAYBJxec6mJyFhcgWYnsNI9ogxckzV/CPwaOAfAGLNfRLYAX8K1PHRnPwUex3WVFAGcB2xw7/sC0AK8NtQ3o/xj/e4qgD4TdSrPaXUYBFfHaJvT8Iu3dvLfb+9kxrhoUuJHsqawAofTYLfZ+OkXZzJj/ChEhN0V9RQcPs7cSaM5c2IMdpuNEJsQGmKj4HAdBYePs2hqwin9cHmltby1t4VRk2u1f84P1uysYOqYKCYnRPZ9sMX1GnDcI9NOyy9vjHEAM04/g4eBP4nIt40xJzodvwM4u4eXuRZ42RhzygQDEYnDFaSW4erf+ZEx5rRJpe5jvwfchWvgwT9wLUfdPNBy1OBs2FPN9KQoxsZE+LsqQSMrLZ7wUButbU5CQ2zcfXEGtSdayS2tYU1hBa0O15T0FoeTn7y+47Tzn95Y2mPZv19TTEr8SCYnRBIiwgfFVTichrf2b9IRiD5Wd7KVzSU13HJ+z8t8BBJPZ4t+BteVz224Mgv0SkTmAkuA2d3sfgjXlU8SMBd4W0TyjTEFXcpYDtwNLAUO47pS+rl7W7/LUYPT1Opg876agFzuNpD1ljF7y76jXPv4FtocTuw2Gz+6OIMpY6J4Pf8Qr+YdwuCarX7JGeNYmjGGNqdhbWEFaworMLhWX7XbhKMNLZRUNdDmzqfSPvFUA47v5BRV0uY0XDgj8G+ngYcDjjHGISLfwDUMuj/GAje6Mxl0EJFI4DJgtjGmAdggIm8A1/HvQNLueuDx9gAiIr8AngXuHmA5ahC27q+hpc3JedP1dpqvZabEdvvH/3OT43n+ltODUWS4nbe3H3FdFdlt3Lhocse+KYlRfLi7qmPfA5fPITMllrzSWlY+uokmdxYJndDrW2sKK0iICmPepNH+ropHeHw9HGPMJmBTP499p4dd0wGHMaa407Z8YHE3x84CXu9yXJJ7YEJyf8sRkVXAKoCRI0cOuyWLvWV09o1EZ36ZC85IxrTqMF2rCxufQUTyGTSVfcb8+3f1a1/Y+Axil95EeNJUPpc+CWdzo6+rHZxsdib9x7M07tpASMgyf9fGI6yaPz4KqOuyrQ7obpJH12Pb/z1qIOUYYx4xxsw3xsyfOHEixpigf6xbt67PY7JW3MjC6Uk4W5r8Xl9/tkOgPJoP7aRu40s0H9rZ733Nh3by68vnIfZQnswp8Pt7CJbPwoe7DmMLj+TF3/2X3993f9uhL1YNOA1AdJdt0UB3OT26Htv+7/oBlqMGqKq+mZ1HjnOeDoce9lKibWSMHcUruQf8XZWgsaawgohQG+dOGz63q60acIoBu4hM67RtDtBdR3+Be1/n4yqMa9TbQMpRA/SRO53NecPoC6G6JyJcMX8S+QfrNJebDxjjGshx3rREIkJD/F0dj7FkwDHGNOKay3OfiESKyCLgUlyj4Lp6GrhJRGaKSCzwE+DJQZSjBmj97mpiR4Yya3yMv6uifGDF3PHYbcLLepXjdQWHj3O4rmlYZBfozJIBx+02XPNqKoHncc2tKRCRZBFpEJFk6Bh48ACwDih1P37WVzm+exvDkzGG9burOGdqAiGaziYoxEeFszRjDK99eohWh7PvE9Sgrd1ZgQhcEODJOrvy+Cg1TzHG1AArutlehmswQOdtvwd+P5By1NDsrmygsr6Z8zS7QFC5Yv4kVhdWkFNUNex+fVvJmsIKMpNjiY8K93dVPILUQlIAAB8rSURBVMrKVzjKwj4sdqez0f6boJKdnkhCVJjeVvOidwvKKTh8nBnjuo53CnwacNSgbNhTTVpCJBNjR/q7KsqHQkNsfGXeBN7fVanLI3hBXmkt337WtU7lS7kHhl3yVQ04asCa2xxsLqnR0WlB6or5k2hzGv657bC/qzLsbCo52pFKqM3hSiU0nGjAUQOWV1rLyVaHLkcQpKYnjWLOxBhezj3Qr8l+qv/OcI/4FCDUbht2qYQ04KgB27C7mhCbkJUW5++qKD+5PHMiu8pdSx0ozzl6wnWbcmVWyrDMzK0BRw3YOzvKGRsdTnFFg7+rovzky3MmEGa36eABD1tdUEFSdDj3fXnWsAs2oAFHDdAHRZWUVDdy+FgTKx/bNOw6NVX/xIwMZdnMJF7PP0xzm8Pf1fGY3lZKbV+Izluf+aZWBx8Uu4abD9el2i07D0dZ0+v5ro5iA7Tq+ihB7Yr5k3hr+xF+8PJ2rj8nNeA/B3mltax8bBMtbU5CbMJ3L5jG5ATXlL+Sqgb+/P5u2hzeW4huw+5qTrQ4WD5rrEfLtRINOGpAnO4J5iEyPDs1Vf+NDHPl+Hoj/zCrC8sDvs9hU8lRmludGMDpMPx2dXG3x3lrIbrVheWMirBz9uTh+53SgKMG5GDtCaYnRXHp3AmnrTSpgsuWfTUd/x4OV7vjYiJoH3MXbrfx66+eyczx0YhA4eHj/PCV7bS4U/qcPdmzA2YcTsPanZUszRhDmH349nRowFH91tTqYPvBOm5YlMq3l0z1d3WUn2WlxRMWIrQ4DCG2wL7abWlz8uj6fcSMsHNtVgpLM5JOCZ7Tk0YxKW4kv3p1M7kVDnJLa5mf6rmgk1daS01jC8tmDt/baaCDBtQA7DhUR4vDGdC/YpXnZKbE8uSNnyPEJiybmRTQn4s/vVfMziPH+e0Vc/nB8oxu30tmSizfnhvOJWeM5Xeri/jsYNe1HQfv3YJywuw2FqcP77ltGnBUv23d7xqdMz+A/7AozzpnagLnTIlnZ3ngzsf5pKyWh3P2ckXmxD4TkooIv/zKGcRHhvPdFz/lREvbkF/fGMPqwnLOnZpAVPjwvumkAUf1W+7+GtISI4ddBls1NNnpY9hb1ciBmhP+rsqAnWhp446X8hkXM4Kffmlmv84ZPTKM3185h33Vjdz/9s4h12FXeT0Hak6yLAiyb2vAUf3idBpyS2tZkKLZBdSpst23gXLcGcQDyW/+bxf7qhv5nyvOZFREaL/PO2dKAqvOT+O5zWWsLigfUh1WF7jXvpmhAUcpAPZUNVB3spX5qXo7TZ0qLSGSSXEj+KCo0t9VGZCP9lTz1MZSbjgnlXOmDDwR7R0XpTN7QjR3/WM7lcebBl2PdwvKyUyOJXHU8L9zoAFH9Uuuu/9mgQdH5qjhQUTInj6Gj/ceDZisA+t3V/GtZ/IYHxPBXZ/PGFQZYXYbf7xyHidbHdzydC7/+/7uAWchOFBzgsIjx1k2a/hf3YAFA46IxInIayLSKCKlInJNL8deLyJ5InJcRA6KyAMiYu+0P0dEmtxLUjeISJFv3sXwk7u/hoSoMFLidf0bdbrF0xM50eLo+GFiZXmltdzwxFbqm9uobmyh8MjgBzxMHRPF9QtTyT9Yx29XF3PNowNL97SmsAJg2A+Hbme5gAM8BLQAScBK4GERmdXDsSOB/wQSgLOBC4A7uxxzuzEmyv1I91Kdh72tpTXMT4lDZHjmeFJDc87UeMJCbOQEwG219burcLiXVXB4YM2Z6BF22r8VzW1O/vbRvn4v27C6sJz0pFGkJkQOqQ6BwlIBR0QigcuAe4wxDcaYDcAbwHXdHW+MedgYs94Y02KMOQQ8CyzyXY2DQ3ldEwdqTmr/jerRyDA7n5scR06R9QcOtM/kt3koPVNWWgLhoTZsAiLw1vYj3PJ0Xp/9OjWNLWzZVxM0t9MAxEoLKInIPOBjY8yITtvuBBYbY77Uj/P/Cewyxtztfp4DzMK1nlER8F/GmJwezl0FrAJITEzMfOmll4b2ZvphT62DXTUOMuJCmBob4vXXG6iGhgaioqLYcqSN/5ffzE8XRpAWY716elt7OwSz/rTBO/taeaGohd8tHkH8CEv9lj3FY581s6W8jS9ODmVm/MC+ez21Q/t3eXqcjb3HDK/ubiHUBhcm2wkNEWZ08x1ff7CVx3e0cO/CCFID7HvVUzssWbIkzxgzv6fzrDbLKAroOn23DhjV14kiciMwH7i50+a7gEJct+iuAt4UkbnGmL1dzzfGPAI8ApCenm6ys7MHU//TbNl3lNe3HWZcTASjIkKprG+i8ngze6oa2HagEQyEhzosmfgwJyeH7Oxsct4oYEToAa774hJCQ6z7h8Rb2tshmPWnDSbOrOeFog9pjptK9tnJvqnYADmchu+vX8vnZ4/n91fPG/D5PbVD1y3frGrgtmc/4Y2SegBs0soXzhjHrAkxJEaFkzAqnK2fFTEq3MkZc+aRGWCDcQb7nfBpwHFfcSzuYfdHwHeA6C7bo4H6PspdAfwauNAYU92+3RizudNhT4nI1cAlwIMDq/ngbNxbzcrHNuPsdBEZYhMSosLc9XNta251sqmk2nIBp11uaQ3zkkcHZbBR/TclMYoJo0eQU1TJNRYNOJ+UuXKW9ZVRYKjSEqP44pnjKCqvd2WfNvB/O8p5c/uR045d+fhmS/7g9AafBhxjTHZv+919OHYRmWaM2e3ePAco6OWczwOPAl8wxnzWVxUAn/R6O5yGn71R0BFsbAK3Zk/hjovSsdmkY+2NJnc69OKKBowxluuUb2huo/DwcW5fOs3fVVEWJyIsTk/k9U8P0dLmtGTW4zWFFYSGSMdkVW9aOCWB8NA9tLY5CbXbePams0kfF011fTOPfFjC81vKgm5dKUt9IowxjcCrwH0iEikii4BLgWe6O15EluIaKHCZMWZLl32jRWS5iESIiF1EVgLnA+969124ciP94q1CiisasNuEEHF1VC7N+PdKfpkpsTx7cxZ3LpvOxbPH8vq2w/zwle20udOfW8WnZbU4jeZPU/2TPT2RxhYHuaU1fR/sY8YY1hRWkJUWP6CsAoPV/h3//rJ01xVMahxR4XZSEyK5LHMi4aG2oFtXymp9OAC3AU8AlcBR4FZjTAGAiCTj6pOZaYwpA+4BYoB/dboyWG+MuRgIBe4HMgAHsAtYYYzx+lycR9eX8OTH+7np3MlccsY4NpUc7XbtmMyUWDJTYjHG8Me1u/nTe7s5drKVB6+eR0SoNToRt+6vxSYwL3m0v6uiAsA5UxMIDRE+KK4a1Ox9b9pb1cC+6ka+sSjVZ6/Z/h3vbvuzN2f1+LdhuLJcwDHG1AArethXhmtgQfvzJb2UUwUs8HgF+/D6tkP88l+7+MKZ4/ivS2Zgs0mfHyYR4XsXTScuMoyfvVHAV//fR1w0M4nzp4/x+wcxd38NM8ZF++QXoQp8UeF2FqTG8UFRFT+6eIa/q3OK1e5JllbJWdZTMBrOLHVLLdB9tKeaO1/OJystjt9/bU7H7bP+uv6cVP7zwmkUHqnnT+/tYeVjA5u17GltTsO2A8c0nY0akMXTE9lVXs+RupP+rsop1hZWMHtCNONHj+j7YOUVGnA85B95B7nxb1sZFz2Cv143n3D74G6JhYbYOkY1tK+d7i8H6p2caHHohE81INnpYwD4wEKTQKvqm/n0wDEumhEcKWSsSgOOB6zdWcEdL+fT4nBSUd/EnsqGQZeVlRZPeKjtlOf+UlzrGsAwX5ckUAMwPSmKcTERfGCh5Qre21mBMXh9OLTqnQacIXI6zSmLMLUNMTdTe2fieVMTcBqIHem/vpPdtQ4mxo5gbEyE3+qgAo+Ia9jxB0VVPDiIDMresKawggmjRzBjXJ9zyJUXacAZoqc27md/dWPH8GdPDHHMTInld1fOwW4Tnt1c5pmKDpAxhuJap/bfqEGZFDuSE60O/rCm2O99kSda2tiwp5qLZiZZbp5bsNGA0426ZtOvL0hReT2/+r9dLM0Yw4urOo2398DIkzGjIlg+eyyv5B2kqdX3a4yU1ZzgeIvR/hs1KK3u+WRO8++Jjf6yfnc1zW1OvZ1mARpwulHbbLi6j3UtmlodfPeFT4mOsPPA5WeSmRrHt5dM9egwx2vPTqHuZCtv5h/2WJn99XLeQQAiwyw3cl4FgHOnJWJ3j9K0h/h3YuOawgqiI1zZrJV/acDpQUubk8fWl/S4/4F3ithVXs//XDGHhCjvLA2blRbH1DFR/N3Ht9XySmt5eJ0rv+ndr263xD14FVgyU2L5y3WZhIiwNMN/88kcTsP7uypZkjFGcwFagP4f6IFNXMn2fvr6DlraTk0380FxFU98tI8bzklliXsIqDeICCvPTib/wDF2HOqaRNt7Nu6t7ligyt+3Q1TgunBGEl89awI5RVUcO9HilzrklfomWafqHw043YgNF15YlcWq89N4emMp1zy6qWMxpaMNzdz5cj7Tk6K4++LBrYU+EF89ayIjQkP4+6ZSr79Wu/asAkJw5XlSnnfzeWmcbHX4bfDL2p2uZJ2Lp3s/WafqmwacbsSEC5+bHM+PL5nBg1fPo+Dwcb744Aae21zKlY9soraxhT9d5Zt8ZzEjQvnynPG8vu0wx5tavf56APuqGwkNES6dEho0adOVd6SPHcV50xJ48uP9NLf5dvBL3v4aXs49wExNzWQZGnD68KU543nt2+dgE/jxazs6JnWeaPHdl+farBROtjp41d2R701Op+GdHeUsSR/DimlhGmzUkN1yXhpV9c28mX/6WjDekldayzWPbab2RCsFh49rP6RFaMDph4yx0Xxt/qSO58YYn/ZrnDExhjkTY/j75jK8vST49kN1lB9v4vOzNQWI8ozzpiWQnjSKx9aXeP3z225TydGOvlenj7+vqmcacPppcfoYIvy4fsXKrBT2VDaweZ931xl5Z0c5dptwQYZ2sirPEBFuOm8yu8rr+WiPb/7wZ6XFdyy1GKb9kJahAaefTltMyce3mr505niiI+xeHTxgjOGdHUdYOCWeGD+m1FHDz6Vzx5MQFc6jvUw18KS0hEgwsDAtTvshLUQDzgBkpsR6fHJnf40IC+HyzEn8344jPPDOLq/cky6qqGf/0RNcPHucx8tWwS3cHsL1C1P4oLiK4op6r79eTnElBrjr4hkabCxEA04AmZscg8MJD+fs9Up+qnd2lCOiGXWVd6zMSiEi1Mbj6/d5/bXe21lJQlQ4Z06I8fprqf6zXMARkTgReU1EGkWkVESu6eXYG0TEISINnR7ZgykrEByocS1oZfDOhMx3dpSzICWOxFHeyZyggltcZBiXnTWRf3xy0GtX6eDK4/ZBcRVLMxIHvAii8i7LBRzgIaAFSAJWAg+LyKxejt9ojInq9MgZQlmWlpUWT7jd/b9MxKMdofurG9lVXs9yHZ2mvCgrLY42p/HaVTrA1v011De1WWYpafVvlgo4IhIJXAbcY4xpMMZsAN4ArvNnWVaRmRLLc7dkcebEGDCGuMgwj5X9bkE5AMtn6ZdUeU+Zl6/SAd7fWUlYiI1zpyZ4vGw1NFZLBTwdcBhjijttywcW93LOPBGpBmqAZ4BfGWPaBlqWiKwCVgEkJiaSk5Mz6Dfhbd+Y5uSuI/CDZ9Zz+zzPLI724saTpEbb2JO/hT3ubQ0NDZZuB1/RdvBcG4Qfc2AXaDOuUcvhx0rJyfHshOY3PznB9FgbWzdu8Gi5oJ+FdoNtB6sFnCiga5bKOqCnZfo+BGYDpcAs4EWgDfjVQMsyxjwCPAKQnp5usrOzB157Hyqz7+b3a4qJSj2T+UNcJO1I3UlK3nmfHyxPJzt7asf2nJwcrN4OvqDt4Lk2yAbmzKvhxr9tZVpSFDd/ZdGQy+yspKqBinc+4NsXZZC9MNWjZYN+FtoNth18ektNRHJExPTw2AA0ANFdTosGuh1HaYwpMcbsM8Y4jTGfAfcBl7t3D6isQHPzeZNJig7n/rd3Dnn29uqCCgDNLqB8YkFqHDcuSiX/wDEOHzvp0bLf21kJwNIM72VxV4Pn04BjjMk2xkgPj3OBYsAuItM6nTYHKOjvS9Axv3jIZVnayDA7dyxLZ9uBY7z92dByVL2zo5xpY6KYkhjlodop1bsrMifhNPCKh/MDvrergoyxo5gYO9Kj5SrPsNSgAWNMI/AqcJ+IRIrIIuBSXH0zpxGRi0Ukyf3vDOAe4PXBlBWILjtrIhljR/Gbd3YNOhNvTWMLm/cd1asb5VPJ8SNZNDWel3IP4HR6Jr9a3clWtu6v1asbC7NUwHG7DRgBVALPA7caYwoARCTZPdcm2X3sBcB2EWkE/oUrwPyyP2UNByE24b++MIMDNSd5ZuPgUt6sKSzHaWD5LA04yreuXJDMwdqTfLzXMyPVPiiuwuE0XDBDA45VWW3QAMaYGmBFD/vKcA0GaH9+J3DnYMoaLs6blsji6Yn8+b3dXJ45kdEjBzZU+sWtB4gZYae51bdrlSi1bGYSMSNCeWFrGedOG/oQ5vd3VhAXGcbcSZrKxqqseIWjBujHl8ygvqmNm5/KHdBEuje2HeKTsmMcP9nGysc365ohyqciQkP4yrwJrC6ooLZxaEtQtzmcrCuqIjs9kRDNLmBZGnCGgYbmNmwi5JbW8rW/bmSNe9RZT6rqm/nJPz/juy9uA7w7CU+p3ly5YBItDievfXpoSOV8UnaMupOtuqyGxWnAGQY2lRzF4Op4dTgN3/x7Lt9/cRuFh4+fctzJFgcPvreb7P9Zx/NbDrB85ljC7f5b40epGeOimTMxhhe3HhjS8P73dlVgtwnnT9fsAlZmuT4cNXBZafGE2W20tjmxh9i4cMYY3iko59VPD7FoajzZ6WPI21/L5n1HqT3RyvJZSfzw8xlMSYwir7SWTSVHyUqL1zTuyi+uXJDMj1/7jPyDdcydNHpQZby3s5Kz0+IYFaHrOFmZBpxhoH1xuM6Bo+5kK89vKeORD0o6VlkUgftXzObarJRTztVAo/zpS3PG8Yu3Cnlxa9mgAk7p0Ub2VDZwzeeS+z5Y+ZXeUhsmui4OFzMilG8tnsINi1I7ZsLacM1VUMpKRkWE8oUzx/HGtsM0NrcN+PwnP94PQFK0LqthdRpwhrlFUxMID9V+GmVtVy6YRGOLY8BZM/JKa3nKHXDueDlfR1panN5SG+a6u92mlNXMT4klLTGSxzfso6q+ud+f1Wc27qc9UUH7SEv9jFuXBpwgoP00yupEhEVTEnhmUym/qygizG7j2Zuzev3c7jhUx78+cy2LbkOv4AOBBhyllCWMinD9OXIaaOnjaqXyeBO3PJ1LQlQY962YTVF5vV7BBwANOEopS7hgRhKPbdhHS5sTp4HEqO7TNDW1OrjlmTyOnWjllVsXMmt8DBfqctIBQQcNKKUsITMlludvyeKmcyeTOCqMe98sZF1R5SnHGGP4wSvb2X7wGH+8ai6zxsf4qbZqMDTgKKUsIzMllnu+OJO3v3MeqfGR3PxULi/nHujY/+D7e3gz/zA/XJ6hGc4DkN5SU0pZzpjoCF78Zha3/v0TfvDKdj49cIyGpjbeyD/MV8+awLcWp/m7imoQNOAopSxpVEQoT9ywgJue2spzm8sAV7aMr82fiIhmhA5EektNKWVZYXYbZ0+OOyVbRl7pMX9WSQ2BBhyllKUtnKLZMoYLywUcEYkTkddEpFFESkXkml6O/Yt7yen2R7OI1HfanyMiTZ32F/nmXSilPKU9W8b3l6X3ORlUWZsV+3AeAlqAJGAu8LaI5BtjCroeaIz5FvCt9uci8iTg7HLY7caYx7xXXaWUt2m2jOHBUlc4IhIJXAbcY4xpMMZsAN4ArhvAuU95t5ZKKaUGw1IBB5gOOIwxxZ225QOz+nHuZUAV8GGX7b8SkWoR+UhEsj1TTaWUUgNltVtqUUBdl211wKh+nHs98LQ5dZ3au4BCXLforgLeFJG5xpi9XU8WkVXAKoDExERycnIGXvthpqGhQdsBbQfQNmin7eAy6HYwxvjsAeQApofHBmAecKLLOXcAb/ZR7iSgDUjr47h3gO/0Vc/p06cbZcy6dev8XQVL0HbQNmin7eDSUzsAuaaXv60+vcIxxmT3tt/dD2MXkWnGmN3uzXOA0wYMdPF14GNjTElfVQB0xphSSvmBpfpwjDGNwKvAfSISKSKLgEuBZ/o49evAk503iMhoEVkuIhEiYheRlcD5wLteqLpSSqk+WCrguN0GjAAqgeeBW417SLSIJLvn0yS3HywiC4GJwMtdygkF7sc1kKAa+A6wwhijc3GUUsoPrDZoAGNMDbCih31luAYWdN62EYjs5tgqYIE36qiUUmrgrHiFo5RSahjSgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonLBdwROR2EckVkWYRebIfx39PRMpFpE5EnhCR8E774kTkNRFpFJFSEbnGq5VXSinVI8sFHOAwcD/wRF8Hishy4G7gAiAVSAN+3umQh4AWIAlYCTwsIrM8XF+llFL9YLmAY4x51RjzT+BoPw6/HnjcGFNgjKkFfgHcACAikcBlwD3GmAZjzAbgDeA679RcKaVUb+z+rsAQzQJe7/Q8H0gSkXggGXAYY4q77F/cXUEisgpY5X7aLCI7vFDfQJMAVPu7Ehag7aBt0E7bwaWndkjp7aRADzhRQF2n5+3/HtXNvvb9o7oryBjzCPAIgIjkGmPme7aqgUfbwUXbQdugnbaDy2Dbwae31EQkR0RMD48NgyiyAYju9Lz93/Xd7GvfXz+I11FKKTVEPg04xphsY4z08Dh3EEUWAHM6PZ8DVBhjjgLFgF1EpnXZXzD4d6CUUmqwLDdoQETsIhIBhAAhIhIhIj3d+nsauElEZopILPAT4EkAY0wj8Cpwn4hEisgi4FLgmX5U45Ghvo9hQtvBRdtB26CdtoPLoNpBjDGersiQiMi9wM+6bP65MeZeEUkGCoGZxpgy9/HfB+4CRgD/AL5ljGl274vDNbz6Ilyj3u42xjznkzeilFLqFJYLOEoppYYny91SU0opNTxpwFFKKeUTGnA6Cdbca73lrxORC0Rkl4icEJF1ItLrxK5AJSLhIvK4+/97vYh8KiIXd9ofFO0AICJ/F5EjInJcRIpF5OZO+4KmHQBEZJqINInI3zttC5o2cE9laRKRBvejqNO+AbeDBpxTBWvutW7z14lIAq6RfvcAcUAu8KLPa+cbduAArkwUMbje80sikhpk7QDwKyDVGBMNfBm4X0Qyg7AdwPU3YWv7kyBtg9uNMVHuRzoMvh100ICbO/daLTC7PR2OiDwDHDLG3O3XyvmIiNwPTDTG3OB+vgq4wRhzjvt5JK50FvOMMbv8VlEfEZHtuJLBxhOk7SAi6UAO8F1gNEHUDiJyFfBVXCNjpxpjrg2274SI5AB/N8Y81mX7oNpBr3D+bTrd514LhiucnszC1QZAx9ymvQRBm4hIEq7PRAFB2A4i8v9E5ASwCzgC/IsgagcRiQbuA+7osito2qCTX4lItYh8JCLZ7m2DagcNOP82oNxrQSIo20REQoFngafcv9aCrh2MMbfhen/n4bp10kxwtcMvcGWiP9BlezC1AbjmOKYBE3BN9nxTRKYwyHbQgPNvmnvtdEHXJiJiw5WNogW43b056NoBwBjjcC/rMRG4lSBpBxGZC1wI/KGb3UHRBu2MMZuNMfXGmGZjzFPAR8AlDLIdNOD8m+ZeO90puerc92mnMEzbREQEeBzXoJHLjDGt7l1B1Q7dsPPv9xsM7ZCNa0HHMhEpB+4ELhORTwieNuiJAYTBtoMxRh/uB/AC8DwQCSzCdYk4y9/18sH7tgMRuEYnPeP+tx1IdLfBZe5tvwE2+bu+XmyHvwCbgKgu24OmHYAxwFW4bpmEAMuBRlx5CIOiHYCRwNhOj98Cr7jff1C0gbsdRrv//7f/PVjp/iykD7Yd/P6mrPTANbzvn+5GLQOu8XedfPS+73X/cun8uNe970JcHccncY1WSvV3fb3UBinu992E63ZB+2NlkLVDIvABcAw4DnwG3NJpf1C0Q5c2uRfXSK2gagP3Z2Errttkx9w/xi4aSjvosGillFI+oX04SimlfEIDjlJKKZ/QgKOUUsonNOAopZTyCQ04SimlfEIDjlJKKZ/QgKOUUsonNOAoFSBEJFpE7hWRGf6ui1KDoQFHqcAxH/gZEOrviig1GBpwlAoc83AtE1Do74ooNRia2kapACAiO4GMLpv/YYy53B/1UWowNOAoFQBEZAGubOYFwC/dm48YY0r9VyulBsbu7woopfolH9dCaA8aYzb5uzJKDYb24SgVGGYBYcAn/q6IUoOlAUepwHAWrvV6tvm7IkoNlgYcpQLDPGCvMea4vyui1GBpwFEqMMxEh0OrAKeDBpQKDMeAs0RkOa615HcbY476uU5KDYgOi1YqAIjIbOBx4EwgAjjPGLPBv7VSamA04CillPIJ7cNRSinlExpwlFJK+YQGHKWUUj6hAUcppZRPaMBRSinlExpwlFJK+YQGHKWUUj6hAUcppZRP/H+nZPHYSAEOsgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred = model.predict(X_valid)\n",
    "plot_series(X_valid[0, :, 0], y_valid[0, 0], y_pred[0, 0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hOmZlELkFpae"
   },
   "source": [
    "---------------\n",
    "\n",
    "To predict several time steps ahead, we may either:\n",
    "- predict one by one and feed the new predicted ones into the inputs (in which, the accuracy is gradually decreasing and is appropriate for only a few steps), and\n",
    "- predict all the steps ahead together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9exog6KmFpae"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "n_steps = 50\n",
    "series = generate_time_series(10000, n_steps + 10)\n",
    "X_train, Y_train = series[:7000, :n_steps], series[:7000, -10:, 0]\n",
    "X_valid, Y_valid = series[7000:9000, :n_steps], series[7000:9000, -10:, 0]\n",
    "X_test, Y_test = series[9000:, :n_steps], series[9000:, -10:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DUNrQc0XFpae"
   },
   "outputs": [],
   "source": [
    "# predict the next 10 values one by one:\n",
    "\n",
    "X = X_valid\n",
    "for step_ahead in range(10):\n",
    "    y_pred_one = model.predict(X)[:, np.newaxis, :]\n",
    "    X = np.concatenate([X, y_pred_one], axis=1)\n",
    "\n",
    "Y_pred = X[:, n_steps:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "n9rF0F4qFpae",
    "outputId": "8ce8e719-8484-4466-b7a9-c13ca6213acf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.027277222"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(keras.metrics.mean_squared_error(Y_valid, Y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PahEuKiYFpaf",
    "outputId": "d6c3deea-65d0-4ee9-e5ca-0ca74f2619cf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.22278848"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Baseline metric 1:\n",
    "\n",
    "Y_naive_pred = Y_valid[:, -1:]\n",
    "np.mean(keras.metrics.mean_squared_error(Y_valid, Y_naive_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nnRqw639Fpaf",
    "outputId": "c8de823c-0bf9-432e-f375-54b362f64f58"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 0s 61us/sample - loss: 0.1348 - val_loss: 0.0610\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 0s 37us/sample - loss: 0.0501 - val_loss: 0.0427\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 0s 37us/sample - loss: 0.0388 - val_loss: 0.0356\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 0s 38us/sample - loss: 0.0334 - val_loss: 0.0314\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 0s 37us/sample - loss: 0.0299 - val_loss: 0.0286\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 0s 38us/sample - loss: 0.0275 - val_loss: 0.0264\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 0s 38us/sample - loss: 0.0258 - val_loss: 0.0251\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 0s 37us/sample - loss: 0.0245 - val_loss: 0.0237\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 0s 39us/sample - loss: 0.0235 - val_loss: 0.0229\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 0s 39us/sample - loss: 0.0227 - val_loss: 0.0222\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 0s 38us/sample - loss: 0.0221 - val_loss: 0.0220\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 0s 41us/sample - loss: 0.0216 - val_loss: 0.0212\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 0s 41us/sample - loss: 0.0211 - val_loss: 0.0207\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 0s 37us/sample - loss: 0.0207 - val_loss: 0.0204\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 0s 36us/sample - loss: 0.0203 - val_loss: 0.0203\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 0s 36us/sample - loss: 0.0200 - val_loss: 0.0198\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 0s 35us/sample - loss: 0.0197 - val_loss: 0.0194\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 0s 35us/sample - loss: 0.0194 - val_loss: 0.0192\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 0s 35us/sample - loss: 0.0191 - val_loss: 0.0190\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 0s 36us/sample - loss: 0.0189 - val_loss: 0.0188\n"
     ]
    }
   ],
   "source": [
    "# Baseline metric 2:\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[50, 1]),\n",
    "    keras.layers.Dense(10)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\")\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZE4uraHBFpaf",
    "outputId": "66511f43-64e6-4215-e332-0e0c7842407f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 4s 612us/sample - loss: 0.0665 - val_loss: 0.0317\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 3s 456us/sample - loss: 0.0268 - val_loss: 0.0221\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 3s 461us/sample - loss: 0.0188 - val_loss: 0.0166\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 3s 458us/sample - loss: 0.0159 - val_loss: 0.0137\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 3s 477us/sample - loss: 0.0138 - val_loss: 0.0127\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 3s 447us/sample - loss: 0.0132 - val_loss: 0.0117\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 3s 451us/sample - loss: 0.0119 - val_loss: 0.0117\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 3s 449us/sample - loss: 0.0115 - val_loss: 0.0119\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 3s 457us/sample - loss: 0.0110 - val_loss: 0.0113\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 3s 460us/sample - loss: 0.0111 - val_loss: 0.0097\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 3s 464us/sample - loss: 0.0104 - val_loss: 0.0098\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 3s 460us/sample - loss: 0.0100 - val_loss: 0.0092\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 3s 457us/sample - loss: 0.0101 - val_loss: 0.0101\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 3s 467us/sample - loss: 0.0100 - val_loss: 0.0103\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 3s 467us/sample - loss: 0.0097 - val_loss: 0.0092\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 3s 461us/sample - loss: 0.0094 - val_loss: 0.0088\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 3s 462us/sample - loss: 0.0091 - val_loss: 0.0088\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 3s 454us/sample - loss: 0.0091 - val_loss: 0.0081\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 3s 468us/sample - loss: 0.0089 - val_loss: 0.0079\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 3s 456us/sample - loss: 0.0088 - val_loss: 0.0081\n"
     ]
    }
   ],
   "source": [
    "# Prediction of the 10 values in the end:\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(20),\n",
    "    keras.layers.Dense(10)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\")\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "p1T_IV4JFpag"
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "\n",
    "series = generate_time_series(1, 50 + 10)\n",
    "X_new, Y_new = series[:, :50, :], series[:, -10:, :]\n",
    "Y_pred = model.predict(X_new)[..., np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OmDpH93LFpag",
    "outputId": "5dd397db-3c99-4db5-9a46-5857ca096b7d"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEUCAYAAACGWlk5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3xUVfr48c8zqYSaQockhCYgTVoQhURduwuCRDHq2sC27rqu6xbXr6uri8vP3VV3FUV0RQwoqCCWFQuJSAklVGmhJBNqIAVIIW3m/P64kxjChLQpd4bzfr3yInPvuWfOJYFn7inPEaUUmqZpmuZNFm83QNM0TdN0MNI0TdO8TgcjTdM0zet0MNI0TdO8TgcjTdM0zet0MNI0TdO8TgcjTdM0zetMGYxE5JcislFEykXk3QbK/kZEjonIKRF5R0RCap2LEJElIlIiIlYRud3tjdc0TdOazJTBCDgCPA+8c75CInIN8AfgSiAWiAOerVXkNaAC6AwkA7NFZJAb2qtpmqa1gJg5A4OIPA/0UErdXc/5BUC2UupPjtdXAilKqS4i0hooBC5WSmU6zs8HDiul/uCRG9A0TdMaJdDbDWihQcCntV5vBTqLSCQQDdiqA1Gt8xOcVSQiM4AZAKGhoSOio6Pd02ITsNvtWCxmfShuOX1/vs2f78+f7w0gMzMzTynVsTnX+nowagOcqvW6+vu2Ts5Vn2/rrCKl1BxgDkD//v3Vnj17XNtSE0lLSyMhIcHbzXAbfX++zZ/vz5/vDUBErM291tdDdDHQrtbr6u+LnJyrPl/kgXZpmqZpTeDrwWgHMLTW66FArlIqH8gEAkWkb53zOzzYPk3TNK0RTBmMRCRQREKBACBAREJFxFmX4nvAfSIyUETCgT8D7wIopUqAT4DnRKS1iIwDJgLzPXITmqZpWqOZMhhhBJUzGNO273B8/2cRiRaRYhGJBlBKfQXMAlIBq+PrmVr1PAy0Ao4DC4GHlFL6yUjTNM1kTDmBQSn1F+Av9ZxuU6fsP4F/1lNPATDJlW3TNE3TXM+sT0aapmnaBUQHI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvE4HI03TNM3rdDDSNE3TvM60wUhEIkRkiYiUiIhVRG6vp9wbjq3Iq7/KRaSo1vk0ESmrdX6P5+5C0zRNawxTbjvu8BpQAXQGhgFfiMhWpdSO2oWUUg8CD1a/FpF3AXudun6plJrr3uZqmqZpzWXKJyMRaQ1MAZ5WShUrpVYBy4A7G3ndPPe3UtM0TXMVUwYjoB9gU0pl1jq2FRjUwHVTgBPAyjrHZ4pInoisFpEE1zVTc6X1Wfm8+L9dZFgLvd0UTdM8TJRS3m7DOUTkcmCxUqpLrWPTgWSlVMJ5rvsO+EEp9Zdax8YAOzG6/G4D/gMMU0rtr3PtDGAGQMeOHUcsWrTIZfdjNsXFxbRp08bbzTjL5txKXt1cgQKCLPD7UaH0CQ9oVl1mvD9X0vfnu/z53gASExMzlFIjm3OtWceMioF2dY61A4qclAVARHoCE4DptY8rpdbVejlPRKYB1wP/rlNuDjAHoH///iohIaG5bTe9tLQ0zHR/a/bn8daKjVR/LKq0Q3mHGBIS+jSrPrPdn6vp+/Nd/nxvLWXWbrpMIFBE+tY6NhTYUU95gLuANUqpAw3UrQBpYfs0F1BKMTttP3fMXUd4WBAhgZaaH0xsZJhX26ZpmmeZMhgppUqAT4DnRKS1iIwDJgLzz3PZXcC7tQ+ISAcRuUZEQkUkUESSgfHAcjc1XWukH/ae4Gf/Wsnfv9rNdYO7svw3E1gwPZ4HE3oTGCCs2pfn7SZqmuZBZu2mA3gYeAc4DuQDDymldohINMYY0EClVA6AiIwFegCL69QRBDwPXATYgN3AJKWUXmvkRWv353HX2+tRQKBFuPfSWNqEBDIiJpwRMeEUlVXy4YaD/OrKvnRt38rbzdU0zQNM+WQEoJQqUEpNUkq1VkpFK6UWOI7nKKXaVAcix7G1jnJFdeo4oZQapZRqq5TqoJSKV0p94+l70c72xvcHasaHlFKkZxWcdf6B8b2xK5izsqEeV03T/IVpg5HmnwpLKtiQXYBFIEAgKNBCfFzkWWV6RoQxaVh3Fq7PIa+43Est1TTNk3Qw0jzqtdR9lFXa+MfUYTx+dX9S7o9nREz4OeUeTuxNeZWdt1dleaGVmuYhKSkQGwsWi/FnSsr5j/sxM48ZaX7mYEEp7621csuIHtx8Sffzlu3dsQ3XD+7K/LVWHhzfm/ZhQR5qpaa5QUoKPPUUE3JyIDoaXnjBOD5jBpSWGt9brcbr1ath3rxzjwMkJ3u+7R6in4w0j/nH13sQgd/8rF+jyj+S0Ifi8irmrc12a7s0za1SUoxgYrUiShnB5b77YPr0nwJOtdJSmD3b+fGnnvJcm71APxlpHvHj4VMs3XKEhxJ6N3qG3MBu7bjyok7MWbkfu11xeb+OTrv0NM3Unnrq3OBS3oyx0Jychsv4MP1kpHnE37/aTXhYEA8l9G7SdVcN7ExxuY1XvttL8tx0nbdO8xmzZkFqKucNIqkkMIvfnX0woJ40WNHRrmucCelgpLndyswT/LA3j19e0Zd2oU0b+ykoqQCMtBkVVXbSD+S7oYWa5nqjRkFSEqR2uvWs47P4HakkkNpuIkksYhQbfjoZFgYzZpAacu3ZQSos7KdxJj+lg5HmVhuzC3hi8VY6tg3mjvimf7KLj4skNND4NbUr6No+1NVN1DS3SEyERYsgqfS/pIZcW3N8FBu4maVMqlzEoj9tJTEmC0QgJgbmzCF16uskBS9hVOeDxnGA227z68kLoIOR5kYZ1kKmvZXO8aJyTpZW8uPh002uY0RMOCnT43lwQhwRYcHM+moPR0+dcUNrNc31EhNh0aehJAUvIbXzbSgR6NwF1aoVEhwMV10F2dlgt0N2NqndkklKMq5JPLYQbDYYOxa++grO+PfvvQ5GmtukH8in0mbkWrDbVbO72EbEhPOH6wbw/v1jKC6v4p7/buB0WaUrm6ppblMdkKZWLeSqxKPcXLaQPz8bzNNPw803w7/+Bfn5xvhSUpLxNJWY6LhYBGbOhCNH4LXXvHof7qaDkeY2o3tFAEaKdGeZFppqYLd2vJ58CfuOF3PHW+v494q9ekKD5hMSE+Gaa2DFis6cOgVPPglPPAGnTsHjjxvrWqdOrROIqk2YYFw8c6ZxgZ/SwUhzm9bBxsqB6wd3qTfTQlON79eRBybEse3wKf7xdaaeYaf5hNRUWL4crrvuCB06wKuvwsqV8P33MGUKFBdDeLgRd5x64QUoKDDGlfw0K4MORprbbMg2EqD+8foBLl0fFBYcWLPvkZ5hp5lddffb4sXw5JOZfPIJPPccVFUZQ0Lffw/XXgv79sEvf1lPJbt3G1O+T52C6oWzM2b4VUDSwUhzm/VZBXTv0Ioe4a7dKC8+LpIQxww7i0iLu/80zV2cjQNVz7KbNMkYM1q0CL780njomT0b3nzTSUVPPQU229nrkvwsK4MORppbKKVYl1VQM27kStUz7Lq0C6Fv5zY6K4NmSk4nJNQiYjzkVH//yivGnw89BF98UadwTg6pJJy7LsmPsjLoYKS5RVZeCXnF5W4JRmAEpBuHdGP/iRLKKm1ueQ9Na4kNG5wHouogtWQJLF1qlAOYONGYWaeU8cSUmlrrmk63ksQiFpFEImk/nfCjrAw6GGlusd6xYZ67ghHAmLhIKqrsbDvkvzOMNN/15JPOn4hqB6nERKNctV//Gu6+GyorjXElcASv0v+yKOSuswORn2Vl0MFIc4v1WQVEtQkmLqq1295jVGw4IrBOT2DQTGZjdgGvpe5zOtOzviBVbe5cGDIE0tLg4Yf5aRHs23cYA0tgzKh74w2/yspg2mAkIhEiskRESkTEKiK311PubhGxiUhxra+Eptajuda6rAJGxUYg1elM3KBDWDD9O7dlfXZBw4U1zUMyrIUkvbmWl5bvadbSg4AAY8yoVStjQsM99ziCV3Kyka1h0SIjY0N1YPITpg1GwGtABdAZSAZmi8igesquVUq1qfWV1sx6NBc4fPIMh0+ecWsXXbUxvSLIsBZSabO7/b00rTGWbj6MXRnJfSubufRg714IDv5pYsO339Y6ee21xslPP3VZm83AlMFIRFoDU4CnlVLFSqlVwDLgTm/UozXNBg+MF1Ub3SuS0gobPx7W40aaOWTmGjkYA6R5mUdqT3CYOBEqKuDnP681oaFtW7jiCiMYKUVqqrFdha8z6+Z6/QCbUiqz1rGtQH3rk4eLSB5QAMwHZiqlqppSj4jMAGYAdOzYkbS0tBbfhFkVFxe79f6W/lhOq0DI3bOJE5nu66YDqCo35sYu/G4Dp3oFA+6/P2/T92deR4rtrMs6w2XdA+gSZuGiiACKsraSlmWcb+jeNm/uwLPPDuSZZ3YicpLLL+/Al18O5syZAG68sYrnn/+R4cNP0m3AAPp99RWv/2E5f3rzCp55ZidpaSc9c5PuopQy3RdwOXCszrHpQJqTsnFAL4ynvMHATuCPTa2n9le/fv2UP0tNTXVr/Ve8lKru+e96t75HbYl13s/d9+dt+v7M6/EPt6iL/vw/lV9c7vT8+e5txQqloqKMP2tbvlypwEDjq0MHx/lDh9QKElRUWMk55b0J2Kia+f++KbvpgGKgXZ1j7YCiugWVUgeUUllKKbtSajvwHHBLU+vRXCOvuJz9J0o80kVXbUyvCDZkF2CzK4+9p6bVdaiwlE+3HGba6GgiWgc3+fr61iVdfTV88IExZwGM7rrUzO4kBX7Cougnzjszz5eYNRhlAoEi0rfWsaHAjkZcq6AmdVlL6tGaoXq8aFSsJ4NRJEVlVew+1vT9kjTNVd5aeQARmD6+V7OuP9+U7ylT4PXX4eRJeO89x3TvOz8ncfdsOHq0Ba02D1MGI6VUCfAJ8JyItBaRccBEjPGgs4jIdSLS2fH9RcDTwKdNrUdzjXVZBYQGWRjcvb3H3rP6KWzdAT3FW/OOE0XlfLDhIJOH96Br+1ZueY8HHjAm0lmtcN11kPibYcaJzz93y/t5mimDkcPDQCvgOLAQeEgptUNEoh1riarzYFwJbBOREuBLjODzt4bq8dRNXGg2ZBdwSXQ4wYGe+9Xq1qEVPcJb1WR90DRP++/qLCpsdh6YEOe290hNhY0bjfVHH34IqScuhl69/GaKt2mDkVKqQCk1SSnVWikVrZRa4Dieo4y1RDmO108opTo7ysUppf5PKVXZUD2a650uq2Tn0dMeHS+qNqZXJOuzC6onqWiax5wuq2T+WivXD+5KXMc2bnmP2klXf/tbY7r3lFuE1OGPG4uQiovd8r6eZNpgpPmejOxClPLM+qK6xvSKoKCkgn3Hff8fpeZb5q+1UlRexUMTerul/rrZvx96CIKCjO+TvptBavlY+Pprt7y3J+lgpLnMuqwCggKE4T09v6VDzbiR7qrTPGjNvjxeW7GP4dEduNgN46TOtqHo1g1uvdWIP++8G0CSLCb1zczzV+QDdDDSXGbF7lw6tglh51HPz2qLiQyjc7sQPW6keUyGtZC73llPaaWNHYdPNzkHXWPUN937sceMnrl9WQEsGvgsG74p9PntyHUw0lxizb48MnOLOXqqrFnJIVtKRBjdK5J1Wfl63EjziPQDeVQ51rbZ7M3LQdeQ+qZ7jxgBl10Gr84sZnzmXJ5Us3x+O3IdjDSXSFln7DjZkuSQLTW6VwS5p8s5cUYHI839ItuEAMaixubkoGupxx6D7BNtWFZ57dknfHQ7ch2MNJfYe7wIofnJIV0h3jFutGhPhcefzLQLz2brSUICLfzqyj6k3B/PiBjPjZXOmgXt20MM2bzMY+cWyMnxuQSqOhhpLbb90Ckyc4u5Z1wsj1/d3+P/MKudPmPM6N+Ya/NKV6F24Sgqq2TZ1iPcPLw7v/lZf4//vo8aBdOmwXVtfmAlE9jE8LPOp3a6laQko5yv0MFIa7H31mYTFhzAYz/rxyOJfbwSiADSa01e8FZXoXZhWLrlCGcqbUwbHd1wYTdITDQmNixmKqGU8gq/rjmXGnKtsU25k4kPZqaDkdYihSUVfOr4hNguNMirbYmPiyQowEhLGGDxTleh5v+UUixYl8PAru0Y0sNzaa/qSkyExctCISiYFG7nGJ1JDb2OpOAlxjblPhSIQAcjrYU+3HiQiio7d42N9XZTGBETzrx7RhMgkHhRR689oWn+bduhU+w6epppY6IRce9+XQ1JTIQ5bwdiI4hbo74jqWI+i5aG+FwgAh2MtBaw2RXz11qJj4ugf5e23m4OAJf2iWJE5wA2ZBdSpbci19xg4focWgUFMHFYN283BYA77zSWF63MG8SD9tdI7LHX201qFh2MtGZL3X2cwyfPmOKpqLYxXQMpKKlgzX49ZqS5VvXEhZ8P7eb1bulqqamQl2d8/x8eJfX1nd5tUDPpYKQ127y12XRpF8rPBnb2dlPOMjgqgLYhgXy29Yi3m6L5mWVbj1BaYWPaGO9MXKirOl3QggUQGqpIaLWepNmJpKZ6u2VNp4OR1iz7TxTzw948ksdEExRgrl+j4ADh6kFd+GrHMcqrbN5ujuZHFq7PYUDXdgz14sSFarXz1t10E9x0k7BajWVB4C9ISlI+F5DM9b+I5jPmr7USFCDc5qWprQ25aWhXisqqWJmZ5+2maH5i+6FT/Hj4NLeP7un1iQvOEqgmJ8OJsnZUllaw6C+7SErCpwKSDkZak63el8eCdTnEx0XSsW2It5vj1Lg+UYSHBemuOs1lXv1uL4EWITaqtbeb4jSB6nXXQUS4nRSSSSz8hEWLjHK+QgcjrUkyrIXc/c56Kmx21mUVmDbLQVCAhesGd+WbnbmUVlR5uzmaj1uy6TDf7Mqlyq6Y/t5Gr//eO0ugGhwMU5MsLLVMpvirVSQmGuV8hWmDkYhEiMgSESkREauI3F5PuV+ISIaInBaRQyIyS0QCa51PE5Eyx1blxSKyx3N34X/SD+RRWZ2p2GbuLAc3DenGmUobK3Yf93ZTNB92qLCUpz/dXvPazNk97rgDSu2tWJrexed2fzVtMAJeAyqAzkAyMFtEBjkpFwY8BkQBY4ArgSfqlPmlY6vyNkqp/m5ss98LDQoAvJepuClG94qgU9sQ3VWnNVtBSQV3vbMeu4KQQItXEwE3xqWXQkznM7xvuw1WrvR2c5oksOEinicirYEpwMVKqWJglYgsA+4E/lC7rFJqdq2Xh0UkBfDB9ce+4fvMPDqEBXHvuF6M6xNl6iwHARbhhiFdSVmXw+myStOsC9F8Q2lFFfe+u4FDhWeYf+9oAgMspB/IJz4u0rS/9xYLJN8VyIv/72fkfvosna+/3ttNajQx40ZkIjIcWKOUalXr2BPABKXUTQ1cuxTYrZT6g+N1GjAI48P8HuAppVSak+tmADMAOnbsOGLRokWuuRkTKi4upk2bNk2+7nCxnadWnWFy3yB+3jvYDS1zjdr3t++kjefTy5g+OJhx3f0jGDX35+crzHB/VXbFq5vL2X7Cxi+HhzCis2s+t7vz3hYu7MlFFxURHl7BPfeM5u8RzzH64/Fnldm8uQO7d7dl2rSDbmlDYmJihlJqZLMuVkqZ7gu4HDhW59h0IK2B6+4BDgFRtY6NAdoCIcAvgCKg9/nq6devn/JnqampzbruDx9vU32f+lLlFZW5tkEuVvv+7Ha7unTmd+oX76zzXoNcrLk/P1/R2Pv7btcx9ep3mWpjdoFL339jVr66/pWVKub3n6uUdKtL63bnz27FCqWioow/h3U9pkaxTqmjR52edxdgo2rm//tmHTMqBtrVOdYOI5A4JSKTgBeB65RSNYtLlFLrlFJFSqlypdQ8YDXgO8+uJlFQUsEnmw4xeXj3mh0ufYGIcNPQbvyQeYKXlu/x+iwozTW+3ZnLve9u5B9fZ7ps76rc02W88OUupr65lh1HThNoEdPkXGyM6m0lkpIgvschNjCazK7jITaW1Ke+PWddktmYcswIyAQCRaSvUqo6699QYIezwiJyLfAWcINSaruzMrUojC47rQkWrLNSXmXn3st6ebspTda7Y2tsCl5L3cfcVQe8tvmf5jqzlu+u+b7CMbutOT/TL7YdYdHGQxw9dYbM3LNnnymlml2vtyQmwqIZ33LL3y4BjDVHCdY0kv42lEV/+pbExKu83cR6mfLJSClVAnwCPCcirUVkHDARmF+3rIhcAaQAU5RS6+uc6yAi14hIqIgEikgyMB5Y7v678B8VVXbeW2vl8r5R9OvsO58Uqx0vKgOMTyFmnparNc53u3LJzC0m0GJ8prQrGNyt6el5Xk/bxyMLNvN95gn2Hi9m2qievHzrMEKDzD9r7nwSU+7nI6YQRBX/5lGSWMQikkhMud/bTTsvUwYjh4eBVsBxYCHwkFJqh4hEO9YLVeeheRpoD3xZay3R/xzngoDngRNAHvAoMEkppdcaNcEX249wvKic+3zwqQggPi4Kx/9bPvsfjGYoraji/z7dQd9ObUi5fwx3jY1BgP/tONaken48fIp/fp1Z89oC9IgIY9Lw7qTcH8/jV/f33SfonBwSSeN6vqSQCBJJJZE0yMnxdsvOy6zddCilCoBJTo7nAG1qva63B1QpdQLwoV3gzUcpxdursujdsTXj+3b0dnOaZURMOPeO68XcVVn8K2mYb/4HowHwyrd7OXzyDIsfHMuo2AjGxEUSHGBh7qosJl/SnVGxEQ3Wcfx0GdPf20j7VkEUl1dRZbOf9SFlREy4b/+OREeTau3FasYRxXE+ZgrL+RnXRGc2fK0XmfnJSDOB9VkF/Hj4NPde1guLxXeH2qaM6AFAaYXO4u2rdh45zdxVWdw2qudZQec3P+tH9w6t+NMn26moOv+GimWVNqbPz+BkaSXv3TeaBdN9/CnIidTkuTVdcyncgZ0AprCE1OS53m7aeelgpNUrw1rIU0t/pE1IAJOH9/B2c1qkf+e2tG8VxLosPV7ki+x2xZ+WbKdDqyD+cN1FZ51rHRLIcxMHsfd4MXNW7q+3DqUUTyzeyrZDJ3n5tmEM6taeETHhPJLYx38CUSokzbmKRX/aSmJMFlfzDTfyObagEG554ypTZ/HWwUhzKsNayO1vpbPveDFllXZ2Hj3t7Sa1iMUijIqNYH1WgbebojVRhrWQGe9nsOXgSf584wA6hJ274PrKAZ25fnAXXv5uL89/sfOcqd4Z1kKS567j821HefKai7hmUBdPNd9jztpW4oWrIDsb/v1v/sHj2AhgzBhMva2EaceMNO9KP5BPuaPLwxenuDoTHxfBt7tyyT1dRud2od5ujtYI1R+KyqvsWASiw8PqLTt5eA++3H6MuT9kMW9NNveO60W3Dq3IyitmfnoONrsiQITRsb79e1wfZ9tKMH48S8lh4jArH38VyxtvGOXqrjVKTTWOezPLtw5GmlO9Io09W3whIWpjjell3MO6rAJ+PrSbl1ujNcbijQdrPhQBpGcVMKKeSQp7cosQHFP4bYo3Vx5wUkqdtw5f5jSQXHwxo1rvYta2KNq2hfffh++/P7tI7Scqb3J5N52I/FtEPnNyvJ2I/EVEBtQ69hsR2SYiurvQZL7PPEFggPDghN5+M7g7oGtb2oQEsk6vMzK98iobM/+3iw82HEQAi0BwAx+K4uMiCXGsEQoJtPDWnSPZ+OereO+e0YT6QMZtt7BYSLxCWNzxYex2+OEH+Oijn0472zHWW1z6ZCQivYEHgEudnB4JPIOxmLXaG8DvMXLG/deVbdGaLye/lI82HeLO+Bh+X2ew2JcFBlgYGRvOOj1uZFoZ1kJSdpXzzIaVWPNLmTY6mhuHdGXLwZMNZsseERNOyv3x52TWHt+/IynTzz1+wRg/nsTPfseSD1/m2tsjePRRuPFGSE83TyAC13fTPQZsVUptdHJuOFAO7Kw+oJQ6IyLvYew/pIORSfwndS8BFuGhhN7eborLje4VQdqePeQVlxPlQzn2LgQZ1kJufXMtVXYFVPHH6y7igQnG7+C4PlGNqqO+NUI+v3aoJcYbmbuvku+YNWsqv/0tXHst7NxpnkAEjeymE5E+IlIpIs/WOT5bRIpEZKSIhAB3AAucXL8LeAkjc3aliCgRqX5Y/AAYKCLOnqa0OjKshbyWus9tCT+t+SV8vOkwt4+O9stB/upxow366ch0lm4+7AhEECDUfK+10PDh0Lo1rFzJ44/DoEHGvntXX22eQASNDEZKqX3AXOA3IhIFICL/B9wL3Ox4EooHOgA/OKniLuAA8Bkw1vH1W8e5LcBp4Nrm38aFYUN2AdPmpPOPr/c0O1NxhrWQz/dX1Hvtf1bsI9AiPOyHT0UAg7u3JzTIorvqTCinoAQw/lO64MZ23CkoyNgCduVKUlPh2DHo1AkWLoT552T79J6mTBx4FggAfi8i92GM/9yplPrWcT4eYyLLNifXbgV6ACuUUumOLyuAUsruuCa+mfdwwfjPin1U2OzYVfMSflZPk/1obyW3v3VuMMvOK+GTzYdJHhNDJz98KgJjEHxEjB43Mpv84nLW7i/guou7MLlvkN9MmjGN8eNJ3RZJ0lQ7ixcb40XBwXD33fD55/VflpoKs2Z5pomNDkZKqWPAyxjJRt8EfqWUqj0ZsBtwWilV4eTyQUAwsKme6k84rtfqkZVXwpr9ebX2vhDG9Gra9NQ1+/NqpsmWV9l59bu9lFX+lB7nP6nGU9GDE+Jc02iTGh0bye5jpzlVWuntpmgOH2w4SIXNzm+v7seNvYN1IHKx1DY3kcSHLPpNOomJ0KsX/O1vYLfDzTfDt986ucYx026Uh7J7NnVK9V6McZ+1SqnX6pwLxZig4MwlGE9NW+o5fwYjQ7fmhN2u+P3H2wgNCuCtu0ZyWZ8obErx+baj1bvZNsqJIuPHUz1V9vvME1zxUhofZxzis62H+XjTIX42sLPfPhVVGxMXgVJGt6fmfVU2O++nW7msTxR9OvneFiVml5oKSS8MYVFgMnFjdHgAACAASURBVIknl9Qcf/xx+NWvoKoKbrjh7MwM3pjy3ehg5Ng36E1gLTBORIbWKZIP1PdxZjiwXylVX06ZCIwtHjQnUtbnsD6rgKdvGMhVAzsz/77R3H9ZL95dk82/vmlcJt6c/FIWbTzIyJhwJvcNYvGDl7Jg+hii2obw28VbeXThFpSCb3bm+v1uqMN6diA4wMJ6HYxM4ZuduRw9VcZdY2O83RS/ZGRmEBLHlhkzF2p5+WVjmndFBTz/vHHMW2uPGjub7hJgKcYkhgQgB/hbnWK7gSARcZZRcyC1pnQ70QvQeww5cfjkGV78cheX9Yli6kjjr1ZEeOqGAdw6sievrtjHM5/+eN4Zdkopnlq6nQARXp02nJsc3SCX9o5i6cPjuGlI15qyVTb/33wuNCiAYT076MWvJvHummx6hLfiygGdvd0Uv/Tkk46gMn48ZGRA8U872orAxx/D4MGwYgU8+KD31h41GIxEpA/wP+Br4FHHmNCzwPUiMr5W0eqQO9pJNSeBoY5dV+NFpGaajIh0APrVul5zUErx1JLt2BXMnDwYkVojRiL8bfJgLo2LZN5aKy8tr3+G3dIth/lhbx5PXnsR3Tqc3RtqsQh3j+vl87tbNtWYuAh+PHKa4vIqbzflgrb72GnWZRVwZ3wMAT68RYlPGD8ebDZYu/asw8HBxphRq1bw5ptGQPLGlO/zBiMR6YIRhHYByY6ZbwDvYTwJvVhdVimVDawHbnJS1f8BuRhPV2uBAbXO3QBUAEucXHdBW7rlMGl7TvDktf3pGXFugsgAixDf2wgcCiirtPPdrtyzyhSUVPDXz3cxrGcH7oh33g1SvXLd3/Z1OZ/RvSKw2ZXfd0ma3bw1VkICLSSN7Ontpvi/sWMhIOCcrjqAHTuMUwCvvOKdzN7nDUZKqWNKqTilVIJSqrzWcZtSaoBSqu5C1dnAZBEJq1PPj0qpMUqpVkopUUqtqnX6DmCxUuqsPhMRiRCRJSJSIiJWEbm9vnY6ctwdE5FTIvKOYwFuk+sxk+925fLHT7bTv0tb7hobW2+5cX2iCA2y1Myym7cmm48zDtVMbHj+i52cPlPJi1MGn/eTp7/t69KQETHhBAjMTnPfAmLt/E6VVrJ082EmDetOeOtzt4XQXKxtW7jkknOCUfUY0dKlRryyWGDqVM8HJFcnKJ0PHAYebkxhERkGJGJ0+9X1GsYTU2cgGZgtIoOc1HEN8AfgSiAWiKtTX6PqMZMN2QVMf28jZZV2svNK2HLwZL1lq59qnrimP/+eNoyB3drx28VbuefdDbz09R4+2XSYicO6cVGXdh68A/PbdbQIO5B+oKDZC4i1llmccZAzlTbuulRPXPCYyy+HdeugrAw4e7LClVfC669DURFcdpnn9z5yaTBSStkwsjKUNvKSLsA9jgwPNUSkNTAFeFopVex4kloG3Omkjl8AbyuldiilCoG/Anc3ox5TsNsVz322g+pMKI2ZUFD9VHPT0O58OGMsf7lpIGv25fOfFcZf6xfbjur/bOtIP5Bv9G1idG9+sumQdxt0gbHZFe+ttTIqNpxB3dp7uzkXDqWgvBzCwkjtMo2kiWVnTVb4+muYOBGWLYO//tV5QHLXQlhpyjoVTxGR4cAapVSrWseeACYopW6qU3Yr8Del1IeO11EYi2ijgOgm1DMDmAEQFhY2orS0sfHUtcKvnE67kRNRNmNgXdlt5H7wFBVHdjepng4J99Bu9M2IWFC2Kk6uSuF0+mJ3NNknBXe7iM63vYAEBIEIIsKZrE2cXJUCCKHRgynL2d7kv3etcdqOnETElfdTuPI9Tq/18kY6F4hpGNOhq8dQZvE7LmYD75PGwppSCcBijNWI+4E/AiOA/1fr/CIgCUhz9jYZSqmRzWmfWYPR5RjjSF1qHZuOMYkioU7Z/cAjSqmvHK+DMLrlegE9G1tPbf3791d79nh+pvk/v8nk1e/2cs+4WG4c3JX0rIJmp7w3tllOp7LKTlCg5ayJCWlpaSQkJLi49ebR2PvLsBaSfiCf4T07sP3wKd5ceYCCkgqqh9aC6/y9mYWv//wyrIVMfWMNdgWhQef+Hfv6/Z2PV+8tNhas1nOPx8QYW5Q7pKYaT0dFRfDGG/DAAz8db2jat4g0OxiZdafXYqDuIEc7oKgRZau/L2piPV711soDvPrdXpJG9uDpGwZisUiLdqOsb28X7Se1txW4tE8Ud8TH8HDKJr7PPAH8lP9P/925Vuru4zXd0Prv2INychp1PDHRmMxw7bXwxBMweTL8+KP71x+ZdYfVTCBQRPrWOjYU2OGk7A7Hudrlch2z85pSj1dkWAt5YP5GXvhyFzcM7srMyUOwuGi9xYU2Q66lWocE8qsr+xLo+PsPCLgw1lx5WpXNWCFiuYDWtZlCdHSjj19xhbHmqLjYmNjgiYWwpgxGSqkSjB1hnxOR1iIyDpiIMVuvrveA+0RkoIiEA38G3m1GPR6XYS1k2px0lu/IxSJw11i98M/bRsSE884vRhEUIIyNi9CB3A22HDpJtw6h/PbqfqbsBvVbL7wAYXXWK4aFGceduOceY/Ld9u0wcKD7F8KaMhg5PIyRPPU4sBB4SCm1Q0SiRaRYRKIBHGNFs4BUwOr4eqahejx3G/VLP5BPheNTogAb9Yw3UxjfvyPJY2JYsz+f3NNl3m6OXzl2qox1WQXcOjKaRxL76kDkScnJMGeOMUZU7c9/No47kZoKu3bB0KHG0qRJk85fvTHrrluzczqZNhgppQqUUpOUUq2VUtFKqQWO4zlKqTZKqZxaZf+plOqslGqnlLqnzgJdp/WYQXxcZM1iVd1dYS73XdYLm13x7ppsbzfFr3y+7QhKwc+H6R1jvCI52ZiskJcHgYFQ4DxZcO3JChs3GrnrPv0Upk1zXm11eShp9jRk0wajC0FcVGsUcHmfKN1dYTI9I8K49uIupKRbKdH561zm0y1HGNKjPb2iWnu7KRe2yEi4/npYsMDIV1dL3VlzgYFG5u++feGDD4zuu/rKw6lmTw7TwciLqncb/fVVurvCjO6/PI7TZVUs3njQJfVlWAvPm13d3x04Ucz2w6f4+VD9VGQKd9wBR45AWlrNofqmb4eEwKZN0LMnvPsuPPTQ+cs3h1mndl8Q0g/kExpkYUiPDt5uiubEJdHG1O93Vmdz59jYFk0uqd7yvdJmN+36JXdbtvUIInDjEB2MTOHGG6FdO3j/fWPKHNV7HzkPLG3awJYtxmSGN94w9kBatsx1s+z0k5EXpR/IZ2RMBMGB+sdgVvdf1oucglK+3nGsRfV8v+c45VV27MpIP/TBhpwm7dLr65RSLNtyhDG9IujS3r93EvYZrVrBLbcYGxo5Ms7U7H1Uj4gI2LzZ+POdd4wnJFfNstP/C3pJQUkFu48VER/X/IWtmvtdPagL0RFhvPXDgWbXYbMrVu41NjKufrZavPEQU99Yy6q9eWRkF/h9992OI6c5kFfCxGHdvd0UrbY77jBSLXz2WaMv2b3byOz99NMwe7brkqnqbjovWZ9lJD/VM+jMLcAi3Dsulr98tpMMa2GzutZe+noPWw6e5MHxcbRtFcSImA7sO17Ca6n7uOPtdYgYQcqfu+8+3XKYoADhuou7NFxY85wJE6BHD6Or7tZbGyxed4woMdF1Y0b6ychL0g8U0CooQI8X+YCpI3sSFhzAHz/Z1uSnly+2HWV22n6mje7JH64fwCOJfYiPM1IPpf0ugasGdkYpsCuoqPLPLd/tdsVnW48yoV9HOoTpfYtMxWKB22+Hr76CEyfOW9TZZIXEROO1K7ab0MHIS9IP5DMyNlyPF/mA3ceKqKiyk5lbzC1vrOGxD7bwfeYJSsqrzjtDbvex0zyxeCuXRHfgLz8/dwutkMAAHprQmxDH74BdUZOKyJ+szy7g2OkybtKz6Mzpjjugqgr69zeCU2wspKScVeR8s+ZqByRo37a5zdDddF5QPV6k/3H6hvQD+dgdkw2UgmVbD7N0y2ECBOwACgIDhDfuGMGVA4wF6Cv3nOBXH24mJMjC7DtGEBIY4LTuETHhLJgeT+ru43z141FmLd9DROtgpvrRNtxvr8oi0CJ0bBPScGHN87ZtAxEodHygslphxgzje0d2hvPNsoOfAtIVV7QOc16iYToYecFP40V68oIviI+LJDjQUrMdxzu/GIVNKf6zYl/NWrFKm+K+eRvpGdGKmIjWrN6fh1LGONChwjN0blf/DLLq7OEPJfTmwfcz+N1H29h26BRd2of6fMb1dQfy+WZnLgD3ztvgt2NiPu2pp4xPWbWVlhrHHcHoyScbrsYIVEdym9sM3UfkBdXjRYO76/EiX1C9HcfjV/cn5f54Lu0TxeV9O/LktRcRGmQhQIyg84uxMVzcrT1bDhbW/Nu2NWKn3mqtQwKZ+4uRjOsdyfx0Ky8t3+PzW6J/XGsH3Uo/HRPzefVtLWG1Gl129XTduZp+MvICPV7ke2rvfVT7mLM9ozKshSQ7Frg2NedgSGAAY3tHsnp/Pgoo9/H9fsQxmT1AbxdhXtHRzjfdg5+OO+m6czUdjDwsv7hcjxf5kXqD1PTmb2w4tncUoUH7KKu0oxT0ivTdPG5HTp0hJiKMpFE9fb7L0W+98IIRaEobyHFap+vO1fRHcw9b7xhj0J8Q/VtLNjasfuJ6YHwcYcEBzF11oGZDOl9isyu25Jzksr5RepNHM6u9tYTI2VtM1FVfl54L6GDkYekH8h3ri9p7uymaiY2ICeeP1w/gxSlD2JRzkn+v2OftJjXZ3uNFFJVX6SDkC6q3lrDbjT/rC0j17RbrAjoYeVj6gQJGxoYTFKD/6rWG/XxoNyYP786/V+wlw+p87xmz2phtTLzQwcgHOdsVNiio3l1hXUH/j+hB+cXl7Mkt0l10WpM8O3EQ3cNb8esPtlBUVunt5jTaJmshUW1CiI5o9tITzVvqdt2FhRnTv+Pj3faWpgtGIhIhIktEpERErCJy+3nK/kJEMkTktIgcEpFZIhJY63yaiJQ5tikvFpE9nrkL5/R4kdYcbUODePnW4Rw9VcYjCzbx+f4Kn5junZFTyIiYDoj4X1aJC0Ltrru9e40s3488cu6aJBcxXTACXgMqgM5AMjBbRM7NpWIIAx4DooAxwJXAE3XK/NKxTXkbpVR/N7W5UZZtPUKgRbDZfW8wWvOuETHhTLmkOysz8/h4b6Xp1x+dKCrHml/KyBi9sNsvdOtmdNEtX169pavLmSoYiUhrYArwtFKqWCm1ClgG3OmsvFJqtlLqB6VUhVLqMJACjPNcixsvw1rIVz8eo8quuOud9ab+j0Qzp56O7i6F+ReQVv9+X6LHi/zHww/DiBHGNPCePV2+GNZs64z6ATalVGatY1uBCY28fjywo86xmSLyIrAHeEoplebsQhGZAcwA6NixI2lpTos127wd5VQ/3FZU2ln47QaKensng3FxcbHL789M/PX+Wp2yESBgU0Y3fshJK2lphxq+0AuW7q4g0AIF+7eQltW0bjp//fmB799b9JAh9MrIQE6fNg5Yrdjuu489u3Zx/KqrWlS3mGm3SRG5HFislOpS69h0IFkpldDAtfcAfwWGKaXyHMfGADsxuv1uA/7jOL//fHX1799f7dnj2uGlW99cw7qswpqV6N7M0ZWWlkZCQoJX3tsT/Pn+Uvcc597/buDnw7rxym3Dvd2cek1+fTUWET566NImX+vPPz+fv7fYWOfZGmJiIDsbEclQSo1sTtUe7aZzTChQ9XytAoqBdnUuawcUNVDvJOBF4LrqQASglFqnlCpSSpUrpeYBq4HrXXtXDdt3vJj12YXcMqJ7TX4zPd1Va47E/p0Y1imANfvzTbsQtqzSxo+HT+vfcX9U36JXFyyG9Wg3XSOebloDgSLSVym113F4KOd2vdW+5lrgLeAGpdT2hprATzs/e8yclfsJCbTwx+sGEKnT6GstNK5bIP/ZUs7q/flM6NfR2805x44jp6iw2fV4kT+qL4+dCxbDmmoCg1KqBPgEeE5EWovIOGAiMN9ZeRG5AmPSwhSl1Po65zqIyDUiEioigSKSjDGmtNy9d3G2o6fOsGTzYZJG9tSBSHOJoZ0CaN8qiI8zzDlepBe7+jFni2HDwlyyGNZUwcjhYaAVcBxYCDyklNoBICLRjvVC1WH4aaA98GWttUT/c5wLAp4HTgB5wKPAJKWUR9cavbMqC7uC6ZfHefJtNT8WZBFuGtqV5TuOmXIRbIa1kNjIMKL0hy//4yyP3Zw5LkmearbZdCilCoBJ9ZzLAdrUel3PvoOglDoBjHJ5A5vgZGkFC9blcNOQrjXTcjXNFSZf0oP303P43/ZjJI0yz66wSik25RQyoV8nbzdFc5fkZLdk7jbjk5HfmL/WSkmFjQcTenu7KZqfGd6zA72iWp+1eZ0ZWPNLySuu0F10WpPpYOQmZypsvLsmm8T+HbmoS90JgprWMiLC5OHdWZdVwMGCBvah8aDqxa46GGlNpYORmyzOOEh+SQUPJfTxdlM0PzVpeHcAlm4+7OWW/GSjtZC2oYH07dSm4cKaVosORm6wPiufl5bvoX+XtoyK1Z8QNffoGRFGfFwEn2w+jFkWr2+yFnJJdDgWi06OqjWNDkYulmEtJHnuOk6XVXHgRDGbck56u0maH5t8SQ+y8krYfND7v2enzlSSebxId9FpzaKDkYulH8in0mZ8SrXblamTWWq+77qLuxAcIDz32Q6vJ9/dnFOIUjBSByOtGXQwcrF+ndoCRpqHoECL3rtIc6vM3GKq7IotB095fVuJL7YdQQTM0WGo+RodjFwsO78EgPsu76Vz0GluV/vJu8KL20pkWAv5eNNhlIL75m3w+lOa5nt0MHKxpVsOM7RHe/58w0AdiDS3i4+LJDjQctZrb/hs6xHsjkcis++1pJmTDkYutO94ETuOnGbisO7ebop2gRgRE07K/fFc2jsSu4JObb2TgmdvrpFYv3qLFN09rTWVDkYutHTzESwCNw7t6u2maBeQETHhvDR1KBaBDzcc9Pj77z9RzJoD+dw8XG+RojWf6XLT+SqlFJ9uPcy4PlF0ahvq7eZoF5huHVoxoV9HFmcc5LGr+hIY4LnPma+nGlukPHXDAJ0cVWs2/WTkIptyCjlYcIZJuotO85JbR0WTe7qctD0nPPaeBwtKWbrlMLePjtGBSGsRHYxcZOnmI4QEWrjm4i4NF9Y0N7hyQCei2oTwgQe76mZ/v58AER6YoLdI0VpGByMXqLTZ+WL7Ua4a2Jk2IbrnU/OOoAALt4zoQeqe4+SeLnP7+x09dYaPNh4iaVQPOrfTXdNay+hg5AI/7D1BQUmF7qLTvO7WUT2x2RWLN7r/6ejN7w9gV4oHxustUrSW08HIBZZuPkKHsCAm9Ovo7aZoF7heUa2Jj4vgw40HsdvdlwvhRFE5C9fncPPw7nrjSM0lTBeMRCRCRJaISImIWEXk9vOUvVtEbLW2HC8WkYTm1NVcJeVVfLMzl+sHdz1r8aGmecu00dEcLDjDmv3uWXiaYS3kkQUZVFTZeThRb5GiuYYZBzheAyqAzsAw4AsR2aqU2lFP+bVKqctcVFeTfbMzlzOVNt1Fp5nGNYO60L5VEB9syOGyvlEurTvDWsjtb6VTXmXHIlBQUkGvqNYufQ/twmSqj/Ii0hqYAjytlCpWSq0ClgF3erOu85m3Npt2oYHo7Vs0swgNCuDm4d35ekcuBSUVLq07/UA+FVX2s15rmiuY7cmoH2BTSmXWOrYVmHCea4aLSB5QAMwHZiqlqppal4jMAGYAdOzYkbS0tAYbuzm3ks05xj/22+es5clRofQJD2jwOm8rLi5u1P35Kn1/0FvsVNjs3D17BT/vHeSy38uQk7aa7wMFQk5aSUs75JK6q/nzz8+f762lzBaM2gCn6hw7BbStp/xK4GLACgwCPgSqgJlNrUspNQeYA9C/f3+VkJDQYGNfeX01Ri8g2BSUd4ghwQe2GU9LS6Mx9+er9P0Z3WmyZg3b8mxknlIuS9ETX2nj7xuXM7h7e55yUzJgf/75+fO9tZRHg5GIpFH/U85q4FGgXZ3j7YAiZxcopQ7UerldRJ4DfocRjIqbUldTHT55hu2HThFgEVBKJ4fUTCX9QH7NxkLVW0u4InBsyC6g0qZ49Iq+La7PbreTl5fHyZMnsdl+euJq3749u3btamlTTcmX7y0gIIAOHToQFRWFxeL6ER6PBiOlVML5zjvGeQJFpK9Saq/j8FCgsRMOFMa+dgCZLazrvF5L3YcIvHb7Jew7Xkx8XKRODqmZRnxcJCGBFsqq7IiIyz4o/bA3j+AAC2PiIlpc16FDhxARYmNjCQoKQsT4p1tUVETbtvV1hvg2X703pRSVlZXk5uZy6NAhoqOjXf4epprAoJQqAT4BnhOR1iIyDpiIMRZ0DhG5TkQ6O76/CHga+LQ5dTXFwYJSFm04yG2jorlmUBceSeyjA5FmKiNiwkmZHs/Arm0JCbQwqFvdToLmWZl5gpGx4YQFt/xzbElJCd27dyc4OLgmEGnmJCIEBwfTvXt3SkpK3PIepgpGDg8DrYDjwELgoeqp2CIS7VhLVB2WrwS2iUgJ8CVG8PlbY+pqiX+v2IvFIjyi11hoJjYiJpw/3zCQ0goby3cca3F9x0+XsftYEZf3dd3ibnd092ju486fl9kmMKCUKgAm1XMuB2NiQvXrJ4AnmlNXc2XnlfDxpsPcNTaGLu11Pi7N3OLjIukR3orFGw+1eNPHlXvzABjfz7VrlzQNzPlkZGqvrthLUIDwUILOx6WZn8UiTLmkB6v353H45JkW1fXD3hNEtQlmQBfXdPlpWm06GDXB/hPFLN18mDvjY/QGeprPuGVED5SCjzOavx7Iblf8sDePy/t2xKJXeJtWWloaIkJeXp63m9JkOhg1wavf7SU0KIAHJ+inIs139IwI49LekXyUcajZyVN3HDlNQUkFl7s4vZAv27x5MwEBAYwbN65J1/3lL3/h4osvdlOrfJcORo20dPNhPt1yhGsGdSFS72ip+ZipI3uQU1DKuqyCZl2/cq+xe6wrJy+4REoKxMaCxWL8mZLisbd+6623ePjhh/nxxx99du2Qmehg1AgZ1kIeX7QFgC+3HyXDWujlFmla01w7qCttQwJZnNG8fY5WZp5gYNd2dGxrog9iKSkwYwZYraCU8eeMGR4JSGfOnGHBggVMnz6dW265hbfffvus80eOHCE5OZnIyEjCwsIYNmwYqamppKSk8Oyzz7Jjxw5EBBHh3XffBYzp0x999NFZ9cTGxvLSSy/VvP7nP//JkCFDaN26Nd27d+f+++/n5MmTbr9fTzDdbDoz+urHo1T3blTZXLeaXdM8pVVwADcO7crSzUd4bmJVk3YkLi6vYlNOIfde1suNLQQeewy2bKGVzQYBjcill54O5eVnHysthfvug7featx7DhsGL7/c5KZ+9NFHxMTEMGTIEO68806SkpKYOXMmQUFBlJSUMGHCBDp16sSSJUvo3r07W7duBWDy5Mns27ePzz//vCZHXfv27Rv9vhaLhZdffpm4uDisViuPPvoojz76KPPnt3j5pNfpYNQI2fmlAAQIOu2P5rOmjuzJwvUH+WLbEW4d1fgV9On786m0KSaYrYuubiBq6LgLzZ07lzvvNDYAmDBhAmFhYSxbtowpU6awYMECjh07xtq1a4mKMsbYevc2xpmLiopo06YNgYGBdOnSpcnv+9hjj9V8Hxsby6xZs5g4cSLz5s3z+TVbOhg14HhRGd9nnuBnAzozLLqDTvuj+azhPTvQu2Nr/rs6m7ziikb/Lv+w9wStggIYEevm33vHE8qZxqbMiY01uubqiokBN2bG3rdvH6tXr2bhwoWA0b2WnJzM3LlzmTJlCps3b2bIkCE1gciVVqxYwcyZM9m1axenTp3CZrNRUVHBsWPH6Natm8vfz5N0MGrAu6uzqbTZ+dMNA/QmYppPExEu7R3F/HQrmbl7CA60NCqb98q9ecTHRRASaLLtUV54wRgjKi396VhYmHHcjebOnYvNZjsrP5tSRj/+wYMHa75vKhE559rKysqa761WKzfccAPTp0/nueeeIzIykk2bNjFt2jQqKly7b5U3+PZznZsVlVUyP93KdRd30YFI8wttQ43Pn3YFlY5s3udzsKCUrLwS882iA0hOhjlzjCchEePPOXOM425SVVXFvHnzmDlzJlu2bKn52rp1K0OGDOG///0vl1xyCdu2bat3rU9wcPBZWcqrdezYkaNHj9a8zs3NPev1xo0bqaio4F//+hdjx46lX79+HDlyxPU36SX6yeg8Fq7PoaisSq8r0vzGlQM6M2flAarsCoul4Wze1VO6x/czYTACI/C4MfjU9cUXX5CXl8f06dOJjDz77+62225j9uzZ7NixgxdffJFJkyYxc+ZMevTowfbt22nbti0jR44kNjYWq9XKpk2biI6Opm3btoSEhHDFFVfw2muvcemllxIQEMCf/vQnQkN/Wlzft29f7HY7L7/8MpMnTyY9PZ2XmzH5wqz0k1E9yqtsvL0qi0t7RzKkRwdvN0fTXGJETDgLZ8TTrX0oIYEWoiPCzlt+2ZYjtA0J5FSp73cDucLbb79NYmLiOYEIYOrUqVitVlavXs33339P9+7duemmmxg0aBDPPPNMTWbyKVOmcP3113PllVfSsWPHmrGnf/zjH8TFxZGQkMAtt9zC/fffT6dOnWrqHzJkCK+88gr//Oc/GThwIHPnzj1r2revk+b2b/qz/v37q7++/y1PfryN9+4dbd5Phc3k77tN6vtr2L7jxVz/6g9cNaATryePcFpm4foc/vjJdgBCgxo3vtQUu3btYsCAAecc99U9fxrDH+6tvp8bgIhkKKVGNqde/WRUjzdW7mdQt3Y6/Ynml/p0asNjV/Xly+3H+HL70XPOr9idy9NLf6x53ZjxJU1rCR2M8UEA/wAAEDJJREFUnCitUhw4UcKDE3rrTb80vzXj8jgGd2/P/336IwUlP3XDLdp4kOnvZRAd0YqQQIteX6d5hA5GTuSfUXRuG8J1Fzd9UZqm+YrAAAv/b+oQTp2p5LnPdqCU4t/f7eXJj7Zxae9Ilj16OQumx/P41f1d3kWnaXXp2XRO2BTkl1Sw9dAp/Q9Q82sXdWnHI4l9ePnbvWw5eJLs/FImDevGrFuGEhxoYURMuP43oHmE6Z6MRCRCRJaISImIWEXk9vOUfcOxDXn1V7mIFNU6nyYiZbXO72lsO5RSuo9cuyCMjYtEMNJeBViEO+JjCA403X8Nmp8z42/ca0AF0BlIBmaLyCBnBZVSDyql2lR/AQuBxXWK/bJWmf6NbYTuI9cuFButhdQMjSrV7G0mNK0lTBWMRKQ1MAV4WilVrJRaBSwD7mzCtfNa2o7wENF95NoFIz4ukmA9UUHzMrONGfUDbEqpzFrHtgITGnHtFOAEsLLO8Zki8iKwB3hKKZXWUEXtQ0QHIu2CMSImnJT740k/kK8TAWteY7Zg1AY4VefYKaAxq8R+Abynzl7F+3tgJ0a3323AZyIyTCm1v+7FIjIDmAFGjqg0N2b99bbi4mJ9fz7MXfc3SKAo6xBpWS6v2qn27dtTVFR0znGbzeb0uD/wh3srKytzz78vpZTHvoA0QNXztQoYDpTWuea3wGcN1NsTqALiGij3FfBoQ+3s16+f8mepqaneboJb6fvzDTt37nR6/PTp0+e97u9/V2rFivPXvWKFUc5sGro3X1Dfz00ppYCNqpnxwaNjRkqpBKWU1PN1GZAJBIpI31qXDQV2NFD1XcAapdSBhpoA6FWsmubDRo2CpCRITXV+PjXVOD9qlHve/+67767ZMrz215YtW9zzhh6QlpaGiNSbadwTTDWBQSlVAnwCPCcirUVkHDARaGhP3buAd2sfEJEOInKNiISKSKCIJAPjgeVuaLqmaR6SmAiLFjkPSNWBaNEio5y7XHXVVRw9evSsr4svvrhZddXes+hCZqpg5PAw0Ao4jjFV+yGl1A4AEYl2rBeq2dVKRMYCPTh3SncQ8DzGpIY84FFgklKq0WuNNE0zJ2cByVOBCCAkJIQuXbqc9RUYGEh5efn/b+/uY6uq8zyOv7/2gVagu6Jtl+lKoZTnjdCKVcLCMmEUdk12NJ00k+KEkjIo2BDWByCyQnlwx+rKBmcZCKlExPWBwMzsGp7+kZp2UEGRAXnQytrCIBh1FUEuD6G//ePcXtrSQm17OT2Hzys5sef87r38Pvci395zfuf3Y86cOWRmZpKSksI999xDTU1N7HnV1dWYGVu2bKGgoIDk5GS2b/d+P37rrbe48847SUlJYcCAASxYsKDZonkXLlzgqaeeIjs7mx49epCTk8OLL74IeNeiSktLGTBgAKmpqQwaNIjnnnuOhoaG2PP379/PxIkTSUtLo3fv3owcOZIdO3ZQV1fHT6NvWHp6OmZGSUlJfN/AVnS3AQw45/4PeKCNtqN4gxyaHnsXuGLlO+fcV0CcvqiLSFebMwf27oVLl1JJaOeisj/5CUyaBH37wokTMGwYLF7sbe0xalRstfMuMXfuXDZs2MDatWvJyclh+fLlTJ48mdraWvr27Rt73Lx583jhhRfIzc2ld+/ebN++nSlTprBixQrGjx/P0aNHeeSRRzh//nxsmYipU6dSXV3NihUryMvLo76+nmPHjgHQ0NBAVlYWGzZsID09nV27djFjxgxuvfVWSktLASguLmbkyJHs2rWLxMRE9u/fT0pKCrfffjubNm2isLCQAwcO0KdPH1JTU7vuTWmnbleMRETa65ZbvEJ09Cj06+ftXw/btm2jV6/LvxePGzeOjRs3smrVKiorK7n//vsBWL16NW+//TYrV65k2bJlsceXl5dz3333xfafeeYZnnzySaZNmwbAwIEDqaio4KGHHuL555/ns88+44033mDr1q1MnjwZgJycnNjzk5KSWLJkSWy/f//+7Nmzh9dffz1WjOrr63niiScYOnQoALm5ubHH9+nTB4CMjAxuu82flQpUjESkW2j8hnL6dKTda/40npp7+mlYtQoWLYr/KTqA8ePHs2bNmth+amoqR44c4eLFi4wdOzZ2PCEhgTFjxnDw4MFmzx89uvmSPx9++CG7du2ioqIidqyhoYFIJMLJkyf56KOPuOmmm2Kn01qzevVqKisrqa+vJxKJcPHiRbKzs2Ptjz32GNOnT2fdunVMnDiRwsLCWGHqDrrjNSMRkWtqeo1oyZK2BzXEw80330xubm5sy8rKarx9pNVlZ1oe69mz+ZWFhoYGFi1axN69e2Pbvn37qK2tJT09PfbabXnzzTeZM2cOJSUlbN++nb179zJr1qxm15zKy8s5ePAgDzzwADt37uSOO+5g7dq1HX0LupyKkYgETmuDFa42yu56yM3NJTk5udmAhUuXLvHuu+8yfPjwqz43Pz+fw4cPNytwjVtiYiL5+fk0NDSwo41gNTU13H333ZSVlZGfn09ubi5Hjlxxbz+DBg1i9uzZbN68mdLSUiorKwFITk6O9dcvKkYiEihXGzXnZ0Hq2bMnM2fOZP78+WzZsoVDhw4xc+ZMvvzyS2bNmnXV5y5cuJDXXnuNhQsX8vHHH3P48GE2btzI3LlzAa+IFBUVMX36dDZt2sTnn39OdXU169d7d70MHjyYPXv2sHXrVmpra1m6dCnvvPNO7PUjkQiPPvooVVVV1NXV8f7771NTUxMrktnZ2ZgZmzdv5quvvuLMmTNxepfapmIkIoGye/fVh283FqTdu69vvwAqKiooKipi2rRpjBo1in379rFt27ZmI+laM2nSJDZv3syOHTsoKCigoKCAZ599ln79Ynex8Morr1BcXMzs2bMZOnQoJSUlnDrlzZ728MMPU1RURHFxMXfddRd1dXU8/vjjsecmJCTw7bffMnXqVIYMGcKDDz7ImDFjWL58OQBZWVksXryYBQsWkJmZSVlZWRzenauza52LvBENGTLEffJJeG9HqqqqYsKECX53I26ULxgOHTrEsGHDrjh++vTpdg9gCJowZGvrcwMwsw+dc6NbbbwGfTMSERHfqRiJiIjvVIxERMR3KkYiIuI7FSMR8Y0GUAVLPD8vFSMR8UVSUhKRSMTvbsiPEIlESEpKistrqxiJiC8yMjI4fvw4Z8+e1Tekbs45x9mzZzl+/DgZGRlx+TM0UaqI+CItLQ2AL774otkCc+fOnSMlJcWvbsVVkLMlJSWRmZkZ+9y6moqRiPgmLS3tin/cqqqqyMvL86lH8RXmbJ2l03QiIuI7FSMREfFdtytGZlZmZh+Y2Xkze7kdj/8XMztpZqfMbK2Z9WjS1sfM/mBmP5hZvZkVx7XzIiLSId2uGAFfAMuAa676ZGaTgPnARKA/kAMsbvKQlcAFIBOYAqwysxFd3F8REemkbleMnHO/d879EfimHQ+fCrzknDvgnPsWWAqUAJhZT6AQeNo5d8Y5VwP8D/Cr+PRcREQ6Kuij6UYA/91k/89AppndCvQDLjnnPm3R/g+tvZCZzQBmRHfPm9nHcehvd3Eb8LXfnYgj5Qu2MOcLczaAIR19YtCLUS/gVJP9xp97t9LW2N7qYiLOuTXAGgAz+6Cja3IEgfIFm/IFV5izgZevo8+9rqfpzKzKzFwbW821X+EKZ4CmNyk0/ny6lbbG9tMd+HNERCSOrmsxcs5NcM5ZG9vfd+AlDwAjm+yPBL50zn0DfAokmtmgFu0HOp5ARETiodsNYDCzRDNLARKABDNLMbO2Tie+ApSa2XAzuwX4V+BlAOfcD8DvgSVm1tPMxgI/B9a3oxtrOpujm1O+YFO+4ApzNuhEPutuExSaWTmwqMXhxc65cjPrBxwEhjvnjkYf/xgwD0gFNgGPOOfOR9v64A0RvxdvdN5859xr1yWIiIi0W7crRiIicuPpdqfpRETkxqNiJCIivlMxaiJsc9ldbZ4/M5toZofN7KyZ7TCzbJ+62SFm1sPMXop+TqfN7CMz+8cm7YHOB2Bmr5rZCTP73sw+NbPpTdoCn6+RmQ0ys3Nm9mqTY4HPF72V5ZyZnYlunzRpC3w+ADP7pZkdiv6becTMxkWP/+h8KkbNhW0uu1bn+TOz2/BGGj4N9AE+AN687r3rnETgGN6MGn+Fl2WDmfUPST6A3wD9nXNpwD8Dy8zszhDla7QS2N24E7J8Zc65XtFtCIQnn5ndC1QA0/AmExgP/G+H8znntHmDOHriFaLBTY6tB571u29dkG0Z8HKT/RnAzhbZI8BQv/vayZz78OYjDF0+vGlWTgBFYcoH/BLYAJQDr0aPhSIfUAVMb+V4WPLtBEq7Kp++GV02mNbnsgvyN6O2jMDLBsTuyTpCgLOaWSbeZ3iAEOUzs9+Z2VngMF4x2kJI8plZGrAEeLxFUyjyRf3GzL42sz+Z2YToscDnM7MEYDSQbmafmdlfzOw/zSyVDuZTMbrsR81lF3ChympmScB/Aeucc4cJUT7n3Cy8fo/DO/VxnvDkW4o36/6xFsfDkm8e3rI2WXg3g75lZgMJR75MIAn4Bd7fzVFAHt7EAx3Kp2J02Y00l11osprZTXinUy8AZdHDockH4Jy75LwlUP4WmEkI8pnZKOBnwH+00hz4fADOufedc6edc+edc+uAPwH/RDjyRaL//a1z7oRz7mtgOZ3Ip2J02Y00l12zOf2iaz8NJGBZzcyAl/B+Syt0zl2MNoUiXysSuZwj6Pkm4C2IedTMTgJPAIVmtodw5GuNA4wQ5HPe+nF/wcvUUsfy+X0RrDttwBvA63gX3MbifbUc4Xe/OpEnEUjBG5W1PvpzIpAezVYYPVYBvOd3fzuQbzXwHtCrxfHA5wMy8C7u98Kbp3ES8APe/IphyHcz8DdNtn8HNkazhSHfX0c/s8b/56ZEP78hYcgXzbgEbxRkBnALUI136rVD+XwP1J02vGGIf4z+pTkKFPvdp07mKcf7zaXpVh5t+xneRfEI3qif/n7390dmy47mOYd3WqBxmxKSfOnAO8B3wPfAfuDXTdoDna+VvOVER9OFIV/089uNd2rqO7xfmu4NS75ohiTgd9F8J4EXgZSO5tPcdCIi4jtdMxIREd+pGImIiO9UjERExHcqRiIi4jsVIxER8Z2KkYiI+E7FSEREfKdiJBIQZpZmZuVmNszvvoh0NRUjkeAYDSzCu/NdJFRUjESCIw9vCYmDfndEpKtpOiCRADCzQ8DQFoc3Oed+4Ud/RLqaipFIAJjZXXizyh8A/i16+IRzrt6/Xol0nUS/OyAi7fJnvMX1fuuce8/vzoh0NV0zEgmGEUAysMfvjojEg4qRSDDk463ftNfvjojEg4qRSDDkAUecc9/73RGReFAxEgmG4WhIt4SYBjCIBMN3QL6ZTQJOAbXOuW987pNIl9HQbpEAMLO/A14C7gBSgHHOuRp/eyXSdVSMRETEd7pmJCIivlMxEhER36kYiYiI71SMRETEdypGIiLiOxUjERHxnYqRiIj4TsVIRER89/8nTq/CZu23PwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_multiple_forecasts(X, Y, Y_pred):\n",
    "    n_steps = X.shape[1]\n",
    "    ahead = Y.shape[1]\n",
    "    plot_series(X[0, :, 0])\n",
    "    plt.plot(np.arange(n_steps, n_steps + ahead), Y[0, :, 0], \"ro-\", label=\"Actual\")\n",
    "    plt.plot(np.arange(n_steps, n_steps + ahead), Y_pred[0, :, 0], \"bx-\", label=\"Forecast\", markersize=10)\n",
    "    plt.axis([0, n_steps + ahead, -1, 1])\n",
    "    plt.legend(fontsize=14)\n",
    "\n",
    "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rtAyI4HjFpag"
   },
   "source": [
    "- predicts series of outputs from the beginning by a sequence to sequence RNN. This way, the model inputs and outputs may overlap a lot, which is not an issue as it is a ***casual*** model (i.e., it remembers only past not future). For this\n",
    "    - we can use `TimeDistributed()`, a wrapper to apply a layer to every temporal slice of the input like a convolutional layer applied to the time axis usually axis 1.\n",
    "    - For evaluation, we use only the values at the last time steps. So, a custom metric should be defined.\n",
    "    - Use some error bars along with predictions, e.g., by firstly adding MC dropout on each layer. Then, during predictions, compute different statistics at each time step.\n",
    "- The above models can be combined to generate mildly long sequences, e.g., texts or music. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "35w2TVBCFpag",
    "outputId": "95db1284-530e-483e-d91b-9cfaa491591f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((7000, 50, 1), (7000, 50, 10))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# An RNN that predicts the next 10 steps at each time step\n",
    "\n",
    "np.random.seed(42)\n",
    "\n",
    "n_steps = 50\n",
    "series = generate_time_series(10000, n_steps + 10)\n",
    "X_train = series[:7000, :n_steps]\n",
    "X_valid = series[7000:9000, :n_steps]\n",
    "X_test = series[9000:, :n_steps]\n",
    "Y = np.empty((10000, n_steps, 10))\n",
    "for step_ahead in range(1, 10 + 1):\n",
    "    Y[..., step_ahead - 1] = series[..., step_ahead:step_ahead + n_steps, 0]\n",
    "Y_train = Y[:7000]\n",
    "Y_valid = Y[7000:9000]\n",
    "Y_test = Y[9000:]\n",
    "\n",
    "X_train.shape, Y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hYhvGRZ9Fpah",
    "outputId": "7c877dbf-54d8-4335-aab8-b0f4e89b8418"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 5s 649us/sample - loss: 0.0498 - last_time_step_mse: 0.0388 - val_loss: 0.0416 - val_last_time_step_mse: 0.0321\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 3s 486us/sample - loss: 0.0389 - last_time_step_mse: 0.0282 - val_loss: 0.0378 - val_last_time_step_mse: 0.0307\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 3s 495us/sample - loss: 0.0327 - last_time_step_mse: 0.0215 - val_loss: 0.0291 - val_last_time_step_mse: 0.0177\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 3s 490us/sample - loss: 0.0291 - last_time_step_mse: 0.0178 - val_loss: 0.0273 - val_last_time_step_mse: 0.0162\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 3s 486us/sample - loss: 0.0273 - last_time_step_mse: 0.0154 - val_loss: 0.0260 - val_last_time_step_mse: 0.0143\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 3s 485us/sample - loss: 0.0245 - last_time_step_mse: 0.0117 - val_loss: 0.0232 - val_last_time_step_mse: 0.0107\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 3s 490us/sample - loss: 0.0224 - last_time_step_mse: 0.0098 - val_loss: 0.0255 - val_last_time_step_mse: 0.0146\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 3s 500us/sample - loss: 0.0211 - last_time_step_mse: 0.0085 - val_loss: 0.0196 - val_last_time_step_mse: 0.0070\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 3s 485us/sample - loss: 0.0203 - last_time_step_mse: 0.0079 - val_loss: 0.0198 - val_last_time_step_mse: 0.0078\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 3s 486us/sample - loss: 0.0197 - last_time_step_mse: 0.0076 - val_loss: 0.0185 - val_last_time_step_mse: 0.0064\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 3s 490us/sample - loss: 0.0194 - last_time_step_mse: 0.0073 - val_loss: 0.0193 - val_last_time_step_mse: 0.0076\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 3s 486us/sample - loss: 0.0191 - last_time_step_mse: 0.0071 - val_loss: 0.0187 - val_last_time_step_mse: 0.0072\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 3s 494us/sample - loss: 0.0187 - last_time_step_mse: 0.0067 - val_loss: 0.0180 - val_last_time_step_mse: 0.0062\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 3s 487us/sample - loss: 0.0190 - last_time_step_mse: 0.0071 - val_loss: 0.0180 - val_last_time_step_mse: 0.0063\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 3s 486us/sample - loss: 0.0188 - last_time_step_mse: 0.0069 - val_loss: 0.0186 - val_last_time_step_mse: 0.0065\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 3s 487us/sample - loss: 0.0188 - last_time_step_mse: 0.0069 - val_loss: 0.0183 - val_last_time_step_mse: 0.0068\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 4s 520us/sample - loss: 0.0187 - last_time_step_mse: 0.0070 - val_loss: 0.0175 - val_last_time_step_mse: 0.0060\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 3s 498us/sample - loss: 0.0184 - last_time_step_mse: 0.0068 - val_loss: 0.0177 - val_last_time_step_mse: 0.0063\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 4s 508us/sample - loss: 0.0182 - last_time_step_mse: 0.0065 - val_loss: 0.0181 - val_last_time_step_mse: 0.0068\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 4s 517us/sample - loss: 0.0181 - last_time_step_mse: 0.0065 - val_loss: 0.0180 - val_last_time_step_mse: 0.0067\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "# `TimeDistributed(Dense(n))` is equivalent to Dense(n), since it only applies to the last axis.\n",
    "# But we use it to emphasise that the models at each time step are separated to each other.\n",
    "\n",
    "def last_time_step_mse(Y_true, Y_pred):\n",
    "    return keras.metrics.mean_squared_error(Y_true[:, -1], Y_pred[:, -1])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=keras.optimizers.Adam(lr=0.01), metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "DR7hHO5mFpah",
    "outputId": "dd04015f-6a7c-461b-99a3-8632ac975a4c"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEUCAYAAACGWlk5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3hUVfrA8e87qQQCpNAhCaEJKEVaEBUiVhRBShSxrAXWsu66rsuq6Oq6uvhjXVddEUV0RQQ0WFDRtZKIAqGEjkAoyQQCBFIgjbSZ8/vjTmIICSRhyp3hfJ4nD5l7zz33XDLJO/fcc84rSik0TdM0zZMsnm6ApmmapulgpGmapnmcDkaapmmax+lgpGmapnmcDkaapmmax+lgpGmapnmcDkaapmmax5kyGInI70Rkg4iUici7Zyn7RxE5IiInROQdEQmqsS9cRD4VkWIRsYrIrS5vvKZpmtZopgxGwCHgOeCdMxUSkWuAx4DRQAwQC/ytRpE5QDnQDpgKzBWRvi5or6ZpmnYOxMwrMIjIc0BnpdRv6tm/GMhQSj3heD0aWKSUai8izYF84EKlVJpj/0IgSyn1mFsuQNM0TWsQf0834Bz1BT6r8XoL0E5EIoAowFYViGrsH1lXRSIyHZgOEBwcPCgqKso1LTYBu92OxWLWm+Jzp6/Pu/ny9fnytQGkpaXlKKXaNOVYbw9GLYATNV5XfR9ax76q/aF1VaSUmgfMA+jVq5favXu3c1tqIsnJyYwaNcrTzXAZfX3ezZevz5evDUBErE091ttDdBHQssbrqu8L69hXtb/QDe3SNE3TGsHbg9EOoH+N1/2BbKVULpAG+ItIj1r7d7ixfZqmaVoDmDIYiYi/iAQDfoCfiASLSF1diu8B94hIHxEJA54E3gVQShUDnwDPikhzERkBjAMWuuUiNE3TtAYzZTDCCConMYZt3+b4/kkRiRKRIhGJAlBKfQ3MBpIAq+Pr6Rr1PAA0A44CS4D7lVL6zkjTNM1kTDmAQSn1DPBMPbtb1Cr7EvBSPfXkAeOd2TZN0zTN+cx6Z6RpmqadR3Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN43Qw0jRN0zxOByNN0zTN40wbjEQkXEQ+FZFiEbGKyK31lHvDkYq86qtMRApr7E8WkdIa+3e77yo0TdO0hjBl2nGHOUA50A4YAHwpIluUUjtqFlJK3QfcV/VaRN4F7LXq+p1Sar5rm6tpmqY1lSnvjESkOTAReEopVaSU+hn4HLi9gcctcH0rNU3TNGcxZTACegI2pVRajW1bgL5nOW4icAxYWWv7LBHJEZFVIjLKec3UnGldei4v/G8nqdZ8TzdF0zQ3E6WUp9twGhG5DFiqlGpfY9s0YKpSatQZjvsB+Ekp9UyNbcOAXzC6/G4BXgMGKKX21Tp2OjAdoE2bNoMSExOddj1mU1RURIsWLTzdjFNsyq7g1U3lKCDAAn8ZEkz3ML8m1WXG63MmfX3ey5evDSA+Pj5VKTW4Kcea9ZlREdCy1raWQGEdZQEQkS7ASGBaze1KqbU1Xi4QkSnAGOA/tcrNA+YB9OrVS40aNaqpbTe95ORkzHR9q/fl8NaKDVR9LKqwQ1nraEaN6t6k+sx2fc6mr897+fK1nSuzdtOlAf4i0qPGtv7AjnrKA9wBrFZK7T9L3QqQc2yf5gRKKeYm7+O2+WsJCwkgyN9S/YOJiQjxaNs0TXMvUwYjpVQx8AnwrIg0F5ERwDhg4RkOuwN4t+YGEWktIteISLCI+IvIVOBy4BsXNV1roJ/2HOOqf6/k/77exXUXdeCbP45k8bQ47hvVDX8/4ee9OZ5uoqZpbmTWbjqAB4B3gKNALnC/UmqHiERhPAPqo5TKBBCR4UBnYGmtOgKA54ALABuwCxivlNJzjTxozb4c7nh7HQrwtwh3XxJDiyB/BkWHMSg6jMLSCj5cf4Dfj+5Bh1bNPN1cTdPcwJR3RgBKqTyl1HilVHOlVJRSarFje6ZSqkVVIHJsW+MoV1irjmNKqSFKqVClVGulVJxS6jt3X4t2qjd+3F/9fEgpRUp63in7f3t5N+wK5q08W4+rpmm+wrTBSPNN+cXlrM/IwyLgJxDgbyEuNuKUMl3CQxg/oBNL1mWSU1TmoZZqmuZOOhhpbjUnaS+lFTb+NXkAj1zdi0X3xjEoOuy0cg/Ed6Os0s7bP6d7oJWa5iaLFkFMDFgsxr+LFp15uw8z8zMjzcccyCvhvTVWJg3qzE0Xdzpj2W5tWjDmog4sXGPlvsu70SokwE2t1DQXWLQIZs5kZGYmREXB888b26dPh5IS43ur1Xi9ahUsWHD6doCpU93fdjfRd0aa2/zr292IwB+v6tmg8g+O6k5RWSUL1mS4tF2a5lKLFhnBxGpFlDKCyz33wLRpvwacKiUlMHdu3dtnznRfmz1AByPNLbZnnWDZ5kPcfWnXBo+Q69OxJaMvaMu8lft4+bs0vUyQ5p1mzjwluMzmzySVDYeTJ+s9JIlRzObPp27MzKy7sI/QwUhzi//7ehdhIQHcP6pbo467sk87ispsvPLDHqbOT9EBSfM+tYLIENaTQCJJjKqzeJJlNAkkMoT1p+6IinJRA81BByPN5VamHeOnPTn87ooetAxu3LOfvOJywFg2o7zSTsr+XBe0UNNcqFYQiSeZRBJIkKUkBV17yr6koGtJCPyUxKA7iCf51x0hIb8+Z/JROhhpLrUhI49Hl26hTWggt8U1/pNdXGwEwf7G29SuoEOrYGc3UdNc6/nnjWBSQ3zIOhIf30xC4KcktbsFREhqd4sRiL4KJf7t2yA6GsSxQNYtt/j04AXQwUhzoVRrPlPeSuFoYRnHSyrYnlXQ6DoGRYexaFoc942MJTwkkNlf7+bwifr72jXNdKZOhXnzIDoaJWIEmXnziH/+ShI/C2ZSxRIeetBOgm0JiZ8FEx/vOCYjA2w2GD4cvv76jM+YfIEORprLpOzPpcJmrLVgt6smd7ENig7jset68/69wygqq+Su/66noLTCmU3VNNdyBJcfV6wwgozjLickxBjb8Npr0L59HceJwKxZcOgQzJlDUhLMnu3WlruNDkaaywztGg4YS6TXtdJCY/Xp2JLXp17M3qNF3PbWWv6zYo8e0KCZ1uzZkJRU//6vv4bLLoOyMujcGbZvh+uugx9+qFVw5Ei45hqSnv2JhMl2hgxxabM9RgcjzWWaBxpzqsdc1L7elRYa6/KebfjtyFi2Zp3gX9+m6RF2mmkNGQIJCXUHpPffhzFjoKICnn4a9u6Fa681AtO118JXX51aPummV0gonE9i6TjiR/vmqgw6GGkusz7DWAD18TG9nRKIqoQE+lfnPdIj7DSzio+HxMTTA9JLL8Htt4NSxtiGp5+GoCAjAN1/P1RWwtix8OGHRvmkJEj4czSJlinEFy83DqxalcGHApIORprLrEvPo1PrZnQOc26ivLjYCIIcI+wsIufc/adprlIzIK1YAW++Gcuf/mTs+7//gyee+LWsCLz+OvzjH2C3w5Qp8OCDxrGJIXcRb6/Vf+djqzLoYKS5hFKKtel51c+NnKlqhF37lkH0aNfCqXddmuZsVQFp/Hj44IMo/P3hn/+EGTPqLv/44/DOO8Yaqa+/btwtxR/9sO7CPrQqgw5Gmkuk5xSTU1TmkmAERkC6oV9H9h0rprTC5pJzaJqzxMfDH/5gfP/YY/Doo2cuHxNjdN2BMdIuqe3NdRf0oVUZdDDSXGKdI2Geq4IRwLDYCMor7Ww9eMJl59A0Z0hKgjfegNtvz+CNN848yi4pyeiaW7QImjeHiy+GhJL/nrZag6+tyqCDkeYS69LziGwRSGxkc5edY0hMGCKwVg9g0ExmQ0Yec5L2kmrNrw4uiYlw990ZdQ5qqFKz7Pjx8MADxraX5gT/uloDGH14b7zhU6symDYYiUi4iHwqIsUiYhWRW+sp9xsRsYlIUY2vUY2tR3Outel5DIkJR6qWM3GB1iGB9GoXyrqMvLMX1jQ3SbXmk/DmGl78Zjc3PZnGhEl2EhONrjqof5RdzUBUVfaRRyAwEJKTIfGzYBJsS0j6a5IxwiE62u3X5kqmDUbAHKAcaAdMBeaKSN96yq5RSrWo8ZXcxHo0J8g6fpKs4ydd2kVXZVjXcFKt+VTY7C4/l6Y1xLJNWdgVnLRGkPXxAO6Yebg6uFSpKyCtX39qIAJjVYZp0+C99yA21ti/3n+4EaE++8x9F+UGpgxGItIcmAg8pZQqUkr9DHwO3O6JerTGWe+G50VVhnaNoKTcxvYs/dxIM4e0bGMNxoojreg0cTN3TKx7akNVQFrvyBQxYwanBS2AP//ZGPY9e7axf8ZTQXDFFUYwUspVl+F2Zk073hOwKaXSamzbAoysp/xAEckB8oCFwCylVGVj6hGR6cB0gDZt2pCcnHzOF2FWRUVFLr2+ZdvLaOYP2bs3cizNdd10AJVlxi/jkh/Wc6JrIOD66/M0fX3mdajIztr0k1zayY/2tx3ggnA/CtO3kJxu7K99bSIwdKjRDVeXJUu6cMEFhVx9dVveeqs98fEpREaW07F3b3p+/TXrFixgVf4Adu0KZcqUAy6/PpdSSpnuC7gMOFJr2zQguY6ysUBXjLu8i4BfgMcbW0/Nr549eypflpSU5NL6r3gxSd3133UuPUdN8bXO5+rr8zR9feb1yIeb1QVP/k/lFpXVub+x17ZihVKRkUq9/75Sfn5KPfKIY8fBg0qBWnHP+yoy0ihnBsAG1cS/+6bspgOKgJa1trUECmsXVErtV0qlK6XsSqltwLPApMbWozlHTlEZ+44Vu6WLrsqwruGsz8jDZvedLgvN+xzML+GzzVlMGRpFePNAp9RZ1ZX38MMwerQxgO7YMaBTJ5J6/paEBWNOe87krcwajNIAfxHpUWNbf2BHA45VUL102bnUozVB1fOiITHuDEYRFJZWsutI4/MlaZqzvLVyPyIw7fKuTq235rOlkhJ4+WXHyLuD/yKxcgLxFxx26vk8xZTBSClVDHwCPCsizUVkBDAO43nQKUTkOhFp5/j+AuAp4LPG1qM5x9r0PIIDLFzUqZXbzll1F7Z2vx7irXnGscIyPlh/gAkDO9OhVTOn1x8fDx9/bAyie/FFxxDwV7ON1OTLlzv9fJ5gymDk8ADQDDgKLAHuV0rtEJEox1yiqnUwRgNbRaQY+Aoj+PzjbPW46yLON+sz8rg4KoxAf/e9tTq2bkbnsGbVqz5omrv9d1U65TY7vx0Z67JzxMfDpElQXm4Mpou/uyt07eozQ7zNOpoOpVQeML6O7ZlAixqvHwXqXempvno05ysoreCXwwX8YXSPsxd2smFdI0jafbRqkIqmuU1BaQUL11gZc1EHYtu0OPsBTZSUBN9+C8HBRvxJShbib7zReJBUVAQtXHdudzDznZHmZVIz8lHKPfOLahvWNZy84nL2Hi1y+7m189vCNVYKyyq5f2Q3l52j5uoMt94Kfn6OCbNRdxoZ+b791mXndhcdjDSnWZueR4CfMLCL+1M6VD830l11mhut3pvDnBV7GRjVmgtd9Jy09jJBkyYZAxkefhgSZg0gqcVY+Pxzl5zbnXQw0pxmxa5s2rQI4pfD7h/VFh0RQruWQfq5keY2qdZ87nhnHSUVNnZkFZBqzXf6Oepar270aGjdGtLSIDFRSChdQNJ7B4zFU704HbkORppTrN6bQ1p2EYdPlDJ1fopLfjHPREQY2jWCtem5+rmR5hYp+3OodMxts9ntpLhg9fi61qsLDIRx44znRpdYl5DIzaxXg7w+HbkORppTLFprZJxUQEWla34xz2Zo13CyC8o4dlIHI831IloY2e8ECPC3EBcb4fRz1Lde3eTJcOIE/PCXb4iv/I4Z/PPXnV6ajlwHI80p9hwtRAA/cd0v5tnEOZ4bJe4ud/udmXb+2WQ9TpC/hd+P7s6ie+MYFO2+Z6VXXgmtWsHSo/Us1+mF6ch1MNLO2baDJ0jLLuKuETE8cnUvt/9iVik4WQHAhmybR7oKtfNHYWkFn285xE0DO/HHq3q5/f0eFAQ33gjLLBMoJ+D0Al6YjlwHI+2cvbcmg5BAPx6+qicPxnf3SCACSKkxeMFTXYXa+WHZ5kOcrLAxZajn/uhPngzH7a1YETTm1B1emo5cByPtnOQXl/OZ4xNiy+A6PqG5UVxsBAF+xrKEfhbPdBVqvk8pxeK1mfTp0JJ+nd237FVtV10FoaGwNO7FX++EmjeHefO8Mh25DkbaOflwwwHKK+3cMTzG001hUHQYC+4aip9A/AVtPHaHpvm2rQdPsPNwAVOGRSHi2nxd9Zk9G9ascXTVbetOxV4rTJwI4eHGrFiMYeGzZ3ukeU2ig5HWZDa7YuEaK3Gx4fRqH+rp5gBwSfdIBrXzY31GPpU6FbnmAkvWZdIswI9xAzp6rA1Dhhjzjy64APLyHKnLr7wSDhyAPXuq5ycNGeKxJjaaDkZakyXtOkrW8ZOmuCuqaVgHf/KKy1m9Tz8z0pyrauDCjf07erRbuiqtxCuvGGvVffQRRr8dkPT6L6dNlPUGOhhpTbZgTQbtWwZzVZ92nm7KKS6K9CM0yJ8vthzydFM0H/P5lkOUlNuYMszzo9WqApJS8OGHUBkVS1K7W0iYG+91gQh0MNKaaN+xIn7ak8PUYVEE+JnrbRToJ1zdtz1f7zhCWaXN083RfMiSdZn07tCS/h4cuFBTfDw8/jgUFMBddwsJJ+aRGHAb8ZdVerppjWauvyKa11i4xkqAn3CLB4e2nsnY/h0oLK1kZVqOp5ui+YhtB0+wPauAW4d28djAhbrMmAEBAfD++3D/9ZnEFy+HDRs83axG08FIa7RVe3NYvDaTuNgI2oQGebo5dRrRPZKwkADdVac5zas/7MHfIsRENvd0U06RkgIi0LIlzE3uTRLx8P33nm5Wo+lgpDVKqjWf37yzjnKbnbXpeaZd5SDAz8J1F3Xgu1+yKSn3vi4LzVw+3ZjFdzuzqbQrpr23wTTv+6pRc3/4g9FVN+sFCwn+H5O01Pt6BEwbjEQkXEQ+FZFiEbGKyK31lLtTRFJFpEBEDorIbBHxr7E/WURKHanKi0Rkt/uuwvek7M+homqlYpu5VzkY268jJytsrNh11NNN0bzYwfwSnvpsW/Vrs6zuUTO9xCOPGNuOHIHEiYkkbJ1J0lcnPdvARjJtMALmAOVAO2AqMFdE+tZRLgR4GIgEhgGjOT0N+e+UUi0cX71c2GafFxzgB7h2pWJnGdo1nLahQbqrTmuyvOJy7nhnHXYFQf4Wjy4EXFPtPEft28PQobB8OcTf241EEkiYYjHmH3kJ/7MXcT8RaQ5MBC5UShUBP4vI58DtwGM1yyql5tZ4mSUiiwAvG9ToPX5My6F1SAB3j+jKiO6Rpl7lwM8iXN+vA4vWZlJQWuHx5Yo071JSXsnd767nYP5JFt49FH8/Cyn7c4mLjfD4+76uPEc33ABPPw3Z3UcQH7SGxKvfZv36B7xmiLeYMRGZiAwEViulmtXY9igwUik19izHLgN2KaUec7xOBvpifJjfDcxUSiXXcdx0YDpAmzZtBiUmJjrnYkyoqKiIFi1aNPq4rCI7M38+yYQeAdzYLdAFLXOOmte397iN51JKmXZRICM6+UYwaurPz1uY4foq7YpXN5Wx7ZiN3w0MYlA753xud+W17d3bgmnTBvPnP+/isRW3EZiXx4Z33nHJueoTHx+fqpQa3KSDlVKm+wIuA47U2jYNSD7LcXcBB4HIGtuGAaFAEHAnUAh0O1M9PXv2VL4sKSmpScc99vFW1WPmVyqnsNS5DXKymtdnt9vVJbN+UHe+s9ZzDXKypv78vEVDr++HnUfUqz+kqQ0ZeU49/4b0XDXmlZUq+i/L1aIUq1PrduXPzm5XqnNnpW66SSn1wgtKgVKHD7vsfHUBNqgm/t036zOjIqBlrW0tMQJJnURkPPACcJ1SqnooiVJqrVKqUClVppRaAKwCxtRXj1a3vOJyPtl4kAkDO1VnuPQGIsLY/h35Ke0YL36z2zSjoLRz8/0v2dz97gb+9W2a03JXZReU8vxXO5n85hp2HCrA3yKmWXOxIUSMrrpvv4XSqmzHHTpATIxXpCE3azBKA/xFpEeNbf2BHXUVFpFrgbeAsUqpbXWVqUFhdNlpjbB4rZWySjt3X9rV001ptG5tmmNTMCdpr0665yNmf7Or+vvycxjd9uXWQ9z5zjqu/vePDPvHD7y1cj+OwaIopUwxaq4xxo6F4mL48YU1v260WmH6dNMHJFMGI6VUMfAJ8KyINBeREcA4YGHtsiJyBbAImKiUWldrX2sRuUZEgkXEX0SmApcD37j+KnxHeaWd99ZYuaxHJD3bec8nxSpHC0sB41OIWYblak33w85s0rKL8LcYnyntCi7q2PjleV5P3suDizfxY9ox9hwtYsqQLrx88wCCA8wzaq6x4uOhmZzki7KrTt1RUgIzZ3qmUQ1kymDk8ADQDDgKLAHuV0rtEJEox3yhqnVongJaAV/VmEv0P8e+AOA54BiQAzwEjFdK6blGjfDltkMcLSzjHi+8KwKIi43E8XfLK//AaL8qKa/kr5/toEfbFiy6dxh3DI9GgP/tONKoerZnneClb9OqX1uAzuEhjB/YiUX3xvHI1b1YdG+cx0fNNVazZnCV+pYvGMtpQ9MyMz3RpAYz5dBuAKVUHjC+ju2ZQIsar+sduKiUOgZ4UUYP81FK8fbP6XRr05zLe7TxdHOaZFB0GHeP6Mr8n9P5d8IAr/sDo/3qle/3kHX8JEvvG86QmHCGxUYQ6Gdh/s/pTLi4E0Niws9ax9GCUqa9t4FWzQIoKquk0mY/5UPKoOgwr36P3BC+hs/zxrGdC7mI7b/uiDLnOpJVzHxnpJnAuvQ8tmcVcPelXbFYvPdR28RBnQEoKdereHurXw4VMP/ndG4Z0uWUoPPHq3rSqXUznvhkG+WVZ06oWFphY9rCVI6XVPDePUNZPM1774Lqc8PfjM/fX1BjFkxICDz/vIda1DA6GGn1SrXmM3PZdloE+TFhYGdPN+ec9GoXSqtmAaxN18+LvJHdrnji0220bhbAY9ddcMq+5kH+PDuuL3uOFjFv5b5661BK8ejSLWw9eJyXbxlA346tGBQdxoPx3X0iEM2ebazM0OF3ExncNZflQRONHX5+MG8eTJ1q6lTkOhhpdUq15nPrWynsPVpEaYWdXw4XeLpJ58RiEYbEhLMuPc/TTdEaKdWaz/T3U9l84DhP3tCb1iGnT7ge3bsdYy5qz8s/7OG5L385bcRkqjWfqfPXsnzrYWZccwHX9G3vrua7TVUq8qQkuOHOCFLKB3H0H/PBZoNLLzV9KnLTPjPSPCtlfy5lji6PqiGu3v7pMS42nO93ZpNdUEq7lsGebo7WAFUfisoq7VgEosJC6i07YWBnvtp2hPk/pbNgdQZ3j+hKx9bNSM8pYmFKJja7wk+EoTHe/T6uT1Xm14QEeO45IwPsV5VX8xsg6Y3dJMyPNnUGWB2MtDp1jTBytnjDgqgNNayrcQ1r0/O4sX9HD7dGa4ilGw5UfygCSEnPY1A9gxR2ZxciOIbw2xRvrtxfRyl1xjq8Xc2AFBEByzd3Jrr5DSS8PJzEr8wbiMAF3XQi8h8R+aKO7S1F5BkR6V1j2x9FZKuI6O5Ck/kx7Rj+fsJ9I7v5zMPd3h1CaRHkz1o9z8j0yiptzPrfTj5YfwABLAKBZ/lQFBcbQZBjjlCQv4W3bh/Mhiev5L27hhJsohW3Xa0qIBUXw+dfCAkV75MY+aCpAxE4+c5IRLoBvwUuqWP3YOBpjMmsVd4A/oKxZtx/ndkWrekyc0v4aONBbo+L5i+1HhZ7M38/C4NjwlirnxuZVqo1n0U7y3h6/UqsuSVMGRrFDf06sPnA8bOulj0oOoxF98adtrL25b3asGja6dt9WXw8jB8PH3wAVw44RPzmhXBktpFrwqSc3U33MLBFKVVXAvaBQBnwS9UGpdRJEXkPI/+QDkYm8VrSHvwswv2junm6KU43tGs4ybt3k1NURqQXrbF3Pki15nPzm2uotCugksevu4DfjjTegyO6RzaojvrmCHn73KHGSkoyMo8HBsKnv/QiiVHE//QTTJ7s6abVq0HdYyLSXUQqRORvtbbPFZFCERksIkHAbcDiOo7fCbyIsXJ2hYgoEfnIsfsDoI+I1HU3pdWSas1nTtJel62vZs0t5uONWdw6NMonH/JXPTdar++OTGfZpixHIAI/ofp7rXFqJt5LSICAQCGBpSQtPuzppp1Rg4KRUmovMB/4o4hEAojIX4G7gZscd0JxQGvgpzqquAPYD3wBDHd8/cmxbzNQAFzb9Ms4P6zPyGPKvBT+9e3uJi/4mWrNZ/m+8nqPfW3FXvwtwgM+eFcEcFGnVgQHWHRXnQll5hUDxh+l8+HZjivUzgA7aRIUFQkzenxCwhe3mTrza2MGDvwN8AP+IiL3YDz/uV0p9b1jfxzGQJatdRy7BegMrFBKpTi+rABKKbvjmLgmXsN547UVeym32bGrpi34WTVM9qM9Fdz61unBLCOnmE82ZTF1WDRtffCuCIyH4IOi9XMjs8ktKmPNvjyuu7A9E3oE+MygGXeqHYgArr4aWrSAva0Gk2ibRMJku2kDUoODkVLqCPAyxmKjbwK/V0rVTIfaEShQSpXXcXhfIBDYWE/1xxzHa/VIzylm9b6cGrkvhGFdGzc8dfW+nOphsmWVdl79YQ+lFb8uj/NaknFXdN/IWOc02qSGxkSw60gBJ0oqPN0UzeGD9Qcot9n509U9uaFboA5ETVBXKvJmzYwcR5/uvZDLWEniH1NYv95zbTyTxg6p3oPx3GeNUmpOrX3BGAMU6nIxxl3T5nr2n8RYoVurg92u+MvHWwkO8OOtOwZzafdIbEqxfOvhqmy2DXKs0PjxVA2V/THtGFe8mMzHqQf5YksWH288yFV92vnsXVGVYbHhKGV0e2qeV2mz836KlUu7R9K9rfelKFH0wJoAACAASURBVDGLGTPqnkc0aRIcOx7IT/5XEH/8U2bMcH/bGqLBwciRN+hNYA0wQkT61yqSC9T3cWYgsE8pVd+aMuEYKR60Oixal8m69Dyeur4PV/Zpx8J7hnLvpV15d3UG//4u7ewVYAzXTtxwgMHRYUzoEcDS+y5h8bRhRIYG8aelW3hoyWaUgu9+yfb55HMDurQm0M/COh2MTOG7X7I5fKKUO4ZHe7opPum664x1Uj9qcz+sXOnp5tSroaPpLgaWYQxiGAVkAv+oVWwXECAida2o2YcaQ7rr0BXQOYbqkHX8JC98tZNLu0cyebDxXysizLy+NzcP7sKrK/by9GfbzzjCTinFzGXb8BPh1SkDGevoBrmkWyTLHhjB2H4dqstW2nw/+VxwgB8DurTWk19N4t3VGXQOa8bo3u083RSfFBICY8bAJwVXYtuwCYqKPN2kOp01GIlId+B/wLfAQ45nQn8DxojI5TWKVoXcoXVUcxzo78i6Gici1cNkRKQ10LPG8ZqDUoqZn27DrmDWhIsQqfHESIR/TLiIS2IjWLDGyovf1D/CbtnmLH7ak8OMay+gY+tTe0MtFuE3I7p6dXbLphgWG872QwUUlVV6uinntV1HClibnsftcdH4eXGKErObNAmOFIey2j4M1qw5+wEecMZgJCLtMYLQTmCqY+QbwHsYd0IvVJVVSmUA66BmEo1qfwWyMe6u1gC9a+y7HigHPm3SFfiwZZuzSN59jBnX9qJL+OkLRPpZhLhuRuBQQGmFnR92Zp9SJq+4nL8v38mALq25La7ubpCqmeu+ltflTIZ2DcdmVz7fJWl2C1ZbCfK3kDC4i6eb4tPGjIHgYMVHMtm0XXVnDEZKqSNKqVil1CilVFmN7TalVG+lVO2JqnOBCSISUque7UqpYUqpZkopUUr9XGP3bcBSpdQpfSYiEi4in4pIsYhYReTW+trpWOPuiIicEJF3HBNwG12PmfywM5vHP9lGr/ah3DE8pt5yI7pHEhxgqR5lt2B1Bh+nHqwe2PDcl79QcLKCFyZedMZPnr6U16UhBkWH4ScwN9l1E4i1MztRUsGyTVmMH9CJsOanp4XQnCc0FK69VvjY/2bsP9Y1FdTznL1A6UIgC3igIYVFZAAQj9HtV9scjDumdsBUYK6I9K2jjmuAx4DRQAwQW6u+BtVjJusz8pj23gZKK+xk5BSz+cDxestW3dU8ek0v/jNlAH06tuRPS7dw17vrefHb3XyyMYtxAzpyQfuWbrwC89t5uBA7kLI/r8kTiLVzszT1ACcrbNxxiR644EpVSfcmTYKsinasTVFQWnpKGTMk3XNqMFJK2TBWZShp4CHtgbscKzxUE5HmwETgKaVUkeNO6nPg9jrquBN4Wym1QymVD/wd+E0T6jEFu13x7Bc7qFoJpSEDCqruasb278SH04fzzNg+rN6by2srjP/WL7ce1n9sa0nZn2v0bWJ0b36y8aBnG3SesdkV762xMiQmjL4dW3m6OT6tKuley5YQ6FfJRxU3GqMaYmJg0SLTJN2TxsxTcRcRGQisVko1q7HtUWCkUmpsrbJbgH8opT50vI7EmEQbCUQ1op7pwHSAkJCQQSUlDY2nzhU2ehotB49D2YwH68puI/uDmZQf2tWoelqPuouWQ29CxIKyVXL850UUpCx1RZO9UmDHC2h3y/OIXwCIICKcTN/I8Z8XAUJw1EWUZm5r9P+71jChg8cTPvpe8le+R8GaxLMfoJ2jUQSRSH/2cIROZBCDAF8xigkkUkYCkOyME6UqpQY35UCzJtdrAZyote0EUNeMuNplq74PbUw9Sql5wDyAXr16qd273T/S/KXv0nj1hz3cNSKGGy7qQEp6nrHk/b92NrouI81yChWVdgKCA/nkgzcZFG380icnJzNq1Cgnt948Gnp9qdZ8UvbnMrBLa7ZlneDNlYE063oxVY/WAv0tphzQ4e0/v1RrPpPfWI1dQYfRv2HFkjdO+T/29us7E09eW1L7KYzLfpNCWrKBwRTRgjtJ5H/tfk/8EeesEVRzxG9jmTUYFQG1H3K0BAobULbq+8JG1uNRb63cz6s/7CFhcGeeur4PFoucUzbK+nK7aL+qmVbgku6R3BYXzQOLNvJj2jHg1/X/9P+dcyXtOlrdDa3/j90n/uiHvE8R4/icP/AKe+hBIgnEH/0RWOLp5pk2GKUB/iLSQym1x7GtP7CjjrI7HPsSa5TLVkrlikhpI+rxiFRrPvNW7uObHdlcf1EHZk3oh8VJ8y3Otxwu56p5kD+/H92DVXtzqLQr/PzOjzlX7lZpM2aIWM6jeW2mEBXFjdbl9CCNNVzC4/yDeJIhyhwDSEyZ7lspVYyREfZZEWkuIiOAcRij9Wp7D7hHRPqISBjwJPBuE+pxu1RrPlPmpfDNjmwsAncM1xP/PG1QdBjv3DmEAD9heGy4DuYusPngcTq2DuZPV/c0ZTeoz3r+eZKCruUoxkoX/+EhkoKuheef93DDDKYMRg4PYCyeehTjHvJ+pdQOEYkSkSIRiQJQSn0NzAaSAKvj6+mz1eO+y6hfyv5cyh2fEgXYoEe8mcLlvdowdVg0q/flkl1QevYDtAY7cqKUtel53Dw4igfje+hA5EZJHaeSEPgpn7S9nwFsIoIcEvw+JqnjVE83DTBxMFJK5SmlxiulmiulopRSix3bM5VSLZRSmTXKvqSUaqeUaqmUuqvWBN066zGDuNiI6smqurvCXO65tCs2u+Ld1RmebopPWb71EErBjQN0xhh3qs519FkwV2Qv4ZHXu2OlK48OSiIhAVPkODJtMDofxEY2RwGXdY/U3RUm0yU8hGsvbM+iFCvFev06p/ls8yH6dW5F18jmnm7KeaOupHs33xNKx+Bcvl8bSuISmykCkg5GHlSVbfQPV+ruCjO697JYCkorWbrhgFPqS7Xmn3F1dV+3/1gR27JOcGN/fVfkTnUl3QsMhIcmZPF9+eWE799AYiIeT7pn1tF054WU/bkEB1jo17m1p5ui1eHiKGM04jurMrh9eMw5DS6pSvleYbObdv6Sq32+5RAicEM/HYzcqb5ker/9Zw+eW1zEv2dX8O7euhPzuZO+M/KglP25DI4OJ9Bf/xjM6t5Lu5KZV8K3O46cUz0/7j5KWaUduzKWH/pgfWajsvR6O6UUn28+xLCu4bRv5duZhL1FWMdm3N0nhcX7hnJo30lPN0cHI0/JKy5n15FC4mKbPrFVc72r+7YnKjyEt37a3+Q6bHbFyj1GIuOqe6ulGw4y+Y01/Lwnh9SMPJ/vvttxqID9OcWMG9DJ003RHGbPhqHjOlCJP3MeTa+zjDsXUNXddB6yLt1Y/FSPoDM3P4tw94gYnvniF1Kt+U3qWnvx291sPnCc+y6PJbRZAIOiW7P3aDFzkvZy29trETGClC933322OYsAP+G6C9t7uimag7GAah8uDVzL3C/78kQxNK8xrqTmwAd30HdGHpKyP49mAX76eZEXmDy4CyGBfjz+ydZG3718ufUwc5P3MWVoFx4b05sH47sTF2ssPZT851Fc2acdSoFdQXmlb6Z8t9sVX2w5zMiebWgdovMWmUV8PCQmCtukP/kVoSx47ddV0uoagedqOhh5SMr+XAbHhOnnRV5g15FCyivtpGUXMemN1Tz8wWZ+TDtGcVnlGUfI7TpSwKNLt3BxVGueufH0FFpB/n7cP7IbQY73gF2Bvw+uwLEuI48jBaWM1aPoTCc+Hj5+7TD+VPD84wXYxJ+k9lNIGFfq1kAEupvOI6qeF+lfTu+Qsj8Xu2OwgVLw+ZYslm3Owk/ADqDA309447ZBjO5tLLWycvcxfv/hJoICLMy9bRBB/n511j0oOozF0+JI2nWUr7cfZvY3uwlvHshkH0rD/fbP6fhbhDYtgs5eWHO7K5qt4XHe5+/qr1zL/9icPYDEoJuIP3QbRj5S99DByAN+fV6kBy94g7jYCAL9LUY6Dn8L79w5BJtSvLZib/VcsQqb4p4FG+gS3ozo8Oas2peDUsZzoIP5J2nXsv4RZFUL2t4/qhv3vZ/Knz/aytaDJ2jfKtjrV1xfuz+X737JBuDuBet99pmYV5s5k2fI5CMm8T1XMZGPiC/7GmbuhKnuC0a6j8gDqp4XXdRJPy/yBlXpOB65uheL7o3jku6RXNajDTOuvYDgAAt+YgSdO4dHc2HHVmw+kE/VqG1bAzL1Vmke5M/8OwczolsEC1OsvPjNbq9Pif5xjQy6FT76TMzrZWbyIyM5Rhtakc/HTOQ//A6sViMbrMVSnRUWXDfCTt8ZeYB+XuR96krHUV/OqFRrPlMdE1wbu+ZgkL8fw7tFsGpfLgoo8/J8P+IYzO6n00WYVlLbm0nIfpVEEjhBSybyCb/nFVpxnDus7xuFrFaYPp2kX9qRMO9Kl4yw08HIzXKLyvTzIh9Sb5Ca1vTEhsO7RRIcsJfSCjtKQdcI713H7dCJk0SHh5AwpIvXdzn6oqQkSCj5r/GMqCwZgAXcwR0s5De8SwuKmMAyZvNn/EsqmPXCQBK/r3tgg7G2Xcd2TW2L/mjuZusczxj0J0TfNig6jAfjuzfpj2/VHddvL48lJNCP+T/vr05I501sdsXmzONc2iOyyf8XmuvUXMk7/u3bIDoaRLgtehWv8TsUFiazlK+4Fn8qeJR/8bj9+XoDUUICQHFJU9ujg5GbpezPdcwvauXppmgmNig6jMfH9OaFif3YmHmc/6zY6+kmNdqeo4UUllXqIGRSpyygOnUqZGSA3Q4ZGTwQ/RV/5yns+DGFJTzD33ieJ5hlmXna6t6nTo49UXj6mRpGByM3S9mfx+CYMAL89H+9dnY39u/IhIGd+M+KPaRa8zzdnEbZkGEMvNDByJxmzDjDPKLnn+fJkH9zI59RQGsKacnf+SsX9qxg/Hj45hujmDMnx+q/iG6UW1TG7uxC3UWnNcrfxvWlU1gz/vDBZgpLKzzdnAbbaM0nskUQUeEhnm6K1lhTp5L08GestlzGk/ydVhznar5jx9E2FBTAddfB2LHOXaXBdMFIRMJF5FMRKRYRq4jceoayd4pIqogUiMhBEZktIv419ieLSKkjTXmRiOx2z1XUTT8v0poiNDiAl28eyOETpTy4eCPL95V7xXDv1Mx8BkW3RsT3VpXwdUlJGKPmvo/g7+opPk2sZJWMYHHXmXz7jaJ/f1i+HO6/33mrNJguGAFzgHKgHcb037kicvpaKoYQ4GEgEhgGjAYerVXmd4405S2UUr1c1OYG+XzLIfwtgs3ufQ+jNc8aFB3GxIs7sTIth4/3VJh+/tGxwjKsuSUMjtYTu71NXV1v8ZMjSXxwJVNS/8S29zdz8CA89RTMneu8DLGmCkYi0hyYCDyllCpSSv0MfA7cXld5pdRcpdRPSqlypVQWsAgY4b4WN1yqNZ+vtx+h0q644511pv5DoplTF0d3l8L8E0ir3t8X6+dFXuVMz4DiXx7H453f49GF/Xm8/G88+5yFRD9jHTtnBCSzzTPqCdiUUmk1tm0BRjbw+MuBHbW2zRKRF4DdwEylVHJdB4rIdGA6QJs2bUhOrrNYky3YUUZVKrXyCjtLvl9PYTfPrGBcVFTk9OszE1+9vmYnbPgJ2BSIQNBxK8nJB89+oAcs21WOvwXy9m0mOb1x3XS++vMD81/bBx904YknChE5Tu1mbtrUmuePTeef/InKggBAEZ/9AUsC8ph4wyc89dyeczu5Uso0X8BlwJFa26YByQ049i7gIBBZY9swIBQIAu4ECoFuZ6urZ8+eytkS3lilov+yXMU+tlz1evIrtSEjz+nnaKikpCSPndsdfPn6VuzKVjF/Wa5+v2Sjp5tyRjfN+VlNfH1Vk4715Z+ft17bihVKRUYqtaLdLUoZ6wWf8rWi3S0qMlIpaLVbNfHvv1u76RwDClQ9Xz8DRUDLWoe1dASRM9U7HngBuE4plVO1XSm1VilVqJQqU0otAFYBY5x7VWe392gR6zLymTSoU/X6Znq4q9YU8b3aMqCtH6v35Zp2ImxphY3tWQX6Pe5DquckHf2wzv3xRz90zDNq3uShk27tplNKjTrTfsczI38R6aGUqrrn68/pXW81j7kWeAu4Xim17WxN4NfMz24zb+U+gvwtPH5dbyL0MvraORrR0Z/XNpexal8uI3u28XRzTrPj0AnKbXb9vMiHzJjh+CYqylinrraoKMczpkPZTT2HqQYwKKWKgU+AZ0WkuYiMAMYBC+sqLyJXYAxamKiUWldrX2sRuUZEgkXEX0SmYjxT+sa1V3GqwydO8ummLBIGd9GBSHOK/m39aNUsgI9Tzfm8SE929WHPPw8htW5+QkKM7efIVMHI4QGgGXAUWALcr5TaASAiUY75QlGOsk8BrYCvaswl+p9jXwDwHHAMyAEeAsYrpdw61+idn9OxK5h2Waw7T6v5sACLMLZ/B77ZccSUk2BTrfnERIQQqT98+Z6pU2HevOp17IiONl47Ie+R2UbToZTKA8bXsy8TaFHjdb3TrZRSx4AhTm9gIxwvKWfx2kzG9utQPSxX05xhwsWdeT8lk/9tO0LCEPNkhVVKsTEzn5E923q6KZqrTJ3qkqR7Zrwz8hkL11gpLrdx36hunm6K5mMGdmlN18jmpySvMwNrbgk5ReW6i05rNB2MXORkuY13V2cQ36sNF7SvPUBQ086NiDBhYCfWpudxIK/Jq/Y7XdVkVx2MtMbSwchFlqYeILe4nPtHdfd0UzQfNX5gJwCWbcrycEt+tcGaT2iwPz3atjh7YU2rQQcjF1iXnsuL3+ymV/tQhsToT4iaa3QJDyEuNpxPNmVVTfL2uI3WfC6OCsNi0Yujao2jg5GTpVrzmTp/LQWllew/VsTGzOOebpLmwyZc3Jn0nGI2HfD8++zEyQrSjhbqLjqtSXQwcrKU/blU2IxPqXa7MvVilpr3u+7C9gT6Cc9+scPji+9uysxHKRisg5HWBDoYOVnPtqGAscxDgL9F5y7SXCotu4hKu2LzgRMeTyvx5dZDiIA5Ogw1b6ODkZNl5BYDcM9lXfUadJrL1bzzLvdgWolUaz4fb8xCKbhnwXqP36Vp3kcHIydbtjmL/p1b8eT1fXQg0lwuLjaCQH/LKa894Ysth7A7bonMnmtJMycdjJxo79FCdhwqYNyATp5uinaeGBQdxqJ747ikWwR2BW1DPbMEz55sY2F9P9Hd01rT6GDkRMs2HcIicEP/Dp5uinYeGRQdxouT+2MR+HD9Abeff9+xIlbvz+WmgTpFitZ0plubzlsppfhsSxYjukfSNjTY083RzjMdWzdjZM82LE09wMNX9sDfz32fM19PMlKkzLy+t14cVWsyfWfkJBsz8zmQd5LxuotO85Cbh0SRXVBG8u5jbjvngbwSlm3O4tah0ToQaedEByMnWbbpEEH+Fq65sL2nm6Kdp0b3bktkiyA+cGNX3dwf9+Enwm9H6hQp2rnRwcgJKmx2vtx2mCv7tKNFkO751DwjwM/CpEGdSdp9lOyCUpef7/CJk3y04SAJQzrTrqXumtbOjQ5GTvDTnmPkFZfrLjrN424e0gWbXbF0g+vvjt78cT92pfjt5TpFinbudDBygmWbDtE6JICRPdt4uinaea5rZHPiYsP5cMMB7HbXrYVwrLCMJesyuWlgJ504UnMK0wUjEQkXkU9FpFhErCJy6xnK/kZEbDVSjheJyKim1NVUxWWVfPdLNmMu6nDK5ENN85QpQ6M4kHeS1ftcM/E01ZrPg4tTKa+080C8TpGiOYcZH3DMAcqBdsAA4EsR2aKU2lFP+TVKqUudVFejffdLNicrbLqLTjONa/q2p1WzAD5Yn8mlPSKdWneqNZ9b30qhrNKORSCvuJyukc2deg7t/GSqj/Ii0hyYCDyllCpSSv0MfA7c7sm6zmTBmgxaBvuj07doZhEc4MdNAzvx7Y5s8orLnVp3yv5cyivtp7zWNGcw251RT8CmlEqrsW0LMPIMxwwUkRwgD1gIzFJKVTa2LhGZDkwHaNOmDcnJyWdt7KbsCjZlGr/st85bw4whwXQP8zvrcZ5WVFTUoOvzVvr6oJvYKbfZ+c3cFdzYLcBp78ug47bq7/0Fgo5bSU4+6JS6q/jyz8+Xr+1cmS0YtQBO1Np2Agitp/xK4ELACvQFPgQqgVmNrUspNQ+YB9CrVy81atSoszb2lddXYfQCgk1BWetoRnlBmvHk5GQacn3eSl+f0Z0mq1ezNcdG2gnltCV64ips/N+Gb7ioUytmumgxYF/++fnytZ0rtwYjEUmm/rucVcBDQMta21sChXUdoJTaX+PlNhF5FvgzRjAqakxdjZV1/CTbDp7AzyKglF4cUjOVlP251YmFqlJLOCNwrM/Io8KmeOiKHudcn91uJycnh+PHj2Oz/XrH1apVK3bu3HmuTTUlb742Pz8/WrduTWRkJBaL85/wuDUYKaVGnWm/4zmPv4j0UErtcWzuDzR0wIHCyGsHkHaOdZ3RnKS9iMCcWy9m79Ei4mIj9OKQmmnExUYQ5G+htNKOiDjtg9JPe3II9LMwLDb8nOs6ePAgIkJMTAwBAQGIGL+6hYWFhIbW1xni3bz12pRSVFRUkJ2dzcGDB4mKinL6OUw1gEEpVQx8AjwrIs1FZAQwDuNZ0GlE5DoRaef4/gLgKeCzptTVGAfySkhcf4BbhkRxTd/2PBjfXQcizVQGRYexaFocfTqEEuRvoW/H2p0ETbMy7RiDY8IICTz3z7HFxcV06tSJwMDA6kCkmZOIEBgYSKdOnSguLnbJOUwVjBweAJoBR4ElwP1VQ7FFJMoxl6gqLI8GtopIMfAVRvD5R0PqOhf/WbEHi0V4UM+x0ExsUHQYT17fh5JyG9/sOHLO9R0tKGXXkUIu6+G8yd2u6O7RXMeVPy+zDWBAKZUHjK9nXybGwISq148CjzalrqbKyCnm441Z3DE8mvat9HpcmrnFxUbQOawZSzccPOekjyv35ABweU/nzl3SNDDnnZGpvbpiDwF+wv2j9HpcmvlZLMLEizuzal8OWcdPnlNdP+05RmSLQHq3d06Xn6bVpINRI+w7VsSyTVncHhetE+hpXmPSoM4oBR+nNn0+kN2u+GlPDpf1aINFz/A2reTkZESEnJwcTzel0XQwaoRXf9hDcIAf943Ud0Wa9+gSHsIl3SL4KPVgkxdP3XGogLzici5z8vJC3mzTpk34+fkxYsSIRh33zDPPcOGFF7qoVd5LB6MGWrYpi882H+Kavu2J0BktNS8zeXBnMvNKWJue16TjV+4xssc6c/CCUyxaBDExYLEY/y5a5LZTv/XWWzzwwANs377da+cOmYkORg2Qas3nkcTNAHy17TCp1nwPt0jTGufavh0IDfJnaWrT8hytTDtGnw4taRNqog9iixbB9OlgtYJSxr/Tp7slIJ08eZLFixczbdo0Jk2axNtvv33K/kOHDjF16lQiIiIICQlhwIABJCUlsWjRIv72t7+xY8cORAQR4d133wWM4dMfffTRKfXExMTw4osvVr9+6aWX6NevH82bN6dTp07ce++9HD9+3OXX6w6mG01nRl9vP0xV70alzXmz2TXNXZoF+nFD/w4s23SIZ8dVNiojcVFZJRsz87n70q4ubCHw8MOweTPNbDbwa8BaeikpUFZ26raSErjnHnjrrYadc8AAePnlRjf1o48+Ijo6mn79+nH77beTkJDArFmzCAgIoLi4mJEjR9K2bVs+/fRTOnXqxJYtWwCYMGECe/fuZfny5dVr1LVq1arB57VYLLz88svExsZitVp56KGHeOihh1i48JynT3qcDkYNkJFbAoCfoJf90bzW5MFdWLLuAF9uPcTNQxo+gz5lXy4VNsVIs3XR1Q5EZ9vuRPPnz+f2240EACNHjiQkJITPP/+ciRMnsnjxYo4cOcKaNWuIjDSesXXrZjxnLiwspEWLFvj7+9O+fftGn/fhhx+u/j4mJobZs2czbtw4FixY4PVztnQwOoujhaX8mHaMq3q3Y0BUa73sj+a1BnZpTbc2zfnvqgxyisob/F7+ac8xmgX4MSjGxe97xx3KyYYumRMTY3TN1RYdDS5cGXvv3r2sWrWKJUuWAEb32tSpU5k/fz4TJ05k06ZN9OvXrzoQOdOKFSuYNWsWO3fu5MSJE9hsNsrLyzly5AgdO3Z0+vncSQejs3h3VQYVNjtPXN9bJxHTvJqIcEm3SBamWEnL3k2gv6VBq3mv3JNDXGw4Qf4mS4/y/PPGM6KSkl+3hYQY211o/vz52Gy2U9ZnU8roxz9w4ED1940lIqcdW1FRUf291Wrl+uuvZ9q0aTz77LNERESwceNGpkyZQnm5c/NWeYJ339e5WGFpBQtTrFx3YXsdiDSfEBpsfP60K6hwrOZ9JgfySkjPKTbfKDqAqVNh3jzjTkjE+HfePGO7i1RWVrJgwQJmzZrF5s2bq7+2bNlCv379+O9//8vFF1/M1q1b653rExgYeMoq5VXatGnD4cOHq19nZ2ef8nrDhg2Ul5fz73//m+HDh9OzZ08OHTrk/Iv0EH1ndAZL1mVSWFqp5xVpPmN073bMW7mfSrvCYjn7at5VQ7ov72nCYARG4HFh8Kntyy+/JCcnh2nTphERcer/3S233MLcuXPZsWMHL7zwAuPHj2fWrFl07tyZbdu2ERoayuDBg4mJicFqtbJx40aioqIIDQ0lKCiIK664gjlz5nDJJZfg5+fHE088QXDwr5Pre/Togd1u5+WXX2bChAmkpKTwchMGX5iVvjOqR1mljbd/TueSbhH069za083RNKcYFB3GkulxdGwVTJC/hajwkDOW/3zzIUKD/DlR4v3dQM7w9ttvEx8ff1ogApg8eTJWq5VVq1bx448/0qlTJ8aOHUvfvn15+umnq1cmnzhxImPGjGH06NG0adOm+tnTv/71L2JjYxk1ahSTJk3i3nvvpW3bttX19+vXj1de/mgMpAAAENlJREFUeYWXXnqJPn36MH/+/FOGfXs7aWr/pi/r1auX+vv73zPj4628d/dQ834qbCJfzzapr+/s9h4tYsyrP3Fl77a8PnVQnWWWrMvk8U+2ARAc0LDnS42xc+dOevfufdp2b8350xC+cG31/dwARCRVKTW4KfXqO6N6vLFyH307ttTLn2g+qXvbFjx8ZQ++2naEr7YdPm3/il3ZPLVse/Xrhjxf0rRzoYNRHUoqFfuPFXPfyG466Zfms6ZfFstFnVrx18+2k1f8azdc4oYDTHsvlajwZgT5W/T8Os0tdDCqQ+5JRbvQIK67sPGT0jTNW/j7Wfjn5H6cOFnBs1/sQCnFf37Yw4yPtnJJtwg+f+gyFk+L45Grezm9i07TatOj6epgU5BbXM6Wgyf0L6Dm0y5o35IH47vz8vd72HzgOBm5JYwf0JHZk/oT6G9hUHSY/h3Q3MJ0d0YiEi4in4pIsYhYReTWM5R9w5GGvOqrTEQKa+xPFpHSGvt3N7QdSindR66dF4bHRiAYy175WYTb4qIJ9DfdnwbNx5nxHTcHKAfaAVOBuSLSt66CSqn7lFItqr6AJcDSWsV+V6NMr4Y2QveRa+eLDdZ8qh+NKtXkNBOadi5MFYxEpDkwEXhKKVWklPoZ+By4vRHHLjjXdoQFie4j184bcbERBOqBCpqHme2ZUU/AppRKq7FtCzCyAcdOBI4BK2ttnyUiLwC7gZlKqeSzVdQqSHQg0s4bg6LDWHRvHCn7c/VCwJrHmC0YtQBO1Np2AmjILLE7gffUqbN4/wL8gtHtdwvwhYgMUErtq32wiEwHpoOxRlSyC1f99bSioiJ9fV7MVdfXV6Aw/SDJ6U6vuk6tWrWisLDwtO022/+3d/fBUVV5Gse/P0kgEciOaBIZRgIhvGoJREEpBhYLFHat2tFiKjUV3CFUWBRkKNYXsGSFgLhjdGQLZxkoCikR15cUuLNr8fYPiUUGFRQzIAFFRoKD4KqrAhIiS87+cTtNJ3RIDOncvpfnU3WL9H3pnKcb+HXfe+455+OuD4MwZDt79mxi/n055zpsASoA18xSCQwHzjQ55mHgzRae9wbg/4DcFvbbCvympXYOGDDAhVl5ebnfTUgo5QuG6urquOtPnjx5yeNKS53bvv3Sz719u7dfsmkpWxA097455xzwnmtjfejQa0bOuXHOOWtm+TnwMZBiZv1jDhsK7G/hqX8N7HTO/aWlJgC6i1UkwEaMgIICKC+Pv7283Ns+YkRifn9RUVF0yvDYpaqqKjG/sANUVFRgZs2ONN4RkqoDg3Pue+ANYImZdTWz0cAvgJbm1P018GLsCjP7iZlNNLM0M0sxsynAWGBbApouIh3kjjugrCx+QWooRGVl3n6JMmHCBI4fP95ouemmm9r0XLFzFl3JkqoYRcwC0oH/weuqPdM5tx/AzHpH7heKzmplZqOAn3Fxl+5UYClep4avgN8A9zjnWn2vkYgkp3gFqaMKEUCXLl24/vrrGy0pKSnU1dUxd+5csrOzSUtL4/bbb6eysjJ63I4dOzAzNm/ezMiRI+ncuTPbtnmfj998801uueUW0tLS6Nu3LwsWLGg0ad4PP/zA448/Tk5ODl26dCE3N5fnn38e8K5FFRcX07dvX9LT0+nfvz/PPPMM9fX10eP37dvH+PHjycjIoHv37gwdOpTy8nKOHDnCHZEXLDMzEzOjqKgosS9gHMnWgQHn3P8C9zSz7SheJ4fYdW8DF81855z7EkjQF3URaW9z50JVFZw/n06nVk4q+9OfwsSJ0LMnHD8OgwfD4sXe0hrDhkVnO28X8+bNo6ysjLVr15Kbm8uyZcuYNGkShw4domfPntH95s+fz3PPPUdeXh7du3dn27ZtTJkyheXLlzN27FiOHj3KAw88QF1dXXSaiKlTp7Jjxw6WL1/O8OHDqamp4bPPPgOgvr6eXr16UVZWRmZmJrt27WLGjBlce+21FBcXA1BYWMjQoUPZtWsXKSkp7Nu3j7S0NG644QY2btzI5MmT2b9/Pz169CA9Pb39XpRWSrpiJCLSWtdc4xWio0ehd2/vcUfYunUr3bpd+Fw8ZswYNmzYwMqVK1mzZg133303AKtWrWL79u2sWLGCpUuXRvcvKSnhrrvuij5+6qmnePTRR5k2bRoA/fr1o7S0lPvuu49nn32WTz75hNdee40tW7YwadIkAHJzc6PHp6amsmTJkujjPn36sGfPHl599dVoMaqpqeGRRx5h0KBBAOTl5UX379GjBwBZWVlcd50/MxWoGIlIUmj4hnLqVG2r5/xpODX3xBOwciUsWpT4U3QAY8eOZfXq1dHH6enpHD58mHPnzjF69Ojo+k6dOjFq1Ciqq6sbHX/rrY2n/Hn//ffZtWsXpaWl0XX19fXU1tZy4sQJPvjgA6666qro6bR4Vq1axZo1a6ipqaG2tpZz586Rk5MT3f7QQw8xffp01q1bx/jx45k8eXK0MCWDZLxmJCLSothrREuWNN+pIRGuvvpq8vLyokuvXr0abh+JO+1M03Vduza+slBfX8+iRYuoqqqKLnv37uXQoUNkZmZGn7s5r7/+OnPnzqWoqIht27ZRVVXFrFmzGl1zKikpobq6mnvuuYedO3dy8803s3bt2ra+BO1OxUhEAideZ4VL9bLrCHl5eXTu3LlRh4Xz58/z9ttvM2TIkEsem5+fz8GDBxsVuIYlJSWF/Px86uvrKW8mWGVlJbfddhuzZ88mPz+fvLw8Dh++6N5++vfvz5w5c9i0aRPFxcWsWbMGgM6dO0fb6xcVIxEJlEv1mvOzIHXt2pWZM2fy2GOPsXnzZg4cOMDMmTP54osvmDVr1iWPXbhwIa+88goLFy7kww8/5ODBg2zYsIF58+YBXhEpKChg+vTpbNy4kU8//ZQdO3awfr1318uAAQPYs2cPW7Zs4dChQzz55JO89dZb0eevra3lwQcfpKKigiNHjvDuu+9SWVkZLZI5OTmYGZs2beLLL7/k9OnTCXqVmqdiJCKBsnv3pbtvNxSk3bs7tl0ApaWlFBQUMG3aNIYNG8bevXvZunVro5508UycOJFNmzZRXl7OyJEjGTlyJE8//TS9e0fvYuGll16isLCQOXPmMGjQIIqKivjuO2/0tPvvv5+CggIKCwsZMWIER44c4eGHH44e26lTJ7755humTp3KwIEDuffeexk1ahTLli0DoFevXixevJgFCxaQnZ3N7NmzE/DqXJq1dC7ySjRw4ED30UfhvR2poqKCcePG+d2MhFG+YDhw4ACDBw++aP2pU6da3YEhaMKQrbn3DcDM3nfO3Rp3Ywv0zUhERHynYiQiIr5TMRIREd+pGImIiO9UjETEN+pAFSyJfL9UjETEF6mpqdTW1vrdDPkRamtrSU1NTchzqxiJiC+ysrI4duwYZ86c0TekJOec48yZMxw7doysrKyE/A4NlCoivsjIyADg888/bzTB3NmzZ0lLS/OrWQkV5GypqalkZ2dH37f2pmIkIr7JyMi46D+3iooKhg8f7lOLEivM2S6XTtOJiIjvVIxERMR3SVeMzGy2mb1nZnVm9mIr9v9nMzthZt+Z2Voz6xKzrYeZ/aeZfW9mNWZWmNDGi4hImyRdMQI+B5YCLc76ZGYTgceA8UAfIBdYHLPLCuAHIBuYAqw0sxvbub0iInKZkq4YOefecM79Efi6FbtPBV5wzu13zn0DPAkUAZhZV2Ay8IRz7rRzrhL4b+AfE9NyERFpq6D3prsR+K+Yx38Gss3sWqA3cN4593GT7X8b74nMbAYwI/Kwzsw+TEB7k8V1wFd+NyKBlC/YwpwvzNkABrb1wKAXo27AdzGPG37uHmdbw/a4k4k451YDqwHM7L22zskRBMoXbMoXXGHOBl6+th7boafpzKzCzFwzS2XLz3CR00DsTQoNP5+Ks61h+6k2/B4REUmgDi1GzrlxzjlrZvl5G55yPzA05vFQ4Avn3NfAx0CKmfVvsn1/2xOIiEgiJF0HBjNLMbM0oBPQyczSzKy504kvAcVmNsTMrgH+BXgRwDn3PfAGsMTMuprZaOAXwPpWNGP15eZIcsoXbMoXXGHOBpeRz5JtgEIzKwEWNVm92DlXYma9gWpgiHPuaGT/h4D5QDqwEXjAOVcX2dYDr4v4nXi98x5zzr3SIUFERKTVkq4YiYjIlSfpTtOJiMiVR8VIRER8p2IUI2xj2V1qnD8zG29mB83sjJmVm1mOT81sEzPrYmYvRN6nU2b2gZn9Xcz2QOcDMLOXzey4mZ00s4/NbHrMtsDna2Bm/c3srJm9HLMu8Pkit7KcNbPTkeWjmG2BzwdgZr8yswOR/zMPm9mYyPofnU/FqLGwjWUXd5w/M7sOr6fhE0AP4D3g9Q5v3eVJAT7DG1Hjb/CylJlZn5DkA/gt0Mc5lwH8A7DUzG4JUb4GK4DdDQ9Clm+2c65bZBkI4clnZncCpcA0vMEExgJ/aXM+55wWrxNHV7xCNCBm3Xrgab/b1g7ZlgIvxjyeAexskr0WGOR3Wy8z51688QhDlw9vmJXjQEGY8gG/AsqAEuDlyLpQ5AMqgOlx1ocl306guL3y6ZvRBQOIP5ZdkL8ZNedGvGxA9J6swwQ4q5ll472H+wlRPjP7g5mdAQ7iFaPNhCSfmWUAS4CHm2wKRb6I35rZV2b2JzMbF1kX+Hxm1gm4Fcg0s0/M7K9m9u9mlk4b86kYXfCjxrILuFBlNbNU4D+Adc65g4Qon3NuFl67x+Cd+qgjPPmexBt1/7Mm68OSbz7etDa98G4GfdPM+hGOfNlAKvBLvL+bw4DheAMPtCmfitEFV9JYdqHJamZX4Z1O/QGYHVkdmnwAzrnzzpsC5WfATEKQz8yGAROAf4uzOfD5AJxz7zrnTjnn6pxz64A/AX9POPLVRv78vXPuuHPuK2AZl5FPxeiCK2ksu0Zj+kXmfupHwLKamQEv4H1Km+ycOxfZFIp8caRwIUfQ843DmxDzqJmdAB4BJpvZHsKRLx4HGCHI57z54/6Kl6mptuXz+yJYMi3Aa8CreBfcRuN9tbzR73ZdRp4UIA2vV9b6yM8pQGYk2+TIulLgHb/b24Z8q4B3gG5N1gc+H5CFd3G/G944jROB7/HGVwxDvquB62OW3wEbItnCkO8nkfes4d/clMj7NzAM+SIZl+D1gswCrgF24J16bVM+3wMl04LXDfGPkb80R4FCv9t0mXlK8D65xC4lkW0T8C6K1+L1+unjd3t/ZLacSJ6zeKcFGpYpIcmXCbwFfAucBPYB/xSzPdD54uQtIdKbLgz5Iu/fbrxTU9/ifWi6Myz5IhlSgT9E8p0AngfS2ppPY9OJiIjvdM1IRER8p2IkIiK+UzESERHfqRiJiIjvVIxERMR3KkYiIuI7FSMREfGdipFIQJhZhpmVmNlgv9si0t5UjESC41ZgEd6d7yKhomIkEhzD8aaQqPa7ISLtTcMBiQSAmR0ABjVZvdE590s/2iPS3lSMRALAzEbgjSq/H/jXyOrjzrka/1ol0n5S/G6AiLTKn/Em1/u9c+4dvxsj0t50zUgkGG4EOgN7/G6ISCKoGIkEQz7e/E1VfjdEJBFUjESCYThw2Dl30u+GiCSCipFIMAxBXbolxNSBQSQYvgXyzWwi8B1wyDn3tc9tEmk36totEgBmdhPwAnAzkAaMcc5V+tsqkfajYiQiIr7TNSMREfGdipGIiPhOxUhERHynYiQiIr5TMRIREd+pGImIiO9UjERExHcqRiIi4rv/B+XYQy1rqNv/AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(43)\n",
    "\n",
    "series = generate_time_series(1, 50 + 10)\n",
    "X_new, Y_new = series[:, :50, :], series[:, 50:, :]\n",
    "Y_pred = model.predict(X_new)[:, -1][..., np.newaxis]\n",
    "\n",
    "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_XQgdqWeFpah"
   },
   "source": [
    "## Unstable gradients\n",
    "\n",
    "The tricks used for Deep NN can also be used here, except that:\n",
    "- Nonsaturating activation functions may not work [1], as the same weights are used for all time steps which can lead to explosion of outputs or gradient. You can also check the lr.\n",
    "- the benefits of the BN layer can be limited (as it does not change the statistics in each time step)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "M0EnqeNlFpah",
    "outputId": "08be7179-e9db-4835-de95-cc7c9c6f6698"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 5s 693us/sample - loss: 0.1936 - last_time_step_mse: 0.1913 - val_loss: 0.0901 - val_last_time_step_mse: 0.0863\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 3s 493us/sample - loss: 0.0531 - last_time_step_mse: 0.0441 - val_loss: 0.0559 - val_last_time_step_mse: 0.0469\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 3s 497us/sample - loss: 0.0470 - last_time_step_mse: 0.0374 - val_loss: 0.0453 - val_last_time_step_mse: 0.0354\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 4s 513us/sample - loss: 0.0437 - last_time_step_mse: 0.0337 - val_loss: 0.0423 - val_last_time_step_mse: 0.0320\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 4s 505us/sample - loss: 0.0414 - last_time_step_mse: 0.0310 - val_loss: 0.0402 - val_last_time_step_mse: 0.0301\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 4s 504us/sample - loss: 0.0390 - last_time_step_mse: 0.0283 - val_loss: 0.0377 - val_last_time_step_mse: 0.0261\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 4s 501us/sample - loss: 0.0368 - last_time_step_mse: 0.0255 - val_loss: 0.0369 - val_last_time_step_mse: 0.0255\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 3s 498us/sample - loss: 0.0354 - last_time_step_mse: 0.0240 - val_loss: 0.0369 - val_last_time_step_mse: 0.0244\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 3s 484us/sample - loss: 0.0342 - last_time_step_mse: 0.0227 - val_loss: 0.0361 - val_last_time_step_mse: 0.0233\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 3s 491us/sample - loss: 0.0332 - last_time_step_mse: 0.0216 - val_loss: 0.0340 - val_last_time_step_mse: 0.0223\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 4s 501us/sample - loss: 0.0324 - last_time_step_mse: 0.0210 - val_loss: 0.0338 - val_last_time_step_mse: 0.0229\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 4s 506us/sample - loss: 0.0316 - last_time_step_mse: 0.0202 - val_loss: 0.0332 - val_last_time_step_mse: 0.0214\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 4s 511us/sample - loss: 0.0309 - last_time_step_mse: 0.0192 - val_loss: 0.0307 - val_last_time_step_mse: 0.0187\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 4s 509us/sample - loss: 0.0305 - last_time_step_mse: 0.0189 - val_loss: 0.0322 - val_last_time_step_mse: 0.0198\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 4s 508us/sample - loss: 0.0304 - last_time_step_mse: 0.0185 - val_loss: 0.0297 - val_last_time_step_mse: 0.0177\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 3s 498us/sample - loss: 0.0296 - last_time_step_mse: 0.0177 - val_loss: 0.0291 - val_last_time_step_mse: 0.0168\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 4s 508us/sample - loss: 0.0293 - last_time_step_mse: 0.0173 - val_loss: 0.0285 - val_last_time_step_mse: 0.0163\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 3s 495us/sample - loss: 0.0288 - last_time_step_mse: 0.0169 - val_loss: 0.0284 - val_last_time_step_mse: 0.0159\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 4s 502us/sample - loss: 0.0286 - last_time_step_mse: 0.0166 - val_loss: 0.0284 - val_last_time_step_mse: 0.0156\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 4s 509us/sample - loss: 0.0282 - last_time_step_mse: 0.0162 - val_loss: 0.0323 - val_last_time_step_mse: 0.0214\n"
     ]
    }
   ],
   "source": [
    "# Deep RNN with Batch Norm\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.SimpleRNN(20, return_sequences=True),\n",
    "    keras.layers.BatchNormalization(),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qKk-R7g2Fpai"
   },
   "source": [
    "- Instead of BN, Layer Normalization [2] may be more useful that normalizes it across the feature dimensions (rather than the batch). This can be implemented by a custom memory cell on the fly, and it does not need the weighted average.\n",
    "- We use  `LayerNormalization` in a custom class, which is very similar to the regular RNN, except for its `call()` method that also updates the current inputs and states (in a simple RNN these two are equal).\n",
    "- We may also use either the `recurrent_dropout` (for hidden states) and dropout (for inputs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0Si9-dVIFpai"
   },
   "outputs": [],
   "source": [
    "# Deep RNNs with Layer Norm\n",
    "\n",
    "from tensorflow.keras.layers import LayerNormalization\n",
    "\n",
    "class LNSimpleRNNCell(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=\"tanh\", **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.state_size = units\n",
    "        self.output_size = units\n",
    "        self.simple_rnn_cell = keras.layers.SimpleRNNCell(units,\n",
    "                                                          activation=None)\n",
    "        self.layer_norm = LayerNormalization()\n",
    "        self.activation = keras.activations.get(activation)\n",
    "    def get_initial_state(self, inputs=None, batch_size=None, dtype=None):\n",
    "        if inputs is not None:\n",
    "            batch_size = tf.shape(inputs)[0]\n",
    "            dtype = inputs.dtype\n",
    "        return [tf.zeros([batch_size, self.state_size], dtype=dtype)]\n",
    "    def call(self, inputs, states):\n",
    "        outputs, new_states = self.simple_rnn_cell(inputs, states)\n",
    "        norm_outputs = self.activation(self.layer_norm(outputs))\n",
    "        return norm_outputs, [norm_outputs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wN-e_Cw8Fpai",
    "outputId": "08d43e0d-b3c0-4753-e4b1-d2d0a48b0b6c",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 8s 1ms/sample - loss: 0.1657 - last_time_step_mse: 0.1622 - val_loss: 0.0743 - val_last_time_step_mse: 0.0699\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 7s 983us/sample - loss: 0.0642 - last_time_step_mse: 0.0580 - val_loss: 0.0572 - val_last_time_step_mse: 0.0510\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 8s 1ms/sample - loss: 0.0546 - last_time_step_mse: 0.0478 - val_loss: 0.0512 - val_last_time_step_mse: 0.0443\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 8s 1ms/sample - loss: 0.0500 - last_time_step_mse: 0.0430 - val_loss: 0.0484 - val_last_time_step_mse: 0.0415\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0465 - last_time_step_mse: 0.0390 - val_loss: 0.0434 - val_last_time_step_mse: 0.0355\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 8s 1ms/sample - loss: 0.0410 - last_time_step_mse: 0.0313 - val_loss: 0.0381 - val_last_time_step_mse: 0.0273\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0373 - last_time_step_mse: 0.0260 - val_loss: 0.0354 - val_last_time_step_mse: 0.0243\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0349 - last_time_step_mse: 0.0232 - val_loss: 0.0341 - val_last_time_step_mse: 0.0229\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0337 - last_time_step_mse: 0.0219 - val_loss: 0.0336 - val_last_time_step_mse: 0.0218\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 7s 990us/sample - loss: 0.0326 - last_time_step_mse: 0.0208 - val_loss: 0.0315 - val_last_time_step_mse: 0.0199\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0319 - last_time_step_mse: 0.0202 - val_loss: 0.0308 - val_last_time_step_mse: 0.0196\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 7s 984us/sample - loss: 0.0309 - last_time_step_mse: 0.0190 - val_loss: 0.0318 - val_last_time_step_mse: 0.0199\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0303 - last_time_step_mse: 0.0182 - val_loss: 0.0291 - val_last_time_step_mse: 0.0169\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 7s 980us/sample - loss: 0.0299 - last_time_step_mse: 0.0177 - val_loss: 0.0291 - val_last_time_step_mse: 0.0165\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 7s 979us/sample - loss: 0.0291 - last_time_step_mse: 0.0167 - val_loss: 0.0306 - val_last_time_step_mse: 0.0187\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 7s 975us/sample - loss: 0.0289 - last_time_step_mse: 0.0167 - val_loss: 0.0291 - val_last_time_step_mse: 0.0175\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 7s 980us/sample - loss: 0.0284 - last_time_step_mse: 0.0162 - val_loss: 0.0287 - val_last_time_step_mse: 0.0165\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 7s 992us/sample - loss: 0.0285 - last_time_step_mse: 0.0165 - val_loss: 0.0283 - val_last_time_step_mse: 0.0161\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 7s 992us/sample - loss: 0.0279 - last_time_step_mse: 0.0156 - val_loss: 0.0272 - val_last_time_step_mse: 0.0149\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 7s 1ms/sample - loss: 0.0274 - last_time_step_mse: 0.0151 - val_loss: 0.0271 - val_last_time_step_mse: 0.0146\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True,\n",
    "                     input_shape=[None, 1]),\n",
    "    keras.layers.RNN(LNSimpleRNNCell(20), return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6qu95NNNFpai"
   },
   "source": [
    "## Short-term memory problem\n",
    "\n",
    "### LSTM cells\n",
    "- The **Long Short-Term Memory (LSTM) cell** may be better and even faster to converge.\n",
    "- To implement it,\n",
    "    - use `LSTMCell()` in an `RNN()`, or\n",
    "    - simply use `LSTM()`, which is more optimized for GPU implementation.\n",
    "- As shown in figure below, here, the output state is split into short-term state $\\mathbf{h}_{\\left(t\\right)}$ and long-term state $\\mathbf{c}_{\\left(t\\right)}$ ($\\mathbf{c}$ means cell).\n",
    "\n",
    "<img src=\"img/lstm.jpg\" width=\"400px\">\n",
    "<center>An LSTM cell.</center>\n",
    "\n",
    "- FC blocks denotes fully connected layers: the major block is $\\mathbf{g}_{\\left(t\\right)}$ and the others are gate controllers.\n",
    "- Below are the LSTM computations:\n",
    "\n",
    "\\begin{equation}\n",
    "\\mathbf{i}_{\\left(t\\right)}=\\sigma\\left({\\mathbf{W}_{xi}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hi}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_i\\right)\\\\\n",
    "\\mathbf{f}_{\\left(t\\right)}=\\sigma\\left({\\mathbf{W}_{xf}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hf}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_f\\right)\\\\\n",
    "\\mathbf{o}_{\\left(t\\right)}=\\sigma\\left({\\mathbf{W}_{xo}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{ho}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_o\\right)\\\\\n",
    "\\mathbf{g}_{\\left(t\\right)}=\\text{tanh}\\left({\\mathbf{W}_{xg}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hg}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_g\\right)\\\\\n",
    "\\mathbf{c}_{\\left(t\\right)}=\\mathbf{f}_{\\left(t\\right)}\\otimes\\mathbf{c}_{\\left(t-1\\right)}+\\mathbf{i}_{\\left(t\\right)}\\otimes\\mathbf{g}_{\\left(t\\right)}\\\\\n",
    "\\mathbf{y}_{\\left(t\\right)}=\\mathbf{h}_{\\left(t\\right)}=\\mathbf{o}_{\\left(t\\right)}\\text{tanh}\\left(\\mathbf{c}_{\\left(t\\right)}\\right)\n",
    "\\end{equation}\n",
    "- where, $\\otimes$ is the element-wise multiplication operator, and $\\sigma$ is the sigmoid function.\n",
    "- The tanh function keeps outputs between -1 to +1, while the sigmoid function keeps values between 0 to 1 for gates. Thus, in TF, $\\mathbf{b}_f$ is initialized with a vector of 1 to forget everything at first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sczYCS9hFpai",
    "outputId": "8c3ed6da-af41-4fad-a095-80f41e258049"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 8s 1ms/sample - loss: 0.0760 - last_time_step_mse: 0.0615 - val_loss: 0.0554 - val_last_time_step_mse: 0.0372\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 5s 758us/sample - loss: 0.0480 - last_time_step_mse: 0.0281 - val_loss: 0.0423 - val_last_time_step_mse: 0.0211\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 5s 765us/sample - loss: 0.0390 - last_time_step_mse: 0.0182 - val_loss: 0.0371 - val_last_time_step_mse: 0.0164\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 5s 764us/sample - loss: 0.0350 - last_time_step_mse: 0.0151 - val_loss: 0.0335 - val_last_time_step_mse: 0.0143\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 5s 752us/sample - loss: 0.0325 - last_time_step_mse: 0.0136 - val_loss: 0.0314 - val_last_time_step_mse: 0.0124\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 5s 764us/sample - loss: 0.0308 - last_time_step_mse: 0.0123 - val_loss: 0.0297 - val_last_time_step_mse: 0.0108\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 6s 834us/sample - loss: 0.0297 - last_time_step_mse: 0.0117 - val_loss: 0.0289 - val_last_time_step_mse: 0.0106\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 6s 863us/sample - loss: 0.0286 - last_time_step_mse: 0.0108 - val_loss: 0.0279 - val_last_time_step_mse: 0.0103\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 6s 859us/sample - loss: 0.0280 - last_time_step_mse: 0.0106 - val_loss: 0.0279 - val_last_time_step_mse: 0.0111\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 6s 819us/sample - loss: 0.0274 - last_time_step_mse: 0.0105 - val_loss: 0.0270 - val_last_time_step_mse: 0.0103\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 6s 865us/sample - loss: 0.0268 - last_time_step_mse: 0.0100 - val_loss: 0.0264 - val_last_time_step_mse: 0.0097\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 6s 832us/sample - loss: 0.0264 - last_time_step_mse: 0.0099 - val_loss: 0.0263 - val_last_time_step_mse: 0.0108\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 6s 877us/sample - loss: 0.0259 - last_time_step_mse: 0.0096 - val_loss: 0.0257 - val_last_time_step_mse: 0.0094\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 6s 881us/sample - loss: 0.0257 - last_time_step_mse: 0.0096 - val_loss: 0.0264 - val_last_time_step_mse: 0.0110\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 6s 868us/sample - loss: 0.0253 - last_time_step_mse: 0.0095 - val_loss: 0.0253 - val_last_time_step_mse: 0.0096\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 6s 859us/sample - loss: 0.0250 - last_time_step_mse: 0.0092 - val_loss: 0.0248 - val_last_time_step_mse: 0.0091\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 6s 893us/sample - loss: 0.0247 - last_time_step_mse: 0.0092 - val_loss: 0.0245 - val_last_time_step_mse: 0.0087\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 6s 867us/sample - loss: 0.0245 - last_time_step_mse: 0.0092 - val_loss: 0.0242 - val_last_time_step_mse: 0.0092\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 6s 789us/sample - loss: 0.0242 - last_time_step_mse: 0.0089 - val_loss: 0.0237 - val_last_time_step_mse: 0.0084\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 6s 802us/sample - loss: 0.0239 - last_time_step_mse: 0.0089 - val_loss: 0.0240 - val_last_time_step_mse: 0.0086\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.LSTM(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.LSTM(20, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7ItbQGy5Fpai",
    "outputId": "ced299fb-3892-461d-c8e0-62d556b88d5e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000/2000 [==============================] - 0s 183us/sample - loss: 0.0240 - last_time_step_mse: 0.0086\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.02401665338873863, 0.008551412]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.evaluate(X_valid, Y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X5ohlfakFpaj",
    "outputId": "83a05bee-8277-4769-cfc3-c21a37422624"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAESCAYAAADAEMPrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdeZxOdf/48dd79jFjbJORbTCWBjG2GEumG1EqRHeLioSWu9vNXe5U1qTSD2lRKkmUpU3afZGp1EgqLYjsRcg+g1nMvH9/nGumMV2Y7Vrwfj4e5+E653yuc95n5nK95/M5n/P5iKpijDHGlLYAXwdgjDHm3GQJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRXk0wIlJRRBaKyFER2S4iN52m7DAR2S0ih0VkpoiE5tuXLCLpIpLmWjZ45wqMMcYUlrdrMNOATCAG6As8LyKNChYSka7ACKATUAuoA4wrUOweVY10LQ08GrUxxpgi81qCEZEIoDcwSlXTVHUF8B5wi5vi/YCXVXWtqh4ExgP9vRWrMcaYkgvy4rnqA9mqujHfth+Ajm7KNgIWFSgXIyKVVHW/a9tjIvI4sAF4SFWT3Z1URAYDgwHCw8Nb1KhR429lAn7bS9jxIxyOrUdw6OmHzsnJySEgwH9uXflTPBaLe/4UC/hXPBaLe/4UC8DGjRv3qeoFRX6jqnplAToAuwtsGwQkuym7GeiWbz0YUKCWa701UBYIxantpAJxZ4qhRYsW6s66xxepgq4c/39u9+e3fPnyM5bxJn+Kx2Jxz59iUfWveCwW9/wpFlVVYLUW43vfmykyDYgqsC3KlRzOVDb3dSqAqn6tqqmqmqGqrwJfAlcWN7DoGzpzjHBCFy86c2FjjDGF4s0EsxEIEpF6+bY1Bda6KbvWtS9/uT36V/NYQQpIcQOLrlmGT4O6Evv9IrDRpY0xplR4LcGo6lHgHeBhEYkQkXZAD2COm+KzgdtFpKGIVABGArMARKS8iHQVkTARCRKRvsClwOLixiYC31bvQYWjv8N33xX3MMYYY/Lx9l2ku4FwYC8wD7hLVdeKSE3X8yw1AVT1E+AJYDmw3bWMcR0jGHgE+BPYB/wb6KmqJXoWZlfzq8gmABZZM5kxxpQGryYYVT2gqj1VNUJVa6rqXNf2Heo8z7IjX9kpqhqjqlGqepuqZri2/6mqrVS1rKqWV9U2qrqkpLFVaRzNCtqT864lGGOMKQ3e7Kbs1+rVg0X0oONP9/Ld21tp3ru2r0My5iRHjhxh7969ZGVlleg45cqVY/369aUUVclYLO55M5bg4GAqV65MVFTBPlglZwnGJT3dSTBTuJd5Ny4i47OhJCb6OipjHEeOHGHPnj1Uq1aN8PBwRIrdp4XU1FTKli1bitEVn8XinrdiUVWOHz/Ozp07AUo9yfjPkzw+9vvvsIU4fqIx3bMWkZzs64iM+cvevXupVq0aZcqUKVFyMSY/EaFMmTJUq1aNvXv3lvrxLcG4dO0KQUFOLaYDn9O52al6RBvjfVlZWYSHh/s6DHOOCg8PL3HTqzuWYFwSE2HePHiPHgSSQ6u9H/o6JGNOYjUX4yme+mxZgsmnTx+o1bsFO6UaJ9623mTGGFMSlmAKuHd4AIv0GvSTT+D4cV+HY4wxZy1LMAW0bg2bGvYgOPMY2f+3zNfhGGNO4YYbbqBPnz5Fek+bNm247777PBTRX0aMGEHLli09fh5/Z92U3bh0zGUcvj6K/U8vok6Pq3wdjjFnpTO16/fr149nnnmm2Md/4YUXckdbL7SPPvqI4ODgYp/TFI0lGDeu7h3Cx5FX0Pbz99AT05GgQF+HZMxZ548//sh7/cEHHzBo0KCTtp2qV1xWVlahkkC5cuWKHFPFihWL/B5TfNZE5kZgIIRc14OKJ/by40tf+zocY0pNSgpMnhxCSornz1WlSpW8pXz58n/bVq5cOTZu3IiI8Oabb9KxY0fCwsJ49dVX2bNnD9dff33esz+NGzfm9ddfP+n4BZvI2rRpw7Bhwxg+fDgVK1akSpUqPPjggyfVcgo2kVWpUoWJEycyYMAAqlatSo0aNXj66adPOs+6deto164dYWFhNGzYkCVLlhAUFMT8+fML/bPIzs5m9OjRVK9endDQUBISEvjoo4/y9qsqo0aNombNmoSGhlK/fn0GDhyYt3/ZsmVccsklREREUL58edq0acOGDSUaftErrAZzCu0nXEnmK8FsfnIRTe9q6+twjDnJ0KGwZk3R3nP4MPz4I+TkhDB+PDRpAkWpBCQkwNSpRTtnYY0YMYLJkyfTtGlTQkNDOX78OG3atOGBBx4gKiqKjz/+mH79+hEbG0v79u1PeZyZM2cyfPhwvv76a1atWsWtt95Kq1at6NWr1ynfM2nSJMaPH8+QIUNYunQp//nPf2jfvj3NmzfnxIkT9OjRg7i4OFatWsWRI0cYNmwYOTk5Rbq+J554gqeffpoXXniBhIQEZs6cSY8ePfjxxx+Jj49n7ty5TJs2jXnz5hEfH8+WLVvyhorJyMigZ8+e/Pvf/2b+/PlkZGSwevVqv5rx8lQswZxCmQvLsal2Eo1+fZdff51IvXpnfo8x/uzwYXC+F4WcHGe9GK1MHvHf//6Xnj17nrRt2LBhea//9a9/sWTJEubPn3/aBNO8eXNGjhwJQL169Zg+fTrLli07bYK56qqruPPOO0lNTSUhIYGnnnqKTz/9lObNm/Phhx+yfft2vvzySypXrgzAxIkT6dSpU5Gub9KkSTz44INcf/31ecf47LPPmDx5MjNmzGD79u1Uq1aNzp07ExgYSIUKFUhKSgJg//79pKWl0aNHD+rUqQNAfHx8kc7vK5ZgTqPyoB5EPXgP40b/wph5F/k6HGPyFKcmkZICnTpBZqYSEiK8/jp+M95ewR5XJ06cYMKECbz11lvs3LmTzMxMMjIyuOKKK057nCZNmpy0XrVq1TMOgXK69/zyyy/UqlUrL7kAtG7d+ozXk9/evXs5cOAA7dq1O2l7+/bt+eqrrwCnuW/atGnUrl2brl270rFjR66//nqCg4OpWrUqN9xwA0lJSXTu3JlOnTpx3XXXUa1atSLF4Qv+X8fyoaibrwEg661F7Nvn42CMKaHERFi2DEaOzGTZMv9JLgAREREnrU+YMIFp06bxwAMPsHz5ctasWcOVV15JZmbmaY9TsHOAiJyxOet071HVEj/lnnsPyN1xcrfVqVOHTZs28eyzz1KmTBn+97//0bp1a9LT0wGYN28eX375JW3btuXtt9+mXr16LF++vERxeYMlmNOpUYPjjVpw5YlFPP+8r4MxpuQSE+HeezP9Krm4s2LFCnr16sVNN91E06ZNqVOnDhs3bvR6HPHx8Wzbto0///wzb9uqVauKdIyYmBgqVarEihUrTtq+YsUKGjZsmLceHh7ONddcw1NPPcWSJUv4/vvvTzpX8+bNeeCBB/jiiy9o3bo1c+a4mwzYv1gT2RmEX9+DNqPHMPip3QwfXsXX4RhzXqhfvz4ffvghKSkplC9fnilTprBr1y5iY2O9Gkf37t2pWbMm/fr14/HHHyc1NZURI0YgIkWq2dx33308+uij1K5dm6ZNmzJz5ky+/fZbXnnlFQBeeuklgoKCaNWqFREREcydO5eQkBDi4uLYsGEDs2fP5qqrrqJq1ar8+uuvrFu3ju7du3vqskuN1WDOpGdPAlAS97/Pa6/5Ohhjzg/jxo2jSZMmdOnShaSkJCpXrlzkp/ZLQ1BQEIsWLeLQoUO0atWKgQMHMnr0aADCwsIKfZzhw4czZMgQhg4dSuPGjfn444959913827Wly9fnunTp9OuXTsuvvhiFi9ezKJFi6hWrRqRkZGsXbuW3r1753VfHjhw4EmdIPyWqp43S4sWLbTIcnI0p3Zt/Tyqu8bHqy5btrzox/Cg5cuX+zqEPBaLe6URy7p160oeiMuRI0dK7VgldTbGsnLlSgX0559/9nkspel0nzFgtRbjO9dqMGcigvToQdvjS9mxPo1Vq+xJYGPOJ2+++SZLly5l27ZtLFu2jIEDB3LJJZfQqFEjX4fm9yzBFEbPngRmZXBTxcW88UYNX0djjPGiw4cPc+edd3LRRRdx66230qxZMz744ANfh3VWsJv8hdGuHVSsyH9qLaLxd7357jto3tzXQRljvCH3nocpOqvBFEZQEFx1FfFbPiAq/DiTJ/s6IGOM8X+WYAqrZ08CDh3kv5e8x4IF8Ntvvg7IGGP8myWYwrr8cggL48Yy7wDw1FM+jscYY/ycJZjCioiAFi2o/cVHPHTZV7z0Ehw54uugjDHGf1mCKayUFFi1iuC0NMZ8dhkNj6QwY4avgzLGGP9lCaawkpNzxzonICuTIdUXMnUqZGX5NixjjPFXlmAKKykJQkJQ1yQ/V+l77P0tnbfe8m1YxpwPZsyYkTcrprt1dx5//HHq1q1b6uf2lJEjR5KQkODx83iTJZjCco11vnXAAHj8ccru3MCsckOZNAnyzchqjHG5+uqr6dy5s9t969evR0T49NNPi3Xsvn37lvroyidOnEBEePfddz1+rvOFPWhZFImJ7MjIoE5SEhw4wA1PPMFH37Xls89uxTX5nDHGZeDAgfTq1Ytt27ZRq1atk/a9/PLLxMbG5s3aWFTh4eGEh4eXPEg/O9e5xmowxTVhAjkdOjKdO3l7zI++jsaYwklJIWTyZKfTiod1796dmJiYvCHpc2VlZTFnzhwGDBiQN6/8fffdR/369QkPD6d27dqMGDGCjIyMUx7bXbPVY489RkxMDGXLlqV///4cO3bspP1ff/01Xbp0ITo6mnLlytGhQ4eT5ltp3LgxAL169UJE8prX3J3rueeeIy4ujpCQEOrVq8fMmTPz9uXWhGbMmEHv3r2JiIggLi6OefPmFfZHB0BOTg7jxo2jevXqhIaG0qRJE95///28/arK2LFjiY2NJTQ0lAsvvJDbbrstb39ycjKtW7cmIiKC8uXL07p1a9avX1+kGErKajDFFRREwBvzya7fnCGf92bDqtU0uMRPJjg3576hQ2HNmqK95/Bh+PFHQnJyYPx4aNIEyhXhM5uQUKS5moOCgujXrx+zZs1izJgxecnk/fffZ9++fSd9GUZFRTFr1iyqVq3K2rVrueOOOwgPD2fMmDGFOtfcuXMZO3Yszz77LB07dmT+/PlMmjTppKmOU1NT6devH08//TQAzzzzDFdccQWbNm2iQoUKJCcnU79+fV555RW6detGUJD7r8c333yToUOHMnXqVDp37sxHH33E4MGDufDCC0+a0nncuHFMnDiRiRMn8sILL9C/f386dOhA9erVC3VNkydPZsqUKUyfPp3mzZsze/ZsevXqxZo1a2jcuDFvvPEGU6dOZd68eTRq1Ig9e/bwzTffAE4S79GjB3feeSdz584lKyuL1atX5/0OvMVqMCVRpQrZry+gNls5ev1tdjPG+LfDhyEnBwGnR+Thwx4/5e23386OHTtYunRp3raXX36Zyy+/nBo1/ho4dvTo0bRt25ZatWrRvXt3RowYUaS/+KdOncqAAQMYNGgQ9evXZ/To0TQvMGBg586dufnmm4mPjyc+Pp5p06YREBDA4sWLAYiOjgacuVmqVKmSt17QpEmT6N+/P3fffTf169dn6NCh3HDDDUycOPGkcv379+emm26ibt26TJgwAeBvs1qezqRJk7j//vu58cYbadCgARMmTKBNmzZMmjQJgO3bt1O1alW6dOlCzZo1adWqFXfffTcABw8e5MiRI1xzzTXExcVx0UUXcfPNN9OgQYNCn780eLUGIyIVgZeBy4F9wAOqOvcUZYcB9wPhwNvAXaqaUaBMPeAn4C1VvdmTsZ9K+as78GbiE1yXci+pYydTdtx9vgjDnG+KUJPIk5ICnTqhmZlISAi8/jqenju5Xr16XHrppcycOZPLL7+cXbt2sXjxYhYsWHBSuQULFvD000+zefNm0tLSOHHiRJH+2l6/fj333HPPSdsSExN5880389b37NnDqFGjSE5OZs+ePWRnZ3Ps2DF27NhRpGtav3593hd5rvbt2+dNRJarSZMmea9DQkKIjo5m7969hTrHgQMH2Lt3L+3atfvbeXI7Rlx//fU888wz1K5dm65du9KtWzeuueYaQkJCqFy5MjfffDOdO3emU6dOdOrUieuuu67QtafS4u0azDQgE4gB+gLPi8jfJlUQka7ACKATUAuoA4w7xfG+8VSwhdVk5jDeojdlxo+Azz/3dTjGuOfqCZk5ciQsW+bx5JJr4MCBvPvuuxw4cIBZs2ZRsWJFrrnmmrz9K1asoG/fvlx55ZW8//77fP/99zz88MNkZmaWahw333wz33//PVOnTuWrr75izZo1VK1atVjncTddcsFtwcHBf9uf43qW7kzU1RpyuvPExsayceNGnnvuOSIjIxk2bBitWrXKu/c0Z84cUlJSaN++PQsXLqR+/fon1SS9wWsJRkQigN7AKFVNU9UVwHvALW6K9wNeVtW1qnoQGA/0L3C8G4BDwDKPBl4IDS4S3rpiJlslDv3n9fDHH74OyRj3EhPJvPderyUXgD59+hAWFsZrr73GzJkzufXWW0/68v3yyy+JjY3loYceolWrVtSrV49t27YV6Rzx8fGsXLnypG0F11esWMGQIUO48soradSoEREREezevTtvf2BgIIGBgWRnZ5/xXAWbulasWEHDhg2LFPPpVKpUicqVK5/xPOHh4Vx99dVMnTqVlStX8uOPP5503QkJCYwYMYLPP/+cdu3aMXv27FKLsTC82URWH8hW1fwdyn8AOrop2whYVKBcjIhUUtX9IhIFPIxTw7n9dCcVkcHAYICYmBiSk5OLfwVAWlqa22O07VaOnh+/zbf7L+F4t278MGUKGhhYonOVJB5fsFjcK41YypUrR2pqaqnEk52dXWrHKqw+ffowZswYDh06xA033JB3/uzsbGrUqMGOHTuYOXMmLVq0YMmSJXlNaLnl0tPTT7s+ePBg7rnnHho1akRiYiILFy7k22+/JTo6Oq9M3bp1eeWVV4iPjyc1NZVRo0YRGhpKRkYGqamp5OTkUKNGDT7++GOaNm1KSEgIFSpU+Nu57rnnHm6//Xbi4+NJSkpi8eLFzJ8/nwULFpCamsqJEycAOH78+Ek/Z1UlPT39lD/7jIwMcnJySE1NJTs7myFDhjBx4kSqVq1KkyZNmDdvHitXrmTy5MmkpqYyZ84cAFq0aEGZMmV48803CQ4OpkqVKqxZs4Y5c+ZwxRVXULVqVTZv3sxPP/1E+/btT3n+9PT00v8/U5x5louzAB2A3QW2DQKS3ZTdDHTLtx4MKFDLtf4UcL/r9VjgtcLE0KJFi8JMTX1ap5pfPSdHtVUr1f/GvKYKqsOHl/hcJYnHFywW90ojltPNl15Uvpjv/dtvv1VA27Zt6zaW4cOHa3R0tEZGRmrv3r312Wef1cDAwLxyL730kpYrV+6U66qq48eP1wsuuEAjIiK0b9++OnLkSI2Li8vb/91332mrVq00NDRU4+Li9PXXX9cGDRro+PHj82JZuHCh1q1bV4OCgvLe6+5c06ZN0zp16mhwcLDWrVtXZ8yYkbcvKytLAV24cOFJ76lWrZo++eSTp/wZPfTQQ9q0adO8WLKzs3Xs2LFarVo1DQ4O1osvvljfe++9vPJvv/22tm7dWsuVK6cRERHaqlUr/fDDD1VVddeuXdqzZ0+98MILNSQkRGvWrKkjRozQrKysU57/dJ8xYLUW53u/OG8q1omgGXCswLZ7gffdlP0B+Ge+9UquBFMJSADWAiHqRwlGVXXBAucnuqT+Xc6Lt98u8flKEo+3WSzuWYI5NYvFPV/E4okE482b/BuBIFfPr1xNXcmioLWuffnL7VHV/UASzo3/HSKyG7gP6C0i33ki6KK48EIQge4bn2S1tOLErbfBr7/6OixjjPEJryUYVT0KvAM8LCIRItIO6AHMcVN8NnC7iDQUkQrASGCWa9+LQBxOTSYBmA58CHT17BWcWe79uExC6aNvkpkTBL17Q4Enio0x5nzg7W7Kd+M817IXmIfzbMtaEakpImkiUhNAVT8BngCWA9tdyxjXvmOqujt3AdKAdFX908vX8jdJSRAW5tRithPLY41eR3/+Ge66yx7CNMacd7z6oKWqHgB6utm+A4gssG0KMKUQxxxbWvGVlOsxA5KTYcsWeGRGNxIuHkPv2WOhXTsYPNjXIRpjjNfYWGSlLDHxr0cM6tSB6x4cxXdVUmj6738jzZtDy5a+DdCctVTV7YN3xpSUeqiFxcYi86AHHoCJTwTQefdr7AuqgvbpA/v3+zoscxYKDg7m+PHjvg7DnKOOHz/+t5EHSoMlGA8bPhwemhLNlcfe4sRvf5DT95a8qZeNKazKlSuzc+dOjh075rG/Ns35R1U5duwYO3fuPGnk6dJiTWReMGwYBAW1YsiQqTy/+G5ODLyDoHp1nF4BXhyyw5y9oqKiANi1axdZWVklOlZ6ejphYWGlEVaJWSzueTOW4OBgYmJi8j5jpckSjJf8+9/wXMCdfHzPIrq9MgMNCEBCQ7066KA5u0VFRZXKl0BycjLNmjUrhYhKzmJxz59iKQlrIvOiu/8lXHCVk0wkJwfNyHC6nBljzDnIEoyXtXzwcrKDw1CAnBwyDx31dUjGGOMRlmC8LTGRoM8+5Yceo/mGloQ8MYHMUePtQUxjzDnHEowvJCaS8O44Nr3yJXO4hZBHRpM14A5wDfNtjDHnAkswPnRT/xCC577K4/IAwbNe4sRVPeGoNZkZY84NlmB87IYbhbgFj/IveQ5Z/DHZl14GhZy32xhj/JklGD9w3XXQ6a27uC7gHbLW/Ex2m7awaZOvwzLGmBKxBOMnrr0W+r3Tg87yKak7DpHTJhFWrfJ1WMYYU2yWYPxIjx4w4t02tA/4ip1HypLZLon1kz7wdVjGGFMslmD8zFVXwYDH69MyK4UfTjSi/vAebL7/RV+HZYwxRWYJxg9lZMC+gBguYzmL6UrcE3dw4F+j7FkZY8xZxRKMH0pKgtBQSA+MpE/QIl4NHEDF5x5hTbPbSE8t2UCHxhjjLTbYpR/KPzNmUlIwtWJn8FbXmvT5YSxfxPxB5ty36NSzrK/DNMaY07IajJ9KTHQmLEtMhAurCn1+GsPaYTNIPL6MCr06MvjqP9i509dRGmPMqVmCOYs0mnI7+u57NA7ewEMfJPJQ3DwOj/iAE1+k+Do0Y4z5G0swZ5ngHlcS8tVnVIs6wisZN3HV109yomMnfnrRkowxxr9YgjkbtWxJ0ODbAQgkh1A9znd3TGfQQGX/fh/HZowxLpZgzlbXXouEh6MiIEI/ZnP7y20ZWOdTZs6EnBxfB2iMOd9Zgjlbubqabb39duSzz+DFF2ke8zsLj3Qi9vZO3NVsJa+9Bo89BinWemaM8QFLMGezxER29O0LHTrAoEGEbPuVnCen0jbqZ174MZFyt1zNggd/4B//sCRjjPE+SzDnkrAwAob+h/Cdm1ly2aO0ZwVrSGBW+vU8ftsGVq60wQCMMd5jCeZcFBlJ5IQHaBi2lUflIbrzIe9saMj6xNu4psk2XnkFjh/3dZDGmHOdJZhzVGIivPNpeWTCI6z/YAvZ//oPtwTN45219Tk64B5aVP2D+++Hbdt8Hakx5lxlCeYcljsaQKvulQl5dgpBWzcRNGgAdwe+wPepcVzw//7HnbU/YXb8Y3w9NcV6nhljSpUlmPNJ9erIC9MJ2PALoTf14V79f3zMFdz8y0M0GfYPbohN4amn4NAhXwdqjDkXWII5H8XFwezZyNChCBCAEkY6jxy4m9eHrqJaNbjzTvjpJ18Haow5m1mCOZ/9858QHg6BgUhQEPXZyCpa81OZ1mTMfI2WTTJISoIJE+CRR6yrszGmaCzBnM9y5wUYPx4+/xz++AOeeYY6lQ7zStYtHIiM5ZrVo5k2chejRjmP24weDXv2+DpwY8zZwKsJRkQqishCETkqIttF5KbTlB0mIrtF5LCIzBSR0Hz7XhORP0TkiIhsFJGB3rmCc1D+eQGiouCee2DdOli8mIikVgw79gjbiWUeN3BJ9leMH69UqQLNm8ODDzp5KcvmQDPGuOHtGsw0IBOIAfoCz4tIo4KFRKQrMALoBNQC6gDj8hV5DKilqlHANcAjItLCs6GfRwIC4PLL4f33+X7BrzwXNIRufMJXtOPP2Ja813sWFcuk88QT0LEjREfDtdfC++9fyI4dvg7eGOMvvJZgRCQC6A2MUtU0VV0BvAfc4qZ4P+BlVV2rqgeB8UD/3J2u7Rm5q64lzpPxn6+aXxfHJZ9PZsbYnWz+33SiIzO4+u3bWLqhBmn/eZCPpu/g+uth9WqYMqUBsbHQqBHcey8sWQLp6b6+AmOMr4h6aewQEWkGfKWq4fm23Qd0VNWrC5T9AXhUVRe41qOBP4FoVd3v2vYcTtIJB74HLlXVNDfnHQwMBoiJiWkxf/78El1HWloakZGRJTpGafJ6PKqUX7OGagsXEv3llwDsa9eOA82ac3DrUb4Iupw3fuvEDz+UJysrgNDQbBISDnHJJQeoUCGTXbvCSUg4RKNGRzwapj/9nvwpFvCveCwW9/wpFoDLLrvsW1VtWeQ3qqpXFqADsLvAtkFAspuym4Fu+daDcWoptQqUCwTaAyOB4DPF0KJFCy2p5cuXl/gYpcmn8Wzbpnr//aply6qC5oBqQIBq3756fOZcTX5+nQ751wmtV0/VGQXNWQIDVR97TPXQIc+F5k+/J3+KRdW/4rFY3POnWFRVgdVajO99b96DSQOiCmyLAlILUTb39UllVTVbnaa26sBdpRSnKazYWHj8cbjvPhBBwJmIZv58wgbcRMe7GvLUy5FsLH8JKRcP5l9Moy1fEp6dygMPQKVKTs+0Rx6Bb76B7GxfX5AxpjQFefFcG4EgEamnqr+6tjUF1ropu9a174185faoq3nMjSDsHozvdOkCjz9OTkYGAaGhsHgxlC8Pa9bkLS02vE0bXsp7y+HoOH6rlMAXWxL4aFRTXhiVwPGK1bmr2Up6lE8m9tYkLrgm0YcXZYwpKa8lGFU9Kv2WkNoAAB4YSURBVCLvAA+7uhUnAD2Atm6KzwZmicjrwB84TWCzAESkMvAP4APgONAZuBE4ZZdn42Gu52m2zZxJnQEDnHWAiy+GW5w+HMGqfPveTra8s4bWYT9Q88Aayq1ZQ+Ndb+dVPY8dKUvosqMISubbIfSP/YRKvZPo2tWp6YSHuz+9McY/ebMGA3A3MBPYC+wH7lLVtSJSE1gHNFTVHar6iYg8ASzHuYn/NjDGdQzFaQ6bjtMLbjswVFUXefdSzEkSE9mRkUGdxFPUOkRo0aM6LXpUB676a3tqqjMmzZo1lJk1C/3mGwQII4MZ2zvz1ZPtWD4licnBSYR2bEOteCfLXHstJCV5+qKMMSXh1QSjqgeAnm627wAiC2ybAkxxU/ZPoKOnYjReVrYstG3rLM2aIZ06QWYmBAYS1KcP7X7ZSIc1jyBZD5O+NJSVS9uQTBJjn0liU6U21G0cRlwc1K3rDLGW+9oY43versEYc2q5Q9ckJzvVk8REAgEOH4YVK1gzMZmIL5IZxXjGMo6MA6Gs/b4Nn65O4oOjSaykDRmE0YYUuoWu4LW4UI4nJJ6UgOrWhcqVYeXKk07jXkpKIQoZY07FEozxL4mJf/8yL1cOundHK3anYycIyzhMUtAKnu2TTPONyTT/bjz3MY6ckFAOV4snavvPSEY22evHM3XXaL49XI9fNZsAcgggh9DgHHKychBy2B6Qw96uOTRpnENMdDZlwnKcnnBbtsD06U7XtpAQ+PRTSzLGFJElGHPW+KuCU46kpO5UTezu7HDVcAKSk6kwbx7knAAgQDMZfmjk3w+Uf+y0HOBj13Iq6enk9OhJwLW9nN4GHTpAzZqldFXGnLtKlGBEJBxoB/yqqttLJyRjTs1dBSe3hkP37s7d/06d0IwMJDgYZsxwRuYMDHTGWAsI4Ls1AdzYN4CMrACCQgKY/GQAYWUC2LQ1kF83B/Dr5gDKrFvNq4euIYQMcgjkuz9r0/jFuUS+8AIAqRVjOdq8A3vqd2BlcAea/PMiEtuK938gxvixIiUYEZkFrFLV50QkBFgFNAIyRaSXqp7u70BjPM9VzdlasMt0Ps3jYFbVv99e6XpSqc6kLVnG7neT+Tk6ia80kanrs9EffqT6ti9oc+ALLl36fzRd+hpNgT+fiubLC9pztHkH5NIOXHB5M+o3DKJMGY9fsTF+q6g1mK7A067X1wBlgSrAAGAsp29oMMY7ztRlmlPUhAqI7JJIZJdEqgPdAGdkombk5DRj584hPDhGWfHKr7TnCzrwBR33f0Htxe/CYkh9KJIVJLIvIpaI8OO82OoQWd2upn58IA0aQPXqToUKrC+BOXcVNcFUwHmGBZz/c2+r6l4RmQ88VKqRGeOnAgKgRg0YOEiYN78+r2bWZ17I7SxbBlUu2Mned1aQtexzWn7zCRUOLkGOAh+/TtbHgWyjNr9QmyWBtUm7oDZHKtXm419qsymnDuOCKzH9BeHyy+GCCyA42NdXakzJFDXB7AYai8gfOLWZwa7tkZx869SYc56bXtVANWL/dz3873p47DEYNQqys9GAAHIuaUe5sCokbN1K2z1vE7l7P+yGUa7jpWZGsvW22nxDbbZSmz3htTlUsTbpVWpTK2IvCRmr2H9xEpktEqlcGWJinC7XlSs7c8UVtut1zddfh9BQ31eX/CkW4xFFTTAzgQXALiAbWOba3hr4pRTjMuascNqmtqQkCAnJG6MtdMrjVM5fODWVH97dyoSBW6metZU6AVvpWn8r7Q9v4Yp9ywg5fhR24iw4Q1jwNeybEc0RosgglFTC+JMwMiSMdA3lIsLYLmFk1g6j4oWhlKkQRmR0GGUvCCP8yB5k5svUPnECXnvNmbCnfXuP/nxOsnevMzbd99/D0qWwbBm1VWHWLHj2WRgwwKpt55giJRhVfVhE1gI1gTdVNdO16wQwsbSDM+asdqox2nKVLUvTW5owrG4TkpOhRRLUyy2iCvv2wdat8OSTsGABooqKUO7iWgRWb0BmagYnUtPJPprO4T3ppB9JI4x0wjSdsC3phG7JcNZJJ5TMvNMKQHo62ZcmcbhaQ7LqNST44njKtm7I2px4/m9rPTp0Di1+pSL3OaLcZJI76OmuXX+VKVcOVJ1YTpyAO++E4cOdpNylC3TuDBddBGI9885mRe6mrKpvu9n2aumEY8w5prgdDkScGzEXXABDhsCiRZCZiYSEEDL9aSoWeENKCuSOshMS4lQQ6taFX3fAjh2wY1sOQSmfM+idKwjKySSbIN7QPpT//TDxv39D7eVvEICSADQmkM2j41hxQUNSqzfkSLV4jtduSE69BkTGRFDrjxSqbkyGpCTC2jan/M61BP2cL5n88IMzxhyggYFIfLwTXELCX8uGDdCp018jcI8c6QS6dCm8/75zUdWqOYmmSxfn/VWqlN7vxXhFUbsp/xM4pKr/51ofjXMfZi3QX1X/KP0QjTnPub/ZU6gilStDy5bgjAubBCmfssVVo+rTLJHff4ftO+CrX4+xYuZG0latI551xLOeiw+so82f7xP0/V8T9eyiCjHsRciB54QcAgjC2X9UItgY3pQNZW7l07QEvtNm/EIjbv9HGJdc4rpnFA2VMyG6VSJBp6rdbd3qJJolS5xk86rr79eLL/4r4Vx6KURElPqP2pSuotZgxgJDAUSkOfAgMBqnR9lkbMh8YzyjEP2qC9P1On+NKgynllO3LvCPMtRtkkCnTgkscNWCli2D+i0y0V83kfH9OjJ/WE+F994iYONuBFCU3XHtWdn8bjaUacZm4jhwKIAff4St+1zny4ann/57GCJQqVIiERFNqLM5okCnhdrExAyi8n8H8XufHA4s+56ugUuJ3bgEnnvOaTIMDnYGSO3c2XnTzp3QrZt1FvAzRU0wscAG1+tewLuq+oSI/B+wuFQjM8Z4lftaUAjSqCFhjRoSdjNwbee8tjgJCaH6nMfoc4bmuoULoVYt5x7/nj3Ov7mv1649RlZWBN9952w7fLhgVAFAC6AFVaveT72mx+kYuIK2x5Zy8YYlVP1sVF5JffhhshteTGDLZkhu5oyL45uDdVn6XUV7zsgHippg0nEergTohNOrDOBwvu3GmLPUGWtBJWiua9Dg74dLTl5LUr6JfTIy/kpAzzwDs2c7/R1yb0kFlQ3nnT1deH5PF/btm8jDjOQBHiOQHBRh+7qjhKxbTg1m5x2zFRBHBTZLXb6/qB5B8XXJrl2XgPp1CYmvS2StaCpsWEmNBafvMm0PxBZdURPMF8BkEVkBtAT6uLbXB34rzcCMMX6qtJrr3AgNdR5irVED7rgD3njjr5rQ88+ffMwTJ+DI4u5I7ynkZGWSExTC14NfY014Igd2Hidg+1Yy122i4sFN1GUTdXUTddenELt+PoHk5B0njTKEc5zaKCdmvMK7ETexrVwCmWWjOVGuEjkVo/lTo5m/NJqD2VEEBQtjx0KbNlChwl9LVBTIyjNnofMpURU1wdwDPI+TWO5U1dx+h1dgTWTGmFJ0pspSUBBU7J4Iy51CAUlJ3JSY6LoRHA40JCWl4UnNdfPnQ0ZsJhkbtpG9YRMBWzZx4efzidiSggBCNr2Ovk7g0Tl/i2cakEUQ+7Ki2fdQNPuIZhPR7KcS+4gmkjTu5nkCOcEJCWZyo1fYViuJ4MoVKHtBGBUqwIEDMHWqkxyDg2HuXKfPQmTkudkju6jPwfwOXO1m+9BSi8gYY1wK23HhVIVOdV+JpvVxGl6AlFYndZkOXLoUGjVynkNyLZtW7uOlx/dTIXsfFwTso3PCPmpk7SPgwFqCD+8jNG0/AfpXrShQM3jw55vgZ2c9nVAOUoFDlKcHFZzXGeXZ1bsCT1GewwEVyCpTnuyoCtQK2UmN4z/xRKMMdjW+nIqVhIoVoWJFp6ZUsSJs3w4//ghduzqzR3hUSgrVnDEni6xYw/WLyD+AhjgPF69T1eXFOY4xxnhaYe8r/a3LdLlyzjSoQN3u0PNKJ1E1TILYgsfLyXG6VffsCVlZTvXqwQedHm4HDxJ64CDR+w6Rs/Egu1YeIiZnDxexgQvDDhKWcYiAnBxIw1ly7XmJo5+Gs43abCeWHdTkJ9e/ueuPT6hKaJkgKlX6q6mufPmT/z3dtjVr4LPP8iXfzEynmrV/v7N89RWMHk0VqFacn31Rn4OpBizE6daR2zxWVURWA73yNZkZY8zZo6QjcAcEONWJTz9126YnQDDOt/SOFFjiKhKXiNOLITUVDh2CSZNg2jQnYYkQkdiM+AtiqL91O/LbNwQd3HfSaU8QyL4T1TicEcsf+2uy42As27Jr8mtmLIeOHKBq5nesIZ5dVKUS+6nIASqx/6TlcvYTLfs5EX6AoGOpJfkp/k1RazBP44xBVldVtwKISB3gNde+Pqd5rzHGnNuK0wFCxOkhEBUFN94IM2b8NcLBpEkEJCYSkFv26FHWvLeD0f13UDVrO7UDdzDgsu00OL6DBttXwO/znWm+zyC9TAX2U4nfjlViDzGspyENEirRqlslqJRv+f13uOsuND1di/PjKGqC6QIk5SYXAFXdIiJD+GvgS2OMMcVxpvHrIiJIuDGeB2rFk5wMlybBBfmLZGc7Y7498ogzm2tOjlO7GjQIhg0jty0tLDCQHQWeV1o2CXCXGxs0YE/btsVqnSrRlMn55Jy5iDHGmDMqSXNdYKDTx7t/f5gz56/s0a/f3x5EKsQjTXkFdzpTtRRZURPMMuBpEblRVX8DEJGawFPAp8UJwBhjTCkrZPYo7vNKhVXUBDMEWARsEZFdOL3IqgE/AP8u5diMMcYUl6ezRyEU9TmY34DmItIFuAinc8Q6YBMwBfhnqUdojDHmrFSsezCqugRYkrsuIk2B3qUVlDHGmLNfwJmLGGOMMUVnCcYYY4xHWIIxxhjjEYW6ByMi752hSFQpxGKMMeYcUtib/PsLsX/rGcoYY4w5jxQqwajqbZ4OxBhjzLnF7sEYY4zxCK8mGBGpKCILReSoiGwXkZtOU3aYiOwWkcMiMlNEQl3bQ0XkZdf7U0XkexG5wntXYYwxpjC8XYOZBmQCMUBf4HkRaVSwkIh0BUYAnYBaQB1gnGt3EPAb0BEoB4wC3hCRWp4N3RhjTFF4LcGISATO0/6jVDVNVVcA7wG3uCneD3hZVdeq6kFgPNAfQFWPqupYVd2mqjmq+gFOB4MWXrkQY4wxhSKqxZpHpugnEmkGfKWq4fm23Qd0VNWrC5T9AXhUVRe41qOBP4FoVd1foGwMsB1IUNVf3Jx3MDAYICYmpsX8+fNLdB1paWlERkaW6BilyZ/isVjc86dYwL/isVjc86dYAC677LJvVbVlkd+oql5ZgA7A7gLbBgHJbspuBrrlWw/GGbm5VoFywcBS4IXCxNCiRQstqeXLl5f4GKXJn+KxWNzzp1hU/Ssei8U9f4pFVRVYrcX43vfmPZg0/v5AZhTgbhLogmVzX+eVFZEAYA7OPZ17Si9MY4wxpcGbCWYjECQi9fJtawqsdVN2rWtf/nJ71NU8JiICvIzTWaC3qmZ5JmRjjDHF5bUEo6pHgXeAh0UkQkTaAT1waiEFzQZuF5GGIlIBGAnMyrf/eSAeuFpVj3s2cmOMMcXh7W7KdwPhwF5gHnCXqq4VkZoikuaafhlV/QR4AliOcwN/OzAGQERigTuABGC3631pItLXy9dijDHmNIo14VhxqeoBoKeb7TuAyALbpuDMklmw7HacmTSNMcb4MRsqxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEd4NcGISEURWSgiR0Vku4jcdJqyw0Rkt4gcFpGZIhKab989IrJaRDJEZJZXgjfGGFMk3q7BTAMygRigL/C8iDQqWEhEugIjgE5ALaAOMC5fkV3AI8BMD8drjDGmmLyWYEQkAugNjFLVNFVdAbwH3OKmeD/gZVVdq6oHgfFA/9ydqvqOqr4L7Pd85MYYY4pDVNU7JxJpBnylquH5tt0HdFTVqwuU/QF4VFUXuNajgT+BaFXdn6/cI0B1Ve1/mvMOBgYDxMTEtJg/f36JriMtLY3IyMgSHaM0+VM8Fot7/hQL+Fc8Fot7/hQLwGWXXfatqrYs8htV1SsL0AHYXWDbICDZTdnNQLd868GAArUKlHsEmFXYGFq0aKEltXz58hIfozT5UzwWi3v+FIuqf8VjsbjnT7GoqgKrtRjf+968B5MGRBXYFgWkFqJs7mt3ZY0xxvghbyaYjUCQiNTLt60psNZN2bWuffnL7dF8zWPGGGP8m9cSjKoeBd4BHhaRCBFpB/QA5rgpPhu4XUQaikgFYCQwK3eniASJSBgQCASKSJiIBHn8IowxxhSat7sp3w2EA3uBecBdqrpWRGqKSJqI1ARQ1U+AJ4DlwHbXMibfcUYCx3G6Mt/sej3Sa1dhjDHmjLz6V7+qHgB6utm+A4gssG0KMOUUxxkLjC39CI0xxpQWGyrGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xGWYIwxxniEJRhjjDEeYQnGGGOMR1iCMcYY4xFeTTAiUlFEForIURHZLiI3nabsMBHZLSKHRWSmiIQW5zjGGGN8w9s1mGlAJhAD9AWeF5FGBQuJSFdgBNAJqAXUAcYV9TjGGGN8x2sJRkQigN7AKFVNU9UVwHvALW6K9wNeVtW1qnoQGA/0L8ZxjDHG+EiQF89VH8hW1Y35tv0AdHRTthGwqEC5GBGpBNQswnEQkcHAYNdqmohsKGb8uaKBfSU8Rmnyp3gsFvf8KRbwr3gsFvf8KRaABsV5kzcTTCRwuMC2w0DZQpTNfV22iMdBVV8EXixqsKciIqtVtWVpHa+k/Ckei8U9f4oF/Csei8U9f4oFnHiK8z5v3oNJA6IKbIsCUgtRNvd1ahGPY4wxxke8mWA2AkEiUi/ftqbAWjdl17r25S+3R1X3F/E4xhhjfMRrCUZVjwLvAA+LSISItAN6AHPcFJ8N3C4iDUWkAjASmFWM43hCqTW3lRJ/isdicc+fYgH/isdicc+fYoFixiOqWtqBnPpkIhWBmUAXYD8wQlXnikhNYB3QUFV3uMr+F7gfCAfeBu5U1YzTHcdrF2KMMeaMvJpgjDHGnD9sqBhjjDEeYQnGGGOMR1iCKSQRCRWRl11jn6WKyPcicoWPY6onIuki8pov43DFcoOIrHeND7dZRDr4KI5aIvKRiBx0jWX3rIh45XkvEblHRFaLSIaIzCqwr5OI/CIix0RkuYjE+iIWEWkjIktE5ICI/Ckib4rIhZ6M5XTxFCgzRkRURDr7KhYRKSMiz4nIPtc4iJ/7MJZ/uv5PpYrIOhHp6eFYTvsdV5zPsCWYwgsCfsMZMaAcMAp4Q0Rq+TCmacA3Pjw/ACLSBZgI3IbzwOulwBYfhfMcsBe4EEjA+X3d7aVz7wIewemAkkdEonF6Po4CKgKrgQW+iAWogNMjqBYQi/P82CsejuV08QAgInFAH+APH8fyIs7vKN717zBfxCIi1YDXgP/iPOc3HJgrIpU9GMspv+OK+xn25pP8ZzVX9+ix+TZ9ICJbgRbANm/HIyI3AIeAr4C63j5/AeOAh1V1pWt9pw9jqQ08q6rpwG4R+QRn6CGPU9V3AESkJVA9365rgbWq+qZr/1hgn4hcpKq/eDMWVf04fzkReRb4zBMxFCaefJ7F6TX6nK9iEZEGwDVAdVU94tr8rS9icb0+lO/39aGIHAXicP6A8kQsp/uOq0QxPsNWgykmEYnBGV/N6w94ikgU8DBwr7fP7SaWQKAlcIGIbBKR313NUuE+Cukp4AZXU0c14ArgEx/FkqsRznh5QN5/5M14KfGdwaX4+CFlEbkOyFTVj3wZB9Aa2A6MczWR/SQivX0Uy2pgvYhcIyKBruaxDOBHbwVQ4DuuWJ9hSzDFICLBwOvAq576C/QMxuOMNv2bD85dUAwQjNO80QGnWaoZzsOxvvAZzof+CPA7zn/Ud30US64ijZ/nLSLSBBiN0/ziqxgigUeBob6KIZ/qQGOc301V4B7gVRGJ93YgqpqN88D5XJzEMhe4w/XF7nFuvuOK9Rm2BFNEIhKAM2pAJs4H0NvnTwA6A096+9yncNz17zOq+oeq7gOmAFd6OxDX72YxTltxBM6ItBVw7g/5kt+NnycidYGPgf+o6he+igOneXWOqm71YQy5jgNZwCOqmqmqnwHLgcu9HYiro8MTQBIQgnNfZIbr/7+nz+3uO65Yn2FLMEUgIgK8jPNXe29VzfJBGEk4N2h3iMhu4D6gt4h854NYcM3X8zvgD0/sVgRq4NyDyXCNXfcKPkh2BZw0tp44cxrF4aOmKVfvn6XAeFX11hBLp9IJGOLq8bcb5/f3hojc74NYvNb8VAgJwOequlpVc1T1G+BrnD8uPeY033HF+gxbgima53F6l1ytqsfPVNhDXsT5xSa4lunAh0BXH8UDzpf4v0Wksjhjxw0FPvB2EK7a01bgLhEJEpHyOJPX/XD6d5YO1znDgEAgUETCxOkivRBoLCK9XftHAz96snn1VLG47kt9CkxT1emeOn9h48FJMI356/O8C7gDp4ekt2P5HNgBPOAq0w7nD7rFPojlG6BDbo1FRJrhNEF7Ogme6juueJ9hVbWlEAtOl04F0nGqi7lLXx/HNRZ4zccxBOP0/jkE7AaeBsJ8FEsCkAwcxJmw6U2gshd/F1pgGeva1xn4BacZJhmo5YtYgDGu1/k/w2m+/NkUKLcN6OzD31MjIAU4ijM+Yi8fxnIPsAmnGWoLcK+HYzntd1xxPsM2FpkxxhiPsCYyY4wxHmEJxhhjjEdYgjHGGOMRlmCMMcZ4hCUYY4wxHmEJxhhjjEdYgjHmLCLOfCl9fB2HMYVhCcaYQhKRWa4v+ILLyjO/25jzj80HY0zRLAVuKbAt0xeBGOPvrAZjTNFkqOruAssByGu+ukdEPnRNK7tdRG7O/2YRuVhElorIcXGmLZ4lIuUKlOnnmoskQ0T2yN+nGK4ozlTHR0Vki5tzjHadO8M1iORsT/wgjDkTSzDGlK5xwHs4Y6K9CMx2zVaIiJTBmfwsDbgE6AW0Jd90uSJyB/ACzgCiTXBGgi44Yu1oYBHO6LYLgJmuEZJxTZB1H8400fWAq4BVHrhOY87IxiIzppBcNYmbcQYDzG+aqt4vIgrMUNVB+d6zFNitqjeLyCBgEs6UvKmu/Uk4c47UU9VNIvI7zuClI04RgwKPq+oDrvUgnMnVBqvqayLyX5zRiBurb6aTMCaP3YMxpmg+BwYX2HYo3+uUAvtSgO6u1/E4Q5znn6TpKyAHaCgiR4BqwLIzxJA3ZLuqnhCRP4HKrk1vAv8BtorIYpwa03uqmnGGYxpT6qyJzJiiOaaqmwos+wr5XuHUE7Opa39hFKyZKK7/y+pMo90ApxZzBJgMfOuaIMoYr7IEY0zpauNmfb3r9TqgqYjkn8e8Lc7/w/WqugfYiTMJV7Gparqqfqiqw4BWOHOctCvJMY0pDmsiM6ZoQkWkSoFt2ar6p+v1tSLyDc6ETH1wkkVr177XcToBzBaR0UAFnBv676jqJleZCcCTIrIHZ6bSMkAnVZ1cmOBEpD/O/+uvcToTXI9T4/m1iNdpTIlZgjGmaDoDfxTYthOo7no9FuiNM6vnn8Bt6synjqoeE5GuwFScnl3pOL3B/pN7IFV9XkQygXuBicAB4KMixHcIuB+nM0EwTq3pWlXdWoRjGFMqrBeZMaXE1cPrOlV9y9exGOMP7B6MMcb8/3bsmAYAAIBBmH/XqOBrTZCNhcAAsHCRAbCwYABYCAwAC4EBYCEwACwEBoBFt7L+9OX+DeAAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HI5xsU1aFpaj"
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "\n",
    "series = generate_time_series(1, 50 + 10)\n",
    "X_new, Y_new = series[:, :50, :], series[:, 50:, :]\n",
    "Y_pred = model.predict(X_new)[:, -1][..., np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6sAVPISCFpaj",
    "outputId": "9f9078fe-3e98-4a1f-d4f8-423738eb716f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAEUCAYAAACGWlk5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nOzdd3xUVfr48c8zqYSaQockhCYgvUVRIMLaFRY0lljWArZ1f+r6dV1d17KruKy6rquLAhbEABsUwbbqColICSVUkU4ygQCBFEiBtJnz++NmQhISSJlyZzjv1ysvMveeOXMumeSZe8pzRCmFpmmapnmSxdMN0DRN0zQdjDRN0zSP08FI0zRN8zgdjDRN0zSP08FI0zRN8zgdjDRN0zSP08FI0zRN8zhTBiMR+a2IbBSRUhH56DxlHxeRoyJyUkQ+EJGgaufCRORzESkWEauI3O7yxmuapmmNZspgBBwG/gp8cK5CInIV8DQwAYgGYoAXqxV5BygDOgIJwCwRGeCC9mqapmnNIGbOwCAifwW6KaV+U8/5BUCGUuqZyscTgESlVCcRaQnkAxcrpfZUnp8PZCmlnnbLBWiapmkN4u/pBjTTAGBZtcdbgY4iEg5EAjZHIKp2flxdFYnIdGA6QHBw8PDIyEjXtNgE7HY7FotZb4qbT1+fd/Pl6/PlawPYs2dPjlKqfVOe6+3BqBVwstpjx/et6zjnON+6roqUUrOB2QB9+/ZVu3fvdm5LTSQlJYXx48d7uhkuo6/Pu/ny9fnytQGIiLWpz/X2EF0EtKn22PF9YR3nHOcL3dAuTdM0rRG8PRjtAAZXezwYyFZK5QJ7AH8R6V3r/A43tk/TNE1rAFMGIxHxF5FgwA/wE5FgEamrS/Fj4D4R6S8iocCfgI8AlFLFwBLgJRFpKSJjgEnAfLdchKZpmtZgpgxGGEHlNMa07Tsqv/+TiESKSJGIRAIopb4FZgLJgLXy6/lq9TwMtACOAQuBh5RS+s5I0zTNZEw5gUEp9QLwQj2nW9Uq+wbwRj315AGTndk2TdM0zfnMemekaZqmXUB0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeN0MNI0TdM8TgcjTdM0zeNMG4xEJExEPheRYhGxisjt9ZR7t3IrcsdXqYgUVjufIiIl1c7vdt9VaJqmaQ1hym3HK70DlAEdgSHA1yKyVSm1o3ohpdSDwIOOxyLyEWCvVddvlVJzXdtcTdM0ralMeWckIi2BqcBzSqkipdQq4AvgzgY+b57rW6lpmqY5iymDEdAHsCml9lQ7thUYcJ7nTQWOAytrHZ8hIjkislpExjuvmZozrU/P5dX/7iTNmu/ppmia5mailPJ0G84iIpcDi5VSnaodmwYkKKXGn+N5y4GflFIvVDs2GvgFo8vvVuBtYIhSan+t504HpgO0b99+eFJSktOux2yKiopo1aqVp5tRw+bsct7aXIYCAizwh5HB9Ar1a1JdZrw+Z9LX5718+doA4uLi0pRSI5ryXLOOGRUBbWodawMU1lEWABHpDowDplU/rpRaV+3hPBG5DbgW+FetcrOB2QB9+/ZV48ePb2rbTS8lJQUzXd+a/TnMWbERx8eicjuUtoti/PheTarPbNfnbPr6vJcvX1tzmbWbbg/gLyK9qx0bDOyopzzAXcAapdSB89StAGlm+zQnUEoxK2U/d8xdR2hIAEH+lqofTHR4iEfbpmmae5kyGCmlioElwEsi0lJExgCTgPnneNpdwEfVD4hIOxG5SkSCRcRfRBKAscB3Lmq61kA/7T3Or/6xkr99u4trBnbmu8fHsWBaLA+O74m/n7BqX46nm6hpmhuZtZsO4GHgA+AYkAs8pJTaISKRGGNA/ZVSmQAicgnQDVhcq44A4K/ARYAN2AVMVkrptUYetHZ/Dne9vx4F+FuEey+NplWQP8OjQhkeFUphSTn/2XCQ303oTee2LTzdXE3T3MCUd0YASqk8pdRkpVRLpVSkUmpB5fFMpVQrRyCqPLa2slxhrTqOK6VGKqVaK6XaKaVilVL/c/e1aDW9++OBqvEhpRSp6Xk1zj8wtid2BbNXnq/HVdM0X2HaYKT5pvziMjZk5GER8BMI8LcQGxNeo0z3sBAmD+nKwvWZ5BSVeqilmqa5kw5Gmlu9k7yPknIbr988hCeu7Evi/bEMjwo9q9zDcT0prbDz/qp0D7RS09wkMRGio8FiMf5NTDz3cR9m5jEjzccczDvFx2ut3DS8G78e1vWcZXu2b8W1Azszf62VB8f2pG1IgJtaqWkukJgIzz7LuMxMiIyEl182jk+fDqdOGd9brcbj1ath3ryzjwMkJLi/7W6i74w0t3n9+92IwOO/6tOg8o+M70VRaQXz1ma4tF2a5lKJiUYwsVoRpYzgct99MG3amYDjcOoUzJpV9/Fnn3Vfmz1A3xlpbvFz1kmWbjnMQ+N7NniGXP8ubZhwUQdmr9yP3a64vE/7Orv0NM3Unn327OBS2oSx0MzM85fxYvrOSHOLv327i9CQAB4a37NRz5vYvyNFpTb+uXwvCXNTdd46zfvUCiIz+T+SGV9/eT8/khnPTP6v5vHISOe3zUR0MNJcbuWe4/y0N4ffXtGbNsGNG/vJKy4DjLQZZRV2Ug/kuqCFmuZCtYLISDYQTxLJbSZBSK1MIyEhJF/3GvEkMZINNY5XjTP5KB2MNJfamJHHk4u30r51IHfENv6TXWxMOMH+xtvUrqBz22BnN1HTXOvll2sEnThSSAq6i3i1iOTHlkFUFIhAVBTJjy0jfs1jJD2zlbiodOM4wK23+vTkBdDBSHOhNGs+t81J5VhhKSdOlfNzVkGj6xgeFUritFgeHBdDWEggM7/dzZGTp13QWk1zkYQEmD0boqJQlUEn7v07SFoWTPzsifwwN4OSU3aSP8wgfvZEkpIg7uWJkJEBNhtccgl8+y2c9u33vQ5GmsukHsil3GbkWrDbVZO72IZHhfL0Nf345P7RFJVWcM+HGygoKXdmUzXNtRISICODH1esMIJMQgJxcfDBB3DNNdC5M9x0E0Ygiqv2PBGYMQMOH4Z33vFU691CByPNZUb1CAOMFOl1ZVporP5d2vDvhGHsO1bEHXPW8a8Ve/WEBs1rnT4Nf/872O1w4gT4+8OwYXUUHDcOrrrKCEonT7q9ne6ig5HmMi0DjZUD1w7sVG+mhcYa26c9D4yLYVvWSV7/fo+eYaeZ1syZkJxc97nycrjlFvjpJwgIMG6cjh0zglFJSR1PePllyMsjuesdzJQ/+GRWBh2MNJfZkGEkQP3jtf2cuj4oJNC/at8jPcNOM6uRIyE+/uyAZLcba16//NJ4/Mor8MknRkA6cABGjTKGimrYtYtkywTiiz9gJOvPZGXwoYCkg5HmMuvT8+jargXdQp27UV5sTDhBlTPsLCLN7v7TNFeIizPGgKoHJKXgiSdgfuXObK+/bjwGIyBNmgTbt8NllxllHZJ//xXx9oUkEU8cKcZBH8vKoIOR5hJKKdal51WNGzmTY4ZdpzZB9O7YSmdl0EyrdkBKTIzkn/80zr322plA5LB0KVxxBaSmwtVXG8eSkyE++62agcjBh7Iy6HRAmkuk5xSTU1TqkmAERkC6flAXPk61UlJuIzjAzyWvo2nN5QhIkydDQUEMfn7wt7/B739fd/kffoDRo+H772HqVFi5EpI6/o647JSzC/tQVgZ9Z6S5xPrKDfNcFYwARseEU1ZhZ9sh351hpPmGuDh45BHj+z/8of5ABMZs7jVrjOneS5bAPfdA3OvX15mtwZeyMuhgpLnE+vQ8IloFEhPR0mWvMTI6FBFYpycwaCazMSOPd5L3Vc30TE6GOXPgzjszmD27/ll2Dj/9dGaN69tvQ3KXMwtnAWOfo3ff9amsDKYNRiISJiKfi0ixiFhF5PZ6yv1GRGwiUlTta3xj69Gca116HiOjwxBHOhMXaBcSSN+OrVmfkXf+wprmJmnWfOLfW8tr3+0mYW4q7y0sJD7e6Kq7996MsyY11JacbJxfsgQeeMBI8D11amVAysgwKrLbzwQmH2HaYAS8A5QBHYEEYJaIDKin7FqlVKtqXylNrEdzgqwTp8k6cdqlXXQOo3uEkWbNp9xmd/lraVpDLN2chV0ZyX0L9ofy5EMhNTIr1DXLzsERiBzlX34ZQkOhS5dq5a++GgIDYdkyd1+aS5kyGIlIS2Aq8JxSqkgptQr4ArjTE/VojbPBDeNFDqN6hHOqzMbPWXrcSDOHPdlGDsayzHCOLRvGa7NO1UzxQ90BqXYgAnj/fWPMaMcOY21SfDwkb2xtTLlbtgyUIjnZWGDr7cw6m64PYFNK7al2bCswrp7yQ0UkB8gD5gMzlFIVjalHRKYD0wHat29PSkpKsy/CrIqKilx6fUt/LqWFP2Tv3sTxPa7rpgOoKDUWYyxcvoGTPQIB11+fp+nrM6/DRXbWpZ/msq5+ZO4L4/Y//EzfzgU4Lqf6tYnAM8+0Y9Gi1ogcZNGi7jzzTCEiJ6rKBwS0Y86c/kRHl/Hee4E8+eRuFi0KoWu/fvT59lv+/fR3PPPeFTz//C+kpJzwxCU7j1LKdF/A5cDRWsemASl1lI0BemDc5Q0EfgH+2Nh6qn/16dNH+bLk5GSX1n/Fa8nqng/Xu/Q1qour9Xquvj5P09dnXk/8Z4u66E//VblFpXWeb8q1rVihVLt2Soko9cgjlQcPHVIrGK8iQorVihVNb6+zARtVE//um7KbDigC2tQ61gYorF1QKXVAKZWulLIrpbYDLwE3NbYezTlyikrZf7zYLV10DqN7hLEhIw+bXZ2/sKa5yKH8UyzbksVtoyIJaxnotHrj4ozJDEFB8O9/w6ZNkLynK/H+S0iKfPKsLkBvZdZgtAfwF5He1Y4NBnY04LkKqlKXNacerQkc40Ujo90ZjMIpLKlg19HG75ekac4yZ+UBRGDa2B5Or9sxxgRw442VY0t3fkXcrllw5IjTX88TTBmMlFLFwBLgJRFpKSJjgEkY40E1iMg1ItKx8vuLgOeAZY2tR3OOdel5BAdYGNi1rdte03EXtu6AnuKtecbxwlIWbTjIlKHd6Ny2hUte44YbjL2PsrLg+ush7vEhxomvvnLJ67mbKYNRpYeBFsAxYCHwkFJqh4hEVq4lcuTBmABsE5Fi4BuM4PPK+epx10VcaDZk5DEsMpRAf/e9tbq0a0G30BZVWR80zd0+XJ1Omc3OA+NiXPYaycmwbp0xq3vhQkg+fjH06OEzU7xNG4yUUnlKqclKqZZKqUil1ILK45nKWEuUWfn4SaVUx8pyMUqpPyulys9Xj+Z8BSXl/HKkwK3jRQ6je4SzPiPPMUlF09ymoKSc+WutXDuwMzHtW7nkNRzTvhcvNnaOsNng5ngheegTRjK7oiKXvK47mTYYad4nLSMfpdyzvqi20T3CyCsuY98x7/+l1LzL/LVWCksreGhcT5fUX3v90YMPQkWFkZUhfvl0kksvMbKqejkdjDSnWZeeR4CfMLS7+7d0qBo30l11mhut2ZfDOyv2MTSyHRe7YJy0roWwAwbA2LGwfDksSvIjXhaT/N6ec1fkBXQw0pxmxa5s2rcK4pcj7p/VFhUeQsc2QXrcSHObNGs+d32wnlPlNnZkFVQlRXWmDRtqBiKHhx6C/fuhQvmR1P9FNvwv30ie6sXbketgpDnFmn057Mku4sjJEhLmprrkF/NcRIRRPcJZl56rx400t0g9kENF5do2m91Oqguyxz/11NmBCGDKFOjQAf79zEHi9s7mKTXT2BrWi7cj18FIc4rEdcaOkwoor3DNL+b5jOoRRnZBKcdP62CkuV54qyDAWNQY4G8hNibcba8dGGjkqvtqUxcyyzrWPOml25HrYKQ5xd5jhQjgJ+7/xXSIrRw3Stpd5vY7M+3Cs9l6giB/C7+b0IvE+2MZHuXesdLp00EhzGHa2Se9cDtyHYy0Ztt+6CR7sou4Z0w0T1zZ1yO/mAAFp40Z/RuzbR7pKtQuHIUl5Xyx9TC/HtqVx3/V1+3v95kzIT0drm2RzFzup7x2zuvISK/L5q2DkdZsH6/NICTQj8d+1YdH4np5JBABpFabvOCprkLtwrB0y2FOl9u4bVTk+Qu7wMiRxiy7MdeHcZTOLGXymZMhISQnzCU+3ijnLXQw0polv7iMZZWfENsEB3i0LbEx4QT4GWkJ/Sye6SrUfJ9SigXrMunfuQ2Durkv7VV1jlx1byQPpWOb08wKetw40bIlyY8tI372xDpn4ZmZDkZas/xn40HKKuzcdUm0p5vC8KhQ5t0zCj+BuIvae+wOTfNt2w6dZOeRAm4bHYmIa/frOhdHQCqsaEFy6aXs+tWjJIdcR/zsCV4XiEAHI60ZbHbF/LVWYmPC6NuptaebA8ClvSIY3tGPDRn5VOityDUXWLg+kxYBfkwa0sXTTSEuDuZXpn3+zd5niT/+NkmvH/K6QAQ6GGnNkLzrGFknTpvirqi60Z39ySsuY81+PWakOZdj4sKNg7t4vFvaYcoU6NcP1mV0ZDqziSv60tNNahIdjLQmm7c2g05tgvlV/47nLetOAyP8aB3kz5dbD3u6KZqP+WLrYU6V2bhttGcmLtQlOdnYVgLgbfkdyQu8c38jHYy0Jtl/vIif9uaQMDqSAD9zvY0C/YQrB3Ti2x1HKa2webo5mg9ZuD6Tfp3bMNhDExdqc+Su++wz6NwZ+rfPJn7N/yP5B+9735vrr4jmNeavtRLgJ9zqoamt53PD4M4UllSwck+Op5ui+Yjth07yc1YBt4/q7tGJCw7Vk6hOnAh33w0bcmKYpR4g/mY7ycmebmHj6GCkNdrqfTksWJdJbEw47VsHebo5dRrTK4LQkADdVac5zVvL9+JvEaIjWnq6KXVm877nHrDZLRygF0m/XkR8PF4VkHQw0holzZrPbz5YT5nNzrr0PNNmOQjws3DNwM7875dsTpVVeLo5mpf7fFMW/9uZTYVdMe3jjR5/39eVzbtPHxgzBj4MepDx++aSlGSU8xamDUYiEiYin4tIsYhYReT2esrdLSJpIlIgIodEZKaI+Fc7nyIiJZVblReJyG73XYXvST2QQ7kjU7HN3FkObhjUhdPlNlbsOubppmhe7FD+KZ5btr3qsRmye9SXzfuee2BXaQ/WrbERN7KIp55yf9uayrTBCHgHKAM6AgnALBEZUEe5EOAxIAIYDUwAnqxV5reVW5W3Ukr1dWGbfV5wgB/gmUzFjTWqRxgdWgfprjqtyfKKy7jrg/XYFQT5WzyaCLgh4uMhJNjGB7a7YOVKTzenUfzPX8T9RKQlMBW4WClVBKwSkS+AO4Gnq5dVSs2q9jBLRBIBL1zy5R1+3JNDu5AA7h3TgzG9Ikyd5cDPIlw3qDOJ6zIpKCk3zboQzTucKqvg3o82cCj/NPPvHYW/n4XUA7nExoSb9n3fujXcfJNi0Se38uZ/Xybk2ms93aQGEzNuRCYiQ4E1SqkW1Y49CYxTSt1wnucuBXYppZ6ufJwCDMD4ML8beFYplVLH86YD0wHat28/PCkpyTkXY0JFRUW0atWq0c/LKrLz7KrTTOkdwI09A13QMueofn37Ttj4a2oJ0wYGMqarbwSjpv78vIUZrq/Crnhrcynbj9v47dAghnd0zud2V17bwoXdueiiQkQUjz8+lNkRT9B78Y01ymze3I5du1pz220HXdKGuLi4NKXUiCY9WSllui/gcuBorWPTgJTzPO8e4BAQUe3YaKA1EATcDRQCPc9VT58+fZQvS05ObtLznv5sm+r97Dcqp7DEuQ1ysurXZ7fb1aUzlqu7P1jnuQY5WVN/ft6iode3fOdR9dbyPWpjRp5TX39jeq669p8rVdQfvlKJqVan1u3Kn92KFUpFRCi1fLlSMWF5Ko7lSh05ctb5FStc1gQFbFRN/Ltv1jGjIqBNrWNtMAJJnURkMvAqcI1SqmpxiVJqnVKqUClVqpSaB6wGvOfe1STyistYsukQU4Z2rdrh0huICDcM7sJPe47z2ne7PT4LSnOOH37J5t6PNvL693uctndVdkEJL3+zk5vfW8uOwwX4W8Q0ORcbwpE49ZZbYFyUlWSu4EDnSyE6muRnfzhrKrjZmHLMCNgD+ItIb6XU3spjg4EddRUWkauBOcB1SqntdZWpRmF02WmNsGCdldIKO/de1sPTTWm0nu1bYlPwTvI+5q464LHN/zTnmfndrqrvyypntzXlZ/r1tsMkbTzEkZOn2ZNdVOOcUqrJ9XpKXBwkTf+Bm14ZBtiZx92Mt6YQ/8pgkp75gbi4iZ5uYr1MeWeklCoGlgAviUhLERkDTALm1y4rIlcAicBUpdT6WufaichVIhIsIv4ikgCMBb5z/VX4jrIKOx+vtXJ57wj6dPSeT4oOxwpLAONTiBmm5WrNs3xnNnuyi/C3GJ8p7QoGdml8ep5/p+zjkQWb+XHPcfYeK+K2kd1585YhBAeYf9bcucQl3s+nTCWACt7k/xFPEknEE5d4v6ebdk6mDEaVHgZaAMeAhcBDSqkdIhJZuV7IkYfmOaAt8E21tUT/rTwXAPwVOA7kAI8Ck5VSeq1RI3y9/TDHCku5zwvvigBiYyKo/LvltX9gNMOpsgr+vGwHvTu0IvH+0dx1SRQC/HfH0UbV83PWSd74fk/VYwvQLSyEyUO7knh/LE9c2dd776AzM4kjhRv5ggLaMYHlxJECmZmebtk5mbWbDqVUHlTfS7fqeCbQqtrjentAlVLHAS/aeNd8lFK8vyqdnu1bMrZ3e083p0mGR4Vy75gezF2Vzj/ih3jnHxgNgH/+sJesE6dZ/OAljIwOY3RMOIF+FuauSmfKsK6MjA47bx3HCkqY9vFG2rYIoKi0ggqbvcaHlOFRod79HomMJNnagx8ZRxtOsIQpJDOeuMh0T7fsnMx8Z6SZwPr0PH7OKuDey3pgsXjvUNvU4d0AOFXmfdmMNcMvhwuYuyqdW0d2rxF0Hv9VH7q2a8EzS7ZTVnHuDRVLym1Mm5/GiVPlfHzfKBZM8/K7oDokJ8yt6pp7gRcpJ5ApfE5ywlxPN+2cdDDS6pVmzefZpT/TKsiPKUO7ebo5zdK3Y2vatghgXboeL/JGdrvimc+3065FAE9fc1GNcy2D/Hlp0gD2Hiti9sr99dahlOLJxVvZdugEb946hAFd2jI8KpRH4nr5TiBKhvjZE0l6ZitxUencx/u04SRDLiohfvZEUydO1cFIq1OaNZ/b56Sy71gRJeV2fjlS4OkmNYvFIoyMDmN9ep6nm6I1Upo1n+mfpLHl4An+dH0/2oWcveB6Qr+OXDuwE28u38tfv/7lrKneadZ8Euau46ttR3jqqou4akAndzXfbWpk8n55ImRk0OZfr/AA77FyT0feegtTZ/I27ZiR5lmpB3Iprezy8MYprnWJjQnjh53ZZBeU0LFNsKebozWA40NRaYUdi0BkaEi9ZacM7cY3248y96d05q3J4N4xPejSrgXpOUXMT83EZlf4iTAq2rvfx/WpK5M3Y8fyO67lH/Ik69ZJVSZvM6410sFIq1OPcGPPFm9IiNpQo3sY17AuPY8bB3fxcGu0hli88WDVhyKA1PQ8htczSWF3diFC5RR+m+K9lQfqKKXOWYc3qzND98UX061dMbdFrGPu3Et4/nlzBiJwQTediPxLRL6s43gbEXlBRPpVO/a4iGwTEd1daDI/7jmOv5/w4LiePjO4269za1oF+bNOrzMyvdIKGzP+u5NFGw4igEUg8DwfimJjwgmqXCMU5G9hzp0j2PiniXx8zyiCvSDjtktYLHD55fy+fAbFxfDee55uUP2cemckIj2BB4BL6zg9AngeYzGrw7vAHzByxn3ozLZoTZeZe4pPNx3iztgo/lBrsNib+ftZGBEdyjo9bmRaadZ8EneW8vyGlVhzT3HbqEiuH9SZLQdPnDdb9vCoUBLvjz0rs/bYvu1JnHb28QvFTNvvGWl9gYljS3nrrSCeeAICaw27JScb3Xee3P/I2d10jwFblVIb6zg3FCgFfnEcUEqdFpGPMfYf0sHIJN5O3oufRXhofE9PN8XpRvUII2X3bnKKSonwohx7F4I0az63vLeWCrsCKvjjNRfxwDjjPTimV0SD6qhvjZDXrx1qhpHXdSD+myT+r9suflg5mIUL4e67z5yvPvHBkxrUPSYivUSkXERerHV8logUisgIEQkC7gAW1PH8ncBrGJmzy0VEicinlacXAf1FpK67Ka2WNGs+7yTvc1nCT2tuMZ9tyuL2UZE+OcjvGDfaoO+OTGfp5qzKQAR+QtX3WvPETetFUvDd/H1JDNHR8Npr4Ng5qMYMPA+PJTUoGCml9gFzgcdFJAJARP4M3Av8uvJOKBZoB/xURxV3AQeAL4FLKr9+X3luC1AAXN30y7gwbMjI47bZqbz+/e4mZypOs+bz1f6yep/79op9+FuEh33wrghgYNe2BAdYdFedCWXmFQPGH6ULbmzHlQICiLu8gqQuj5ObCz//DN9/b65ABI2bwPAi4Af8QUTuwxj/uVMp9UPl+ViMiSzb6njuVqAbsEIplVr5ZQVQStkrnxPbxGu4YLy9Yh9lNjt21bSEn45psp/uLef2OWcHs4ycYpZsziJhdBQdfPCuCIxB8OFRetzIbHKLSlm7P49rLu7ElN4BPjNpxjTGjiUu/QM+/bAAiwUeeMBcgQgaEYyUUkeBNzGSjb4H/E4pVb2XsQtQoJQqq+PpA4BAYFM91R+vfL5Wj/ScYtbsz6m294Uwukfjpqeu2Z9TNU22tMLOW8v3UlJ+Jj3O28nGXdGD42Kc02iTGhUdzq6jBZw8Ve7ppmiVFm04SJnNzu+v7MP1PQN1IHK2sWNBKa4M/JGJE8FqhWuuMU8ggsZP7d6LMe6zVin1Tq1zwRgTFOoyDOOuaUs9509jZOjW6mC3K/7w2TaCA/yYc9cILusVgU0pvtp2xLGbbYMcLzR+PI6psj/uOc4Vr6XwWdohvtyaxWebDvGr/h199q7IYXRMGLWyW4UAACAASURBVEoZ3Z6a51XY7HySauWyXhH06uB9W5R4hVGjIDCQ5E+y2LQJWrWChQthxQpPN+yMBgejyn2D3gPWAmNEZHCtIrlAfR9nhgL7lVL15ZQJw9jiQatD4vpM1qfn8dx1/ZnYvyPz7xvF/Zf14KM1Gfzjf3vOXwHGdO2kjQcZERXKlN4BLH7wUhZMG01E6yB+v3grjy7cglLwv1+yfX431CHd2xHoZ2G9Dkam8L9fsjlysoS7LonydFN8V3AwyX0fJH7JLSQlwd/+BhUVMGWKedIDNXQ23TBgKcYkhvFAJvBKrWK7gAARqSujZn+qTemuQw9A7zFUh6wTp3n1m51c1iuCm0cY/7UiwrPX9eOWEd15a8U+nl/28zln2CmleHbpdvxEeOu2odxQ2Q1yac8Ilj48hhsGda4qW2Hz/c3nggP8GNK9nV78ahIfrcmgW2gLJvTr6Omm+KzkZIjf/wpJ9puJG1nEffdB9+7QubN58tWdNxiJSC/gv8D3wKOVY0IvAteKyNhqRVdW/juqjmpOAIMrd12NFZGqaTIi0g7oU+35WiWlFM9+vh27ghlTBiJSbcRIhFemDOTSmHDmrbXy2nf1z7BbuiWLn/bm8NTVF9GlXc3eUItF+M2YHl6/u2VjjY4J4+fDBRSVVni6KRe0XUcLWJeex52xUfh58RYlZlY1a+75X4izL4e1awkKgueeg1274IknzBGQzhmMRKQTRhDaCSRUznwD+BjjTuhVR1mlVAawHrihjqr+DGRj3F2tBfpVO3cdUAZ83qQr8GFLt2SRsvs4T13dl+5hZyeI9LMIsT2NwKGAknI7y3dm1yiTV1zGX77ayZDu7bgjtu5uEMfKdV/b1+VcRvUIw2ZXPt8laXbz1lgJ8rcQP6K7p5vis6oSqD50Efj5wUrjc/9vfgM9esBnn8F//mOU86RzBiOl1FGlVIxSarxSqrTacZtSqp9SqvZC1VnAFBEJqVXPz0qp0UqpFkopUUqtqnb6DmCxUqpGn4mIhInI5yJSLCJWEbm9vnZW5rg7KiInReSDygW4ja7HTJbvzOaPS7bTt1Nr7rokut5yY3pFEBxgqZplN29NBp+lHaqa2PDXr3+h4HQ5r04deM5Pnr62r8v5DI8KxU9gVorrFhBr53byVDlLN2cxeUhXQluevS2E5hxPPVU5a651axg2rCoY/eMfcPPNkJYGhYV1pwJKToaZM93TTmcnKJ0PZAEPN6SwiAwB4jC6/Wp7B+OOqSOQAMwSkQF11HEV8DQwAYgGYmrV16B6zGRDRh7TPt5ISbmdjJxithw8UW9Zx13Nk1f15V+3DaF/lzb8fvFW7vloA699v5slm7KYNKQLF3Vq48YrML+dRwqxA6kH8pq8gFhrnsVpBzldbuOuS/XEBbe5/HJYtw5KShg5Ej74ALp2hT//Gey1Nsl1dO+NHOmepjk1GCmlbBhZGU418CmdgHsqMzxUEZGWwFTgOaVUUeWd1BfAnXXUcTfwvlJqh1IqH/gL8Jsm1GMKdrvipS934MiE0pAJBY67mhsGd+U/0y/hhRv6s2ZfLm+vMP5bv952RP+xrSX1QK7Rt4nRvblk0yHPNugCY7MrPl5rZWR0KAO6tPV0cy4cSkFpKYSEEHdPNEnTf6CgALZtgyXVUlh7IjuDNGadiruIyFBgjVKqRbVjTwLjlFI31Cq7FXhFKfWfyscRGItoI4DIRtQzHZgOEBISMvzUqYbGU+cKnTCNNiMmoWzGwLqy28he9Cxlh3c1qp524++hzahfI2JB2So4sSqRgtTFrmiyVwrschEdb30Z8QsAEUSE0+mbOLEqERCCIwdSkrm90f/vWsO0HjGZsAn3k7/yYwrWejhD5wXiNozp0NXHUIqB67mCFL4DDgK9gLFAEhAPpDT2ZdKUUiOa0j6zbq7XCjhZ69hJoK4VcbXLOr5v3Zh6lFKzgdkAffv2Vbt3u3+m+Rv/28Nby/dyz5horh/YmdT0PCPl/es7G12Xsc1yKuUVdgKCA1my6D2GRxm/9CkpKYwfP97JrTePhl5fmjWf1AO5DO3eju1ZJ3lvZSAtegzDMbQW6G8x5YQOb//5pVnzufndNdgVdJ7wG1YsfLfG/7G3X9+5ePTaoqON1AvVtASSo/bz57v8+ctfejBpko3Vqx13RI2fXld9xm9jmTUYFQG1BznaAIUNKOv4vrCR9XjUnJUHeGv5XuJHdOO56/pjsUizdqOsb28X7Yzq2wpc2iuCO2KjeDhxEz/uOQ6cyf+n/++cK3nXsapuaP1/7EaZmfUef+EFmDsXli2Dp5/2TJogs+6wugfwF5He1Y4NBnbUUXZH5bnq5bIrZ+c1ph6PSLPm88D8jbz8zU6uG9iZGVMGYXHSeosLbYZcc7UM8ud3E3rjX/n/7+d3Yay5crcKmzFSbrmA1rWZQmRkvcd//BEcIxP/+pdn1hyZMhgppYoxdoR9SURaisgYYBLGbL3aPgbuE5H+IhIK/An4qAn1uF2aNZ/bZqfy3Y5sLAJ3XaIX/nna8KhQPrh7JAF+wiUxYTqQu8CWQyfo0i6Y31/Zx5TdoD7r5ZchpNZ6xZAQkhPmEh8Pn38ON95ozKq7+Wb3ByRTBqNKD2MkTz0GLAQeUkrtEJFIESkSkUgApdS3wEwgGbBWfj1/vnrcdxn1Sz2QS1nlp0QBNuoZb6Ywtm97EkZHsWZ/LtkFJZ5ujk85erKEdel53DIikkfieutA5E4JCTB7NkSdmUqffNts4mdPrJo197e/QVkZXHaZ+7MymDYYKaXylFKTlVItlVKRSqkFlcczlVKtlFKZ1cq+oZTqqJRqo5S6p9YC3TrrMYPYmPCqxaq6u8Jc7rusBza74qM1GZ5uik/5atthlIIbh+gdYzwiIQEyMiAnh2TLBOIXTKoxffuii4y9jr76ytgR1p0BybTB6EIQE9ESBVzeK0J3V5hM97AQrr64E4mpVop1/jqnWbblMIO6taVHREtPN+XCFh7Ohr53kBRyD3Fjz+xpNnMmTJhg9OZ9/rkxq652miBXZWXQwciDHLuN/r+JurvCjO6/PIaCkgoWbzzolPrSrPnnzK7u6w4cL2J71kluHKzviszgqRdbEpf7KaSkVB0bOfLMLrDLloHFUjNNkCuzMuhg5EGpB3IJDrAwqFs7TzdFq8OwSGPq9werM7DZm7c43LHl++vf159d3dd9sfUwInD9IB2MTOH666FNG/jkk6pDcXHG3dCyZRARAU8+eSZNkKuzMuhg5EGpB3IZERVGoL/+MZjV/Zf1IDPvFN/vONqsen7cfYzSCjt2ZaQfWrQhs1G79Ho7pRRfbDnM6B5hdGrr2zsJe40WLeCmm4y03dUyzjgCUmkpbNwIixa5Jz2Q/ivoIXnFZew6WkhsTNMXtmqud+WATkSGhTDnpwNNrsNmV6zca2xk7JiwsnjjIW5+dy2r9uaQlpHn8913Ow4XcCCnmElDunq6KVp1d9xhpOz+8ssah+PijDEjf3948EFjqrer89SZNQODz1ufbiQ/1TPozM3PItw7JpoXvvyFNGt+k8b2Xvt+N1sOnuDBsTG0bhHA8Kh27DtWzDvJ+7jj/XWIGEHKrOmHnGHZliwC/IRrLu7k6aZo1Y0bB926GV11t9xS49SECXD33fD++zBwoOuzMug7Iw9JPZBHiwA/PV7kBW4e0Z2QQD/+uGRbo+9evt52hFkp+7ltVHeevrYfj8T1IjbGSD2U8n/jmdi/I0qBXUFZhW9u+W63K77ceoRxfdrTLkTvW2QqFgvcfjt8+y0cP17jVHKyMXY0dixs325sM+HSpri2eq0+qQdyGREdqseLvMCuo4WUVdjZk13ETe+u4bFFW/hxz3GKSyvOOUNu19ECnly8lWGR7XjhxrO30Ary9+OhcT0JqnwP2BVVqYh8yfqMPI4WlHCDnkVnTnfcARUV0LevEZyio0l+9oeqMaIffjDWH/31r7DYhYn/dTedBzjGi/Qvp3dIPZCLvXKygVLwxdYslm7Jwk/ADqDA3094947hTOjXEYCVu4/zu/9sJijAwqw7hhPk71dn3cOjQlkwLZbkXcf49ucjzPxuN2EtA7nZh7bhfn9VOv4WoX2roPMX1txv2zYQgXzjA1WytQfxrwwm6ZkfiIubCBgT7/btM9bMhofDFVecXY2xOLZLx6Y2QwcjDzgzXqQnL3iD2JhwAv0txnYc/hY+uHskNqV4e8W+qrVi5TbFffM20j2sBVFhLVm9PweljHGgQ/mn6dim/hlkjuzhD43vyYOfpPF/n25j26GTdGob7PUZ19cdyOV/v2QDcO+8DT47JubVnn3W+JQFJDOeeJJIIp64xHR4OQOAa6+Fd9+FoiKYNAm++KLmGJJjth0UN3kjON1H5AGO8aKBXfV4kTdwbMfxxJV9Sbw/lkt7RXB57/Y8dfVFBAdY8BMj6Nx9SRQXd2nLloP5jt9tbA3YqdehZZA/c+8ewZie4cxPtfLad96/JumzajvolvvomJjXq7a1xAZGGoGIFGPvo+hosFiIuyeaZY/+QEAAlJTAN9+ceXr1ad9wssnb8+g7Iw/Q40Xep/reR9WP1bVnVJo1n4Q5qZTb7I3OORjk78clPcNZvT8XBZR6+X4/UjmZ3U9vF2FekZFVm+49xd9rnnNsxme1csU/J5H44H+55e2xfPklvPIKrFrlvPVHOhi5WW5RqR4v8iH1BqlpTd/Y8JKeEQQH7KOk3I5S0CPce/O4HT55mqiwEOJHdvf6Lkef9fLLMH16jYWvdTp1ipu/uIufn8vgpZfgqquMWXbOWn+kP5q72frKMQb9CdG3NWdjQ8cd1wNjYwgJ9GPuqgNVG9J5E5tdsSXzBJf1jtCbPJpZ9a0lRGpsMXGWzExefBGuvNLonnvoIeetP9LByM1SD+RWri9q6+mmaCY2PCqUP17bj1enDmJT5gn+tWKfp5vUaHuPFVJYWqGDkDdwbC1htxv/1heQIiNJToZNm+C552DWLOdtMaGDkZulHshjRHQoAX76v147vxsHd2HK0K78a8Ve0qx5nm5Oo2zMMCZe6GDkheraFTYgoGpX2KQkeOkl419n7Xmk/yK6UW5RKbuzC3UXndYoL04aQNfQFvy/RVsoLCn3dHMabJM1n4hWQUSGhZy/sGYutbvuQkJIto0lflZcjTEiR1JVZwQk0wUjEQkTkc9FpFhErCJy+znK3i0iaSJSICKHRGSmiPhXO58iIiWV25QXichu91xF3fR4kdYUrYMDePOWoRw5WcIjCzbx1f4yr5junZaZz/Codoj4XlaJC0K1rrvkj6zEq0Uk9fwjceNrZpuvHpCgbeumvpzpghHwDlAGdAQSgFkicnYuFUMI8BgQAYwGJgBP1irz28ptylsppfq6qM0N8sXWw/hbBJvd+wajNc8aHhXK1GFdWbknh8/2lpt+/dHxwlKsuacYEaUXdvuCDekRJD2ykriNf3csKKrBEZCgZZNvg00VjESkJTAVeE4pVaSUWgV8AdxZV3ml1Cyl1E9KqTKlVBaQCIxxX4sbLs2az7c/H6XCrrjrg/Wm/kOimVP3yu4uhfkXkDre38P0eJFPeOopiHtzEgwfbkwD7969Ko8diYmAo+vucHZTX8Ns64z6ADal1J5qx7YC4xr4/LHAjlrHZojIq8Bu4FmlVEpdTxSR6cB0gPbt25OSUmexJpu3oxTHzW1ZuZ2FP2ygsKdnMhgXFRU5/frMxFevr8VJG34CNmV04wedsJKScuj8T/SApbvK8LdA3v4tpKQ3rpvOV39+4P3XFjloED3S0pCCAuOA1YrtvvvYvXMnxyZObFbdYqbdJkXkcmCxUqpTtWPTgASl1PjzPPce4C/AEKVUTuWx0cAvGN1+twJvV57ff666+vbtq3bvdu7w0i3vrWFden7VSnRP5uhKSUlh/PjxHnltd/Dl60vefYx7P9zAjUO68M9bh3q6OfWa8u/VWET49KFLG/1cX/75ef21RUefycpQXVQUZGQgImlKqRFNqdqt3XSVEwpUPV+rgCKgTa2ntQHOme9IRCYDrwLXOAIRgFJqnVKqUClVqpSaB6wGrnXuVZ3fvmNFrM/I56bhXavym+nprlpTxPXtwJAOfqzZn2vahbAl5TZ+zirQ73FfVC2PXYOON4Jbu+kacHfTEvAXkd5Kqb2Vhwdzdtdb9edcDcwBrlNKbT9fEziz87PbzF65nyB/C3+8ph/hOo2+1kxjuvjz9pZSVu/PZVyf9p5uzll2HD5Jmc2ux4t8UbU8dmcdbyZTTWBQShUDS4CXRKSliIwBJgHz6yovIldgTFqYqpRaX+tcOxG5SkSCRcRfRBIwxpS+c+1V1HTk5Gk+35xF/IjuOhBpTjG4gx9tWwTwWZo5x4v0YlcfVtdi2JAQ43gzmSoYVXoYaAEcAxYCDymldgCISGTleiFHGH4OaAt8U20t0X8rzwUAfwWOAznAo8BkpZRb1xp9sCodu4Jpl8e482U1HxZgEW4Y3Jnvdhw15SLYNGs+0eEhROgPX76nrjx2s2cbx5vJbLPpUErlAZPrOZcJtKr2uN4UfUqp48BIpzewEU6cKmPBukxuGNS5alqupjnDlGHd+CQ1k/9uP0r8SPPsCquUYlNmPuP6dPB0UzRXSUhwSvCpzYx3Rj5j/lorxWU2Hhzf09NN0XzM0O7t6BHRssbmdWZgzT1FTlGZ7qLTGk0HIxc5XWbjozUZxPVtz0Wdak8Q1LTmERGmDO3KuvQ8DuY1eadnp3MsdtXBSGssHYxcZHHaQXKLy3hofC9PN0XzUZOHdgVg6eYsD7fkjI3WfFoH+9O7Q6vzF9a0anQwcoH16bm89t1u+nZqzcho/QlRc43uYSHExoSxZHMWZlm8vsmaz7DIUCwWnRxVaxwdjJwszZpPwtx1FJRUcOB4EZsyT3i6SZoPmzKsG+k5xWw+6Pn32cnT5ew5Vqi76LQm0cHIyVIP5FJuMz6l2u3K1MksNe93zcWdCPQTXvpyh8eT727OzEcpGKGDkdYEOhg5WZ8OxnYegpGDTu9dpLnSnuwiKuyKLQdPenxbia+3HUYEzNFhqHkbHYycLCO3GID7Lu+hc9BpLlf9zrvMg9tKpFnz+WxTFkrBffM2ePwuTfM+Ohg52dItWQzu1pY/XddfByLN5WJjwgn0t9R47Alfbj2MvfKWyOx7LWnmpIORE+07VsiOwwVMGtLV003RLhDDo0JJvD+WS3uGY1fQobVnUvDszTYS6zu2SNHd01pj6WDkREs3H8YicP3gzp5uinYBGR4Vyms3D8Yi8J8NB93++vuPF7HmQC6/Hqq3SNGaznS56byVUoplW7MY0yuCDq2DPd0c7QLTpV0LxvVpz+K0gzw2sTf+fu77nPnvZGOLlGev66eTo2pNpu+MnGRTZj4H804zWXfRaR5yy8hIsgtKSdl93G2veTDvFEu3ZHH7qCgdiLRm0cHISZZuPkyQv4WrLu50/sKa5gIT+nUgolUQi9zYVTfrx/34ifDAOL1FitY8Ohg5QbnNztfbjzCxf0daBemeT80zAvws3DS8G8m7j5FdUOLy1zty8jSfbjxE/MhudGyju6a15tHByAl+2nucvOIy3UWnedwtI7tjsysWb3T93dF7Px7ArhQPjNVbpGjNp4OREyzdfJh2IQGM69Pe003RLnA9IloSGxPGfzYexG53XS6E44WlLFyfya+HdtUbR2pOYbpgJCJhIvK5iBSLiFVEbj9H2d+IiK3aluNFIjK+KXU1VXFpBf/7JZtrB3ausfhQ0zzltlGRHMw7zZr9rll4mmbN55EFaZRV2Hk4Tm+RojmHGQc43gHKgI7AEOBrEdmqlNpRT/m1SqnLnFRXo/3vl2xOl9t0F51mGlcN6ETbFgEs2pDJZb0jnFp3mjWf2+ekUlphxyKQV1xGj4iWTn0N7cJkqo/yItISmAo8p5QqUkqtAr4A7vRkXecyb20GbYL90du3aGYRHODHr4d25fsd2eQVlzm17tQDuZRV2Gs81jRnMNudUR/AppTaU+3YVmDcOZ4zVERygDxgPjBDKVXR2LpEZDowHaB9+/akpKSct7Gbs8vZnGn8st8+ey1PjQymV6jfeZ/naUVFRQ26Pm+lrw96ip0ym53fzFrBjT0DnPa+DDphq/reXyDohJWUlENOqdvBl39+vnxtzWW2YNQKOFnr2EmgdT3lVwIXA1ZgAPAfoAKY0di6lFKzgdkAffv2VePHjz9vY//579UYvYBgU1DaLorxXrDNeEpKCg25Pm+lr8/oTpM1a9iWY2PPSeW0FD2x5Tb+tvE7BnZty7MuSgbsyz8/X7625nJrMBKRFOq/y1kNPAq0qXW8DVBY1xOUUgeqPdwuIi8B/4cRjIoaU1djZZ04zfZDJ/GzCCilk0NqppJ6ILdqYyHH1hLOCBwbMvIotykevaJ3s+uz2+3k5ORw4sQJbLYzd1xt27Zl586dzW2qKXnztfn5+dGuXTsiIiKwWJw/wuPWYKSUGn+u85XjPP4i0lsptbfy8GCgoRMOFMa+dgB7mlnXOb2TvA8ReOf2Yew7VkRsTLhODqmZRmxMOEH+Fkoq7IiI0z4o/bQ3h0A/C6Njwppd16FDhxARoqOjCQgIQMT41S0sLKR16/o6Q7ybt16bUory8nKys7M5dOgQkZGRTn8NU01gUEoVA0uAl0SkpYiMASZhjAWdRUSuEZGOld9fBDwHLGtKXY1xMO8USRsOcuvISK4a0IlH4nrpQKSZyvCoUBKnxdK/c2uC/C0M6FK7k6BpVu45zojoUEICm/85tri4mK5duxIYGFgViDRzEhECAwPp2rUrxcXFLnkNUwWjSg8DLYBjwELgIcdUbBGJrFxL5AjLE4BtIlIMfIMRfF5pSF3N8a8Ve7FYhEf0GgvNxIZHhfKn6/pzqszGdzuONru+YwUl7DpayOW9nbe42xXdPZrruPLnZbYJDCil8oDJ9ZzLxJiY4Hj8JPBkU+pqqoycYj7blMVdl0TRqa3Ox6WZW2xMON1CW7B446Fmb/q4cm8OAGP7OHftkqaBOe+MTO2tFXsJ8BMeGq/zcWnmZ7EIU4d1Y/X+HLJOnG5WXT/tPU5Eq0D6dXJOl5+mVaeDUSPsP17E0s1Z3BkbpTfQ07zGTcO7oRR8ltb09UB2u+KnvTlc3rs9Fr3C27RSUlIQEXJycjzdlEbTwagR3lq+l+AAPx4cp++KNO/RPSyES3uG82naoSYnT91xuIC84jIud3J6IW+2efNm/Pz8GDNmTKOe98ILL3DxxRe7qFXeSwejBlq6OYtlWw5z1YBOhOsdLTUvc/OIbmTmnWJdel6Tnr9yr7F7rDMnLzhFYiJER4PFYvybmOi2l54zZw4PP/wwP//8s9euHTITHYwaIM2azxNJWwD4ZvsR0qz5Hm6RpjXO1QM60zrIn8VpTdvnaOWe4/Tv3Ib2rU30QSwxEaZPB6sVlDL+nT7dLQHp9OnTLFiwgGnTpnHTTTfx/vvv1zh/+PBhEhISCA8PJyQkhCFDhpCcnExiYiIvvvgiO3bsQEQQET766CPAmD796aef1qgnOjqa1157rerxG2+8waBBg2jZsiVdu3bl/vvv58SJEy6/Xncw3Ww6M/r25yM4ejcqbM5bza5p7tIi0I/rB3dm6ebDvDSpolE7EheVVrApM597L+vhwhYCjz0GW7bQwmYDvwbk0ktNhdLSmsdOnYL77oM5cxr2mkOGwJtvNrqpn376KVFRUQwaNIg777yT+Ph4ZsyYQUBAAMXFxYwbN44OHTrw+eef07VrV7Zu3QrAlClT2LdvH1999VVVjrq2bds2+HUtFgtvvvkmMTExWK1WHn30UR599FHmz2/28kmP08GoATJyTwHgJ+i0P5rXunlEdxauP8jX2w5zy8iGr6BP3Z9LuU0xzmxddLUD0fmOO9HcuXO5805jA4Bx48YREhLCF198wdSpU1mwYAFHjx5l7dq1REQYY2w9exrjzIWFhbRq1Qp/f386derU6Nd97LHHqr6Pjo5m5syZTJo0iXnz5nn9mi0djM7jWGEJP+45zq/6dWRIZDud9kfzWkO7t6Nn+5Z8uDqDnKKyBr+Xf9p7nBYBfgyPdvH7vvIO5XRDU+ZERxtdc7VFRYELM2Pv27eP1atXs3DhQsDoXktISGDu3LlMnTqVzZs3M2jQoKpA5EwrVqxgxowZ7Ny5k5MnT2Kz2SgrK+Po0aN06dLF6a/nTjoYncdHqzMot9l55rp+ehMxzauJCJf2jGB+qpU92bsJ9Lc0KJv3yr05xMaEEeRvsu1RXn7ZGCM6derMsZAQ47gLzZ07F5vNViM/m1JGP/7Bgwervm8sETnrueXl5VXfW61WrrvuOqZNm8ZLL71EeHg4mzZt4rbbbqOszLn7VnmCd9/XuVhhSTnzU61cc3EnHYg0n9A62Pj8aVdQXpnN+1wO5p0iPafYfLPoABISYPZs405IxPh39mzjuItUVFQwb948ZsyYwZYtW6q+tm7dyqBBg/jwww8ZNmwY27Ztq3etT2BgYI0s5Q7t27fnyJEjVY+zs7NrPN64cSNlZWX84x//4JJLLqFPnz4cPnzY+RfpIfrO6BwWrs+ksKRCryvSfMaEfh2ZvfIAFXaFxXL+bN6OKd1j+5gwGIEReFwYfGr7+uuvycnJYdq0aYSH1/y/u/XWW5k1axY7duzg1VdfZfLkycyYMYNu3bqxfft2WrduzYgRI4iOjsZqtbJp0yYiIyNp3bo1QUFBXHHFFbzzzjtceuml+Pn58cwzzxAcfGZxfe/evbHb7bz55ptMmTKF1NRU3mzC5Auz0ndG9SitsPH+qnQu7RnOoG7tPN0cTXOK4VGhLJweS5e2wQT5W4gMCzln+S+2HKZ1kD8nT3l/N5AzvP/++8TFxZ0ViABuvvlmrFYrq1ev5scff6Rr167ccMMNDBgwgOeff74qM/nUqVO59tprmTBh7rPNlgAAEP9JREFUAu3bt68ae3r99deJiYlh/Pjx3HTTTdx///106NChqv5Bgwbxz3/+kzfeeIP+/fszd+7cGtO+vZ00tX/Tl/Xt21f95ZMfeOqzbXx87yjzfipsIl/fbVJf3/ntO1bEtW/9xMR+Hfh3wvA6yyxcn8kfl2wHIDigYeNLjbFz50769et31nFv3fOnIXzh2ur7uQGISJpSakRT6tV3RvV4d+V+BnRpo9OfaD6pV4dWPDaxN99sP8o324+cdX7FrmyeW/pz1eOGjC9pWnPoYFSHUxWKA8eLeXBcT73pl+azpl8ew8Cubfnzsp/JKz7TDZe08SDTPk4jMqwFQf4Wvb5OcwsdjOqQe1rRsXUQ11zc+EVpmuYt/P0s/P3mQZw8Xc5LX+5AKcW/lu/lqU+3cWnPcL549HIWTIvliSv7Or2LTtNq07Pp6mBTkFtcxtZDJ/UvoObTLurUhkfievHmD3vZcvAEGbmnmDykCzNvGkygv4XhUaH6d0BzC9PdGYlImIh8LiLFImIVkdvPUfbdym3IHV+lIlJY7XyKiJRUO7+7oe1QSuk+cu2CcElMOIKR9srPItwRG0Wgv+n+NGg+zozvuHeAMqAjkADMEpEBdRVUSj2olGrl+AIWAotrFftttTJ9G9oI3UeuXSg2WvOpGhpVqsnbTGhac5gqGIlIS2Aq8JxSqkgptQr4ArizEc+d19x2hAaJ7iPXLhixMeEE6okKmoeZbcyoD2BTSu2pdmwrMK4Bz50KHAdW1jo+Q0ReBXYDzyqlUs5XUdsg0YFIu2AMjwol8f5YUg/k6kTAmseYLRi1Ak7WOnYSaMgqsbuBj1XNVbx/AH7B6Pa7FfhSRIYopfbXfrKITAemg5EjKsWFWX89raioSF+fF3PV9Q0QKEw/REq606uuU9u2bSksLDzruM1mq/O4L/CFayspKXHN75dSym1f/7+9uw+OqsrTOP79SQKJQHZEk8gwEgjhVUsgCkoxsFigsGvVjhZTqangDqHCoiBDsb6AJSsExB2jI1s4y0BRSIm4vqTAnV2Lt39ILDKooJgBCSgyEhwEV10VkBBZcvaP22k6oUNiSOf2vTyfqlvS9/TtnKcb+aXvPfccoAJwzWyVwHDgTJNjHgbebOF1bwD+D8ht4Xlbgd+01M8BAwa4MCsvL/e7CwmlfMFQXV0dd//JkycveVxpqXPbt1/6tbdv956XbFrKFgTNfW7OOQe859pYHzr0mpFzbpxzzprZfg58DKSYWf+Yw4YC+1t46V8DO51zf2mpC4DuYhUJsBEjoKAAysvjt5eXe+0jRiTm5xcVFUWXDI/dqqqqEvMDO0BFRQVm1uxM4x0hqQYwOOe+B94AlphZVzMbDfwCaGlN3V8DL8buMLOfmNlEM0szsxQzmwKMBbYloOsi0kHuuAPKyuIXpIZCVFbmPS9RJkyYwPHjxxttN910U5teK3bNoitZUhWjiFlAOvA/eEO1Zzrn9gOYWe/I/ULRVa3MbBTwMy4e0p0KLMUb1PAV8BvgHudcq+81EpHkFK8gdVQhAujSpQvXX399oy0lJYW6ujrmzp1LdnY2aWlp3H777VRWVkaP27FjB2bG5s2bGTlyJJ07d2bbNu/34zfffJNbbrmFtLQ0+vbty4IFCxotmvfDDz/w+OOPk5OTQ5cuXcjNzeX5558HvGtRxcXF9O3bl/T0dPr3788zzzxDfX199Ph9+/Yxfvx4MjIy6N69O0OHDqW8vJwjR45wR+QNy8zMxMwoKipK7BsYR7INYMA597/APc20HcUb5BC7723gopXvnHNfAgn6oi4i7W3uXKiqgvPn0+nUykVlf/pTmDgRevaE48dh8GBYvNjbWmPYsOhq5+1i3rx5lJWVsXbtWnJzc1m2bBmTJk3i0KFD9OzZM/q8+fPn89xzz5GXl0f37t3Ztm0bU6ZMYfny5YwdO5ajR4/ywAMPUFdXF10mYurUqezYsYPly5czfPhwampq+OyzzwCor6+nV69elJWVkZmZya5du5gxYwbXXnstxcXFABQWFjJ06FB27dpFSkoK+/btIy0tjRtuuIGNGzcyefJk9u/fT48ePUhPT2+/N6WVkq4YiYi01jXXeIXo6FHo3dt73BG2bt1Kt24Xfi8eM2YMGzZsYOXKlaxZs4a7774bgFWrVrF9+3ZWrFjB0qVLo88vKSnhrrvuij5+6qmnePTRR5k2bRoA/fr1o7S0lPvuu49nn32WTz75hNdee40tW7YwadIkAHJzc6PHp6amsmTJkujjPn36sGfPHl599dVoMaqpqeGRRx5h0KBBAOTl5UWf36NHDwCysrK47jp/VipQMRKRpNDwDeXUqdpWr/nTcGruiSdg5UpYtCjxp+gAxo4dy+rVq6OP09PTOXz4MOfOnWP06NHR/Z06dWLUqFFUV1c3Ov7WWxsv+fP++++za9cuSktLo/vq6+upra3lxIkTfPDBB1x11VXR02nxrFq1ijVr1lBTU0NtbS3nzp0jJycn2v7QQw8xffp01q1bx/jx45k8eXK0MCWDZLxmJCLSothrREuWND+oIRGuvvpq8vLyoluvXr0abh+Ju+xM031duza+slBfX8+iRYuoqqqKbnv37uXQoUNkZmZGX7s5r7/+OnPnzqWoqIht27ZRVVXFrFmzGl1zKikpobq6mnvuuYedO3dy8803s3bt2ra+Be1OxUhEAifeYIVLjbLrCHl5eXTu3LnRgIXz58/z9ttvM2TIkEsem5+fz8GDBxsVuIYtJSWF/Px86uvrKW8mWGVlJbfddhuzZ88mPz+fvLw8Dh++6N5++vfvz5w5c9i0aRPFxcWsWbMGgM6dO0f76xcVIxEJlEuNmvOzIHXt2pWZM2fy2GOPsXnzZg4cOMDMmTP54osvmDVr1iWPXbhwIa+88goLFy7kww8/5ODBg2zYsIF58+YBXhEpKChg+vTpbNy4kU8//ZQdO3awfr1318uAAQPYs2cPW7Zs4dChQzz55JO89dZb0devra3lwQcfpKKigiNHjvDuu+9SWVkZLZI5OTmYGZs2beLLL7/k9OnTCXqXmqdiJCKBsnv3pYdvNxSk3bs7tl8ApaWlFBQUMG3aNIYNG8bevXvZunVro5F08UycOJFNmzZRXl7OyJEjGTlyJE8//TS9e0fvYuGll16isLCQOXPmMGjQIIqKivjuO2/2tPvvv5+CggIKCwsZMWIER44c4eGHH44e26lTJ7755humTp3KwIEDuffeexk1ahTLli0DoFevXixevJgFCxaQnZ3N7NmzE/DuXJq1dC7ySjRw4ED30UfhvR2poqKCcePG+d2NhFG+YDhw4ACDBw++aP+pU6daPYAhaMKQrbnPDcDM3nfO3Rq3sQX6ZiQiIr5TMRIREd+pGImIiO9UjERExHcqRiLiGw2gCpZEfl4qRiLii9TUVGpra/3uhvwItbW1pKamJuS1VYxExBdZWVkcO3aMM2fO6BtSknPOcebMGY4dO0ZWVlZCfoYmShURX2RkZADw+eefN1pg7uzZs6SlpfnVrYQKcrbU1FSys7Ojn1t7UzESEd9kZGRc9I9bRUUFw4cP96lHiRXmbJdLp+lERMR3KkYiIuK7pCtGZjbbzN4zszoze7EVz/9nMzthZt+Z2Voz6xLT1sPM/tPMvjezGjMrTGjnRUSkTZKuGAGfA0uBFld9MrOJwGPAeKAPkAssjnnKCuAHIBuYAqw0sxvbub8iInKZkq4YOefecM79Efi6FU+fCrzgnNvvnPsGeBIoAjCzrsBk4Ann3GnnXCXw38A/JqbnIiLSVkEfTXcj8F8xj/8MZJvZtUBv4Lxz7uMm7X8b74XMbAYwI/Kwzsw+TEB/k8V1wFd+dyKBlC/YwpwvzNkABrb1wKAXo27AdzGPG/7cPU5bQ3vcxUScc6uB1QBm9l5b1+QIAuULNuULrjBnAy9fW4/t0NN0ZlZhZq6ZrbLlV7jIaSD2JoWGP5+K09bQfqoNP0dERBKoQ4uRc26cc86a2X7ehpfcDwyNeTwU+MI59zXwMZBiZv2btO9vewIREUmEpBvAYGYpZpYGdAI6mVmamTV3OvEloNjMhpjZNcC/AC8COOe+B94AlphZVzMbDfwCWN+Kbqy+3BxJTvmCTfmCK8zZ4DLyWbJNUGhmJcCiJrsXO+dKzKw3UA0Mcc4djTz/IWA+kA5sBB5wztVF2nrgDRG/E2903mPOuVc6JIiIiLRa0hUjERG58iTdaToREbnyqBiJiIjvVIxihG0uu0vN82dm483soJmdMbNyM8vxqZttYmZdzOyFyOd0ysw+MLO/i2kPdD4AM3vZzI6b2Ukz+9jMpse0BT5fAzPrb2ZnzezlmH2Bzxe5leWsmZ2ObB/FtAU+H4CZ/crMDkT+zTxsZmMi+390PhWjxsI2l13cef7M7Dq8kYZPAD2A94DXO7x3lycF+AxvRo2/wctSZmZ9QpIP4LdAH+dcBvAPwFIzuyVE+RqsAHY3PAhZvtnOuW6RbSCEJ5+Z3QmUAtPwJhMYC/ylzfmcc9q8QRxd8QrRgJh964Gn/e5bO2RbCrwY83gGsLNJ9lpgkN99vcyce/HmIwxdPrxpVo4DBWHKB/wKKANKgJcj+0KRD6gApsfZH5Z8O4Hi9sqnb0YXDCD+XHZB/mbUnBvxsgHRe7IOE+CsZpaN9xnuJ0T5zOwPZnYGOIhXjDYTknxmlgEsAR5u0hSKfBG/NbOvzOxPZjYusi/w+cysE3ArkGlmn5jZX83s380snTbmUzG64EfNZRdwocpqZqnAfwDrnHMHCVE+59wsvH6PwTv1UUd48j2JN+v+Z032hyXffLxlbXrh3Qz6ppn1Ixz5soFU4Jd4fzeHAcPxJh5oUz4VowuupLnsQpPVzK7CO536AzA7sjs0+QCcc+edtwTKz4CZhCCfmQ0DJgD/Fqc58PkAnHPvOudOOefqnHPrgD8Bf0848tVG/vt759xx59xXwDIuI5+K0QVX0lx2jeb0i6z91I+AZTUzA17A+y1tsnPuXKQpFPniSOFCjqDnG4e3IOZRMzsBPAJMNrM9hCNfPA4wQpDPeevH/RUvU1Nty+f3RbBk2oDXgFfxLriNxvtqeaPf/bqMPClAGt6orPWRP6cAmZFskyP7SoF3/O5vG/KtAt4BujXZH/h8QBbexf1uePM0TgS+x5tfMQz5rgauj9l+B2yIZAtDvp9EPrOG/+emRD6/gWHIF8m4BG8UZBZwDbAD79Rrm/L5HiiZNrxhiH+M/KU5ChT63afLzFOC95tL7FYSaZuAd1G8Fm/UTx+/+/sjs+VE8pzFOy3QsE0JSb5M4C3gW+AksA/4p5j2QOeLk7eEyGi6MOSLfH678U5NfYv3S9OdYckXyZAK/CGS7wTwPJDW1nyam05ERHyna0YiIuI7FSMREfGdipGIiPhOxUhERHynYiQiIr5TMRIREd+pGImIiO9UjEQCwswyzKzEzAb73ReR9qZiJBIctwKL8O58FwkVFSOR4BiOt4REtd8dEWlvmg5IJADM7AAwqMnujc65X/rRH5H2pmIkEgBmNgJvVvn9wL9Gdh93ztX41yuR9pPidwdEpFX+jLe43u+dc+/43RmR9qZrRiLBcCPQGdjjd0dEEkHFSCQY8vHWb6ryuyMiiaBiJBIMw4HDzrmTfndEJBFUjESCYQga0i0hpgEMIsHwLZBvZhOB74BDzrmvfe6TSLvR0G6RADCzm4AXgJuBNGCMc67S316JtB8VIxER8Z2uGYmIiO9UjERExHcqRiIi4jsVIxER8Z2KkYiI+E7FSEREfKdiJCIivlMxEhER3/0/eNHiAdapDP0AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1QqrBQgqFpam"
   },
   "source": [
    "### Peephole connections\n",
    "- It is a variant of LSTM with peephole connections, which\n",
    "    - feed $\\mathbf{c}_{\\left(t-1\\right)}$ into the controllers, i.e., the FC blocks, of the forget and input gates and also\n",
    "    - feed $\\mathbf{c}_{\\left(t\\right)}$ into the controller of the output gate.\n",
    "- This way, it may or may not work better.\n",
    "- For implementation, use pass `experimental.PeepholeLSTMCell` to the constructor of the `RNN` layer.\n",
    "\n",
    "### GRUs\n",
    "**Gated Recurrent Unit (GRU)** is another variant of LSTM that may performs just as well with these differences:\n",
    "- State vector are merged into one vector.\n",
    "- The gait controller $z$ controls both input and forget gaits by **update gate**.\n",
    "- There is not an output gait, but the new gait $\\mathbf{r}_{\\left(t\\right)}$ controls which part of the previous state can be transferred to the main layer $\\mathbf{g}_{\\left(t\\right)}$.\n",
    "\n",
    "<img src=\"img/gru.jpg\" width=\"400px\">\n",
    "<center>An GRU cell.</center>\n",
    "\n",
    "- Here, the equations are\n",
    "\n",
    "\\begin{equation}\n",
    "\\mathbf{z}_{\\left(t\\right)}=\\sigma\\left({\\mathbf{W}_{xz}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hz}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_z\\right)\\\\\n",
    "\\mathbf{r}_{\\left(t\\right)}=\\sigma\\left({\\mathbf{W}_{xr}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hr}}^T\\mathbf{h}_{\\left(t-1\\right)}+\\mathbf{b}_r\\right)\\\\\n",
    "\\mathbf{g}_{\\left(t\\right)}=\\text{tanh}\\left({\\mathbf{W}_{xg}}^T\\mathbf{x}_{\\left(t\\right)}+{\\mathbf{W}_{hg}}^T\\left(\\mathbf{r}_{\\left(t\\right)}\\otimes\\mathbf{h}_{\\left(t-1\\right)}\\right)+\\mathbf{b}_g\\right)\\\\\n",
    "\\mathbf{h}_{\\left(t\\right)}=\\mathbf{z}_{\\left(t\\right)}\\otimes\\mathbf{h}_{\\left(t-1\\right)}+\\left(\\mathbf{1}-\\mathbf{z}_{\\left(t\\right)}\\right)\\otimes\\mathbf{g}_{\\left(t\\right)}\n",
    "\\end{equation}\n",
    "\n",
    "- To implement it, use `GRU`, which is based on `layers.GRUCell`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3ky4C4z7Fpam"
   },
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(20, return_sequences=True, input_shape=[None, 1]),\n",
    "    keras.layers.GRU(20, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MK04WRIzFpam"
   },
   "outputs": [],
   "source": [
    "model.evaluate(X_valid, Y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PPVb8XmxFpam"
   },
   "outputs": [],
   "source": [
    "plot_learning_curves(history.history[\"loss\"], history.history[\"val_loss\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UVA7BUfaFpam"
   },
   "outputs": [],
   "source": [
    "np.random.seed(43)\n",
    "\n",
    "series = generate_time_series(1, 50 + 10)\n",
    "X_new, Y_new = series[:, :50, :], series[:, 50:, :]\n",
    "Y_pred = model.predict(X_new)[:, -1][..., np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T2JlR_nBFpan"
   },
   "outputs": [],
   "source": [
    "plot_multiple_forecasts(X_new, Y_new, Y_pred)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4V_yjvEbFpan"
   },
   "source": [
    "### Using 1D Convolutional Layers\n",
    "- By downsampling and shortening the sequences, GRU layers may better capture the longer patterns, by making several 1D feature maps (or multi-dimensional sequences).\n",
    "- Depending on the size of kernels and strides, the number of input and outputs to each layer may change."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "id": "-jXSM6nTFpan"
   },
   "source": [
    "1D conv layer with kernel size 4, stride 2, VALID padding:\n",
    "\n",
    "              |-----2-----|     |-----5---...------|     |-----23----|\n",
    "        |-----1-----|     |-----4-----|   ...      |-----22----|\n",
    "  |-----0----|      |-----3-----|     |---...|-----21----|\n",
    "X: 0  1  2  3  4  5  6  7  8  9  10 11 12 ... 42 43 44 45 46 47 48 49\n",
    "Y: 1  2  3  4  5  6  7  8  9  10 11 12 13 ... 43 44 45 46 47 48 49 50\n",
    "  /10 11 12 13 14 15 16 17 18 19 20 21 22 ... 52 53 54 55 56 57 58 59\n",
    "\n",
    "Output:\n",
    "\n",
    "X:     0 to 3     2 to 5     4 to 7   ...  44 to 47   46 to 49\n",
    "Y:     4 to 13    6 to 15    8 to 17  ...  48 to 57   50 to 59"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EMCtVPurFpan",
    "outputId": "c865b194-fe53-430b-b737-5c76af1b644f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 6s 808us/sample - loss: 0.0683 - last_time_step_mse: 0.0605 - val_loss: 0.0482 - val_last_time_step_mse: 0.0405\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 3s 483us/sample - loss: 0.0416 - last_time_step_mse: 0.0342 - val_loss: 0.0368 - val_last_time_step_mse: 0.0283\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 3s 470us/sample - loss: 0.0334 - last_time_step_mse: 0.0251 - val_loss: 0.0307 - val_last_time_step_mse: 0.0220\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 3s 475us/sample - loss: 0.0273 - last_time_step_mse: 0.0172 - val_loss: 0.0251 - val_last_time_step_mse: 0.0141\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 3s 473us/sample - loss: 0.0243 - last_time_step_mse: 0.0134 - val_loss: 0.0238 - val_last_time_step_mse: 0.0128\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 3s 479us/sample - loss: 0.0230 - last_time_step_mse: 0.0121 - val_loss: 0.0226 - val_last_time_step_mse: 0.0116\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 3s 471us/sample - loss: 0.0224 - last_time_step_mse: 0.0116 - val_loss: 0.0220 - val_last_time_step_mse: 0.0110\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 3s 482us/sample - loss: 0.0218 - last_time_step_mse: 0.0111 - val_loss: 0.0216 - val_last_time_step_mse: 0.0107\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 3s 499us/sample - loss: 0.0214 - last_time_step_mse: 0.0107 - val_loss: 0.0211 - val_last_time_step_mse: 0.0103\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 3s 498us/sample - loss: 0.0211 - last_time_step_mse: 0.0105 - val_loss: 0.0209 - val_last_time_step_mse: 0.0102\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 3s 497us/sample - loss: 0.0207 - last_time_step_mse: 0.0103 - val_loss: 0.0206 - val_last_time_step_mse: 0.0099\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 3s 478us/sample - loss: 0.0206 - last_time_step_mse: 0.0102 - val_loss: 0.0203 - val_last_time_step_mse: 0.0097\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 4s 511us/sample - loss: 0.0202 - last_time_step_mse: 0.0098 - val_loss: 0.0200 - val_last_time_step_mse: 0.0095\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 4s 510us/sample - loss: 0.0200 - last_time_step_mse: 0.0096 - val_loss: 0.0207 - val_last_time_step_mse: 0.0105\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 3s 495us/sample - loss: 0.0197 - last_time_step_mse: 0.0095 - val_loss: 0.0198 - val_last_time_step_mse: 0.0093\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 3s 496us/sample - loss: 0.0194 - last_time_step_mse: 0.0091 - val_loss: 0.0191 - val_last_time_step_mse: 0.0087\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 4s 512us/sample - loss: 0.0191 - last_time_step_mse: 0.0089 - val_loss: 0.0188 - val_last_time_step_mse: 0.0085\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 3s 500us/sample - loss: 0.0187 - last_time_step_mse: 0.0085 - val_loss: 0.0183 - val_last_time_step_mse: 0.0077\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 4s 527us/sample - loss: 0.0181 - last_time_step_mse: 0.0078 - val_loss: 0.0178 - val_last_time_step_mse: 0.0074\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 4s 522us/sample - loss: 0.0175 - last_time_step_mse: 0.0072 - val_loss: 0.0173 - val_last_time_step_mse: 0.0067\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Conv1D(filters=20, kernel_size=4, strides=2, padding=\"valid\",\n",
    "                        input_shape=[None, 1]),\n",
    "    keras.layers.GRU(20, return_sequences=True),\n",
    "    keras.layers.GRU(20, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(keras.layers.Dense(10))\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train[:, 3::2], epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid[:, 3::2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fwr81Fy8Fpan"
   },
   "source": [
    "#### WaveNet\n",
    "\n",
    "- WaveNet [3] is a stack of 1D convolutional layers while their dilation rates are doubling to speed up the model (of originally for long audio files), as show below:\n",
    "\n",
    "<img src=\"img/wavenet.jpg\" width=\"400px\">\n",
    "<center>A WaveNet architecture.</center>\n",
    "\n",
    "- It uses the ***causal padding*** (for left padding with zeros to avoid smaller and smaller sequences), as shown below. \n",
    "\n",
    "<img src=\"img/causal.png\" width=\"200px\">\n",
    "<center>Comparing same/valid padding (left) with causal padding (right) [4].</center>\n",
    "\n",
    "- 1d convolutions with causal padding and dilation are often referred to as ***temporal convolution*** [4]."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "id": "9uxkpSYcFpan"
   },
   "source": [
    "C2  /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\.../\\ /\\ /\\ /\\ /\\ /\\\n",
    "   \\  /  \\  /  \\  /  \\  /  \\  /  \\  /  \\       /  \\  /  \\  /  \\\n",
    "     /    \\      /    \\      /    \\                 /    \\\n",
    "C1  /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\ /\\  /\\ /.../\\ /\\ /\\ /\\ /\\ /\\ /\\\n",
    "X: 0  1  2  3  4  5  6  7  8  9  10 11 12 ... 43 44 45 46 47 48 49\n",
    "Y: 1  2  3  4  5  6  7  8  9  10 11 12 13 ... 44 45 46 47 48 49 50  # from these\n",
    "  /10 11 12 13 14 15 16 17 18 19 20 21 22 ... 53 54 55 56 57 58 59  # to these"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "CZJect0aFpan",
    "outputId": "5810255b-b7bb-4743-ebf3-d647b44564b3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/20\n",
      "7000/7000 [==============================] - 3s 377us/sample - loss: 0.0684 - last_time_step_mse: 0.0550 - val_loss: 0.0387 - val_last_time_step_mse: 0.0248\n",
      "Epoch 2/20\n",
      "7000/7000 [==============================] - 2s 277us/sample - loss: 0.0342 - last_time_step_mse: 0.0216 - val_loss: 0.0307 - val_last_time_step_mse: 0.0182\n",
      "Epoch 3/20\n",
      "7000/7000 [==============================] - 2s 280us/sample - loss: 0.0293 - last_time_step_mse: 0.0172 - val_loss: 0.0275 - val_last_time_step_mse: 0.0152\n",
      "Epoch 4/20\n",
      "7000/7000 [==============================] - 2s 275us/sample - loss: 0.0270 - last_time_step_mse: 0.0149 - val_loss: 0.0259 - val_last_time_step_mse: 0.0140\n",
      "Epoch 5/20\n",
      "7000/7000 [==============================] - 2s 276us/sample - loss: 0.0254 - last_time_step_mse: 0.0133 - val_loss: 0.0248 - val_last_time_step_mse: 0.0126\n",
      "Epoch 6/20\n",
      "7000/7000 [==============================] - 2s 274us/sample - loss: 0.0245 - last_time_step_mse: 0.0125 - val_loss: 0.0237 - val_last_time_step_mse: 0.0116\n",
      "Epoch 7/20\n",
      "7000/7000 [==============================] - 2s 278us/sample - loss: 0.0236 - last_time_step_mse: 0.0117 - val_loss: 0.0231 - val_last_time_step_mse: 0.0112\n",
      "Epoch 8/20\n",
      "7000/7000 [==============================] - 2s 278us/sample - loss: 0.0231 - last_time_step_mse: 0.0113 - val_loss: 0.0229 - val_last_time_step_mse: 0.0112\n",
      "Epoch 9/20\n",
      "7000/7000 [==============================] - 2s 270us/sample - loss: 0.0228 - last_time_step_mse: 0.0111 - val_loss: 0.0222 - val_last_time_step_mse: 0.0105\n",
      "Epoch 10/20\n",
      "7000/7000 [==============================] - 2s 275us/sample - loss: 0.0224 - last_time_step_mse: 0.0107 - val_loss: 0.0221 - val_last_time_step_mse: 0.0104\n",
      "Epoch 11/20\n",
      "7000/7000 [==============================] - 2s 270us/sample - loss: 0.0219 - last_time_step_mse: 0.0102 - val_loss: 0.0217 - val_last_time_step_mse: 0.0099\n",
      "Epoch 12/20\n",
      "7000/7000 [==============================] - 2s 275us/sample - loss: 0.0217 - last_time_step_mse: 0.0101 - val_loss: 0.0219 - val_last_time_step_mse: 0.0102\n",
      "Epoch 13/20\n",
      "7000/7000 [==============================] - 2s 278us/sample - loss: 0.0214 - last_time_step_mse: 0.0099 - val_loss: 0.0214 - val_last_time_step_mse: 0.0099\n",
      "Epoch 14/20\n",
      "7000/7000 [==============================] - 2s 278us/sample - loss: 0.0213 - last_time_step_mse: 0.0098 - val_loss: 0.0211 - val_last_time_step_mse: 0.0094\n",
      "Epoch 15/20\n",
      "7000/7000 [==============================] - 2s 275us/sample - loss: 0.0211 - last_time_step_mse: 0.0096 - val_loss: 0.0209 - val_last_time_step_mse: 0.0092\n",
      "Epoch 16/20\n",
      "7000/7000 [==============================] - 2s 274us/sample - loss: 0.0208 - last_time_step_mse: 0.0093 - val_loss: 0.0208 - val_last_time_step_mse: 0.0092\n",
      "Epoch 17/20\n",
      "7000/7000 [==============================] - 2s 272us/sample - loss: 0.0206 - last_time_step_mse: 0.0091 - val_loss: 0.0206 - val_last_time_step_mse: 0.0091\n",
      "Epoch 18/20\n",
      "7000/7000 [==============================] - 2s 276us/sample - loss: 0.0205 - last_time_step_mse: 0.0089 - val_loss: 0.0203 - val_last_time_step_mse: 0.0090\n",
      "Epoch 19/20\n",
      "7000/7000 [==============================] - 2s 282us/sample - loss: 0.0203 - last_time_step_mse: 0.0088 - val_loss: 0.0198 - val_last_time_step_mse: 0.0083\n",
      "Epoch 20/20\n",
      "7000/7000 [==============================] - 2s 274us/sample - loss: 0.0200 - last_time_step_mse: 0.0084 - val_loss: 0.0200 - val_last_time_step_mse: 0.0083\n"
     ]
    }
   ],
   "source": [
    "# simplified implementation\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(keras.layers.InputLayer(input_shape=[None, 1]))\n",
    "for rate in (1, 2, 4, 8) * 2:\n",
    "    model.add(keras.layers.Conv1D(filters=20, kernel_size=2, padding=\"causal\",\n",
    "                                  activation=\"relu\", dilation_rate=rate))\n",
    "model.add(keras.layers.Conv1D(filters=10, kernel_size=1))\n",
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=20,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u9tYUmZxFpao"
   },
   "source": [
    "- The original WaveNet uses ***resedual connections*** [5] and ***skip connections*** to speed up training (figure below).\n",
    "\n",
    "<img src=\"img/original_wavenet.jpg\" width=\"500px\">\n",
    "<center>Original WaveNet [3].</center>\n",
    "\n",
    "- Also it uses the ***gated activation unit*** (originally defined in the gated PixelCNN [6]), which they found outperfoms the ReLU in the case of modeling audio signals:\n",
    "\n",
    "\\begin{equation}\n",
    "\\mathbf{z} =  \\text{tanh} \\left( {W}_{f,k} \\ast \\mathbf{x} \\right) \\otimes \\sigma \\left( {W}_{g,k} \\ast \\mathbf{x} \\right),\n",
    "\\end{equation}\n",
    "\n",
    "- where, $\\ast$ denotes the convolution operator, k denotes the layer index, f and g are respectively the filter and gate, and W is a learnable convolution filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aTENIEa5Fpao"
   },
   "outputs": [],
   "source": [
    "class GatedActivationUnit(keras.layers.Layer):\n",
    "    def __init__(self, activation=\"tanh\", **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.activation = keras.activations.get(activation)\n",
    "    def call(self, inputs):\n",
    "        n_filters = inputs.shape[-1] // 2\n",
    "        linear_output = self.activation(inputs[..., :n_filters])\n",
    "        gate = keras.activations.sigmoid(inputs[..., n_filters:])\n",
    "        return self.activation(linear_output) * gate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JS1ySjYJFpao"
   },
   "outputs": [],
   "source": [
    "def wavenet_residual_block(inputs, n_filters, dilation_rate):\n",
    "    z = keras.layers.Conv1D(2 * n_filters, kernel_size=2, padding=\"causal\",\n",
    "                            dilation_rate=dilation_rate)(inputs)\n",
    "    z = GatedActivationUnit()(z)\n",
    "    z = keras.layers.Conv1D(n_filters, kernel_size=1)(z)\n",
    "    return keras.layers.Add()([z, inputs]), z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nMWY8SpRFpao",
    "outputId": "526bce88-555c-4fde-e579-4b679f3bbfd6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x000001923FEFB248>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "WARNING: AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x000001923FEFB248>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x000001923B0FE6C8>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "WARNING: AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x000001923B0FE6C8>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x0000019239B5C408>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "WARNING: AutoGraph could not transform <bound method GatedActivationUnit.call of <__main__.GatedActivationUnit object at 0x0000019239B5C408>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_layers_per_block = 3 # 10 in the paper\n",
    "n_blocks = 1 # 3 in the paper\n",
    "n_filters = 32 # 128 in the paper\n",
    "n_outputs = 10 # 256 in the paper\n",
    "\n",
    "inputs = keras.layers.Input(shape=[None, 1])\n",
    "z = keras.layers.Conv1D(n_filters, kernel_size=2, padding=\"causal\")(inputs)\n",
    "skip_to_last = []\n",
    "for dilation_rate in [2**i for i in range(n_layers_per_block)] * n_blocks:\n",
    "    z, skip = wavenet_residual_block(z, n_filters, dilation_rate)\n",
    "    skip_to_last.append(skip)\n",
    "z = keras.activations.relu(keras.layers.Add()(skip_to_last))\n",
    "z = keras.layers.Conv1D(n_filters, kernel_size=1, activation=\"relu\")(z)\n",
    "Y_proba = keras.layers.Conv1D(n_outputs, kernel_size=1, activation=\"softmax\")(z)\n",
    "\n",
    "model = keras.models.Model(inputs=[inputs], outputs=[Y_proba])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Fgkodhr2Fpao",
    "outputId": "eca28c07-1dbe-49b1-e2b6-e1005ce68fc2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 7000 samples, validate on 2000 samples\n",
      "Epoch 1/2\n",
      "7000/7000 [==============================] - 4s 523us/sample - loss: 0.1299 - last_time_step_mse: 0.1258 - val_loss: 0.1229 - val_last_time_step_mse: 0.1199\n",
      "Epoch 2/2\n",
      "7000/7000 [==============================] - 2s 304us/sample - loss: 0.1222 - last_time_step_mse: 0.1178 - val_loss: 0.1218 - val_last_time_step_mse: 0.1190\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"adam\", metrics=[last_time_step_mse])\n",
    "history = model.fit(X_train, Y_train, epochs=2,\n",
    "                    validation_data=(X_valid, Y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PlBSs_BHFpap"
   },
   "source": [
    "## Customization\n",
    "\n",
    "### Custom RNN cell\n",
    "\n",
    "- E.g., when nested structures like `[batch, timestep, {\"video\": [height, width, channel], \"audio\": [frequency]}]` are implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oHcOYiCpFpap"
   },
   "outputs": [],
   "source": [
    "# E.g., using random Numpy data (from https://www.tensorflow.org/guide/keras/rnn)\n",
    "\n",
    "unit_1 = 10\n",
    "unit_2 = 20\n",
    "unit_3 = 30\n",
    "\n",
    "i1 = 32\n",
    "i2 = 64\n",
    "i3 = 32\n",
    "batch_size = 64\n",
    "num_batches = 10\n",
    "timestep = 50\n",
    "\n",
    "input_1_data = np.random.random((batch_size * num_batches, timestep, i1))\n",
    "input_2_data = np.random.random((batch_size * num_batches, timestep, i2, i3))\n",
    "target_1_data = np.random.random((batch_size * num_batches, unit_1))\n",
    "target_2_data = np.random.random((batch_size * num_batches, unit_2, unit_3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "a0tprjVxFpap"
   },
   "outputs": [],
   "source": [
    "class NestedCell(keras.layers.Layer):\n",
    "    def __init__(self, unit_1, unit_2, unit_3, **kwargs):\n",
    "        self.unit_1 = unit_1\n",
    "        self.unit_2 = unit_2\n",
    "        self.unit_3 = unit_3\n",
    "        self.state_size = [tf.TensorShape([unit_1]), tf.TensorShape([unit_2, unit_3])]\n",
    "        self.output_size = [tf.TensorShape([unit_1]), tf.TensorShape([unit_2, unit_3])]\n",
    "        super(NestedCell, self).__init__(**kwargs)\n",
    "\n",
    "    def build(self, input_shapes):\n",
    "        # expect input_shape to contain 2 items, [(batch, i1), (batch, i2, i3)]\n",
    "        i1 = input_shapes[0][1]\n",
    "        i2 = input_shapes[1][1]\n",
    "        i3 = input_shapes[1][2]\n",
    "\n",
    "        self.kernel_1 = self.add_weight(\n",
    "            shape=(i1, self.unit_1), initializer=\"uniform\", name=\"kernel_1\"\n",
    "        )\n",
    "        self.kernel_2_3 = self.add_weight(\n",
    "            shape=(i2, i3, self.unit_2, self.unit_3),\n",
    "            initializer=\"uniform\",\n",
    "            name=\"kernel_2_3\",\n",
    "        )\n",
    "\n",
    "    def call(self, inputs, states):\n",
    "        # inputs should be in [(batch, input_1), (batch, input_2, input_3)]\n",
    "        # state should be in shape [(batch, unit_1), (batch, unit_2, unit_3)]\n",
    "        input_1, input_2 = tf.nest.flatten(inputs)\n",
    "        s1, s2 = states\n",
    "\n",
    "        output_1 = tf.matmul(input_1, self.kernel_1)\n",
    "        output_2_3 = tf.einsum(\"bij,ijkl->bkl\", input_2, self.kernel_2_3)\n",
    "        state_1 = s1 + output_1\n",
    "        state_2_3 = s2 + output_2_3\n",
    "\n",
    "        output = (output_1, output_2_3)\n",
    "        new_states = (state_1, state_2_3)\n",
    "\n",
    "        return output, new_states\n",
    "\n",
    "    def get_config(self):\n",
    "        return {\"unit_1\": self.unit_1, \"unit_2\": unit_2, \"unit_3\": self.unit_3}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jYhbDRI7Fpap"
   },
   "outputs": [],
   "source": [
    "cell = NestedCell(unit_1, unit_2, unit_3)\n",
    "rnn = keras.layers.RNN(cell)\n",
    "\n",
    "input_1 = keras.Input((None, i1))\n",
    "input_2 = keras.Input((None, i2, i3))\n",
    "\n",
    "outputs = rnn((input_1, input_2))\n",
    "\n",
    "model = keras.models.Model([input_1, input_2], outputs)\n",
    "\n",
    "model.compile(optimizer=\"adam\", loss=\"mse\", metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Y4_Lf2EvFpaq",
    "outputId": "55eea749-277e-43fe-dfd7-8c23296119bc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10/10 [==============================] - 3s 182ms/step - loss: 0.7372 - rnn_loss: 0.2671 - rnn_1_loss: 0.4702 - rnn_accuracy: 0.0766 - rnn_1_accuracy: 0.0362\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2f7569ab700>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_data = [input_1_data, input_2_data]\n",
    "target_data = [target_1_data, target_2_data]\n",
    "\n",
    "model.fit(input_data, target_data, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ky4IEWyaFpaq"
   },
   "source": [
    "### Custom RNN layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZgF7WHtRFpaq"
   },
   "outputs": [],
   "source": [
    "# for demonstration (from the 3rd ed. of Geron's book)\n",
    "\n",
    "class MyRNN(tf.keras.layers.Layer):\n",
    "    def __init__(self, cell, return_sequences=False, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.cell = cell\n",
    "        self.return_sequences = return_sequences\n",
    "\n",
    "    def get_initial_state(self, inputs):\n",
    "        try:\n",
    "            return self.cell.get_initial_state(inputs)\n",
    "        except AttributeError:\n",
    "            # fallback to zeros if self.cell has no get_initial_state() method\n",
    "            batch_size = tf.shape(inputs)[0]\n",
    "            return [tf.zeros([batch_size, self.cell.state_size],\n",
    "                             dtype=inputs.dtype)]\n",
    "\n",
    "    @tf.function\n",
    "    def call(self, inputs):\n",
    "        states = self.get_initial_state(inputs)\n",
    "        shape = tf.shape(inputs)\n",
    "        batch_size = shape[0]\n",
    "        n_steps = shape[1]\n",
    "        # `@tf.function` requires creating 'outputs' before the `for` loop.\n",
    "        # When the function is converted to a graph, this will be pruned.\n",
    "        # For a similar reason, 'sequences' is created befor the `if` statement.\n",
    "        sequences = tf.TensorArray(\n",
    "            inputs.dtype, size=(n_steps if self.return_sequences else 0))\n",
    "        outputs = tf.zeros(shape=[batch_size, self.cell.output_size],\n",
    "                           dtype=inputs.dtype)\n",
    "        \n",
    "        for step in tf.range(n_steps):\n",
    "            outputs, states = self.cell(inputs[:, step], states)\n",
    "            if self.return_sequences:\n",
    "                sequences = sequences.write(step, outputs)\n",
    "\n",
    "        if self.return_sequences:\n",
    "            # returning the outputs of the shape [time steps, batch size, dims].\n",
    "            return tf.transpose(sequences.stack(), [1, 0, 2])\n",
    "        else:\n",
    "            return outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "eKU99fd6Fpaq",
    "outputId": "b37bc0d0-4b96-46a1-a254-3eb12f0f744f",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "219/219 [==============================] - 5s 18ms/step - loss: 0.1604 - accuracy: 0.1803 - val_loss: 0.0830 - val_accuracy: 0.3080\n",
      "Epoch 2/5\n",
      "219/219 [==============================] - 4s 17ms/step - loss: 0.0738 - accuracy: 0.3522 - val_loss: 0.0682 - val_accuracy: 0.4011\n",
      "Epoch 3/5\n",
      "219/219 [==============================] - 4s 17ms/step - loss: 0.0686 - accuracy: 0.4029 - val_loss: 0.0636 - val_accuracy: 0.4347\n",
      "Epoch 4/5\n",
      "219/219 [==============================] - 4s 17ms/step - loss: 0.0621 - accuracy: 0.4384 - val_loss: 0.0619 - val_accuracy: 0.4340\n",
      "Epoch 5/5\n",
      "219/219 [==============================] - 4s 17ms/step - loss: 0.0642 - accuracy: 0.4333 - val_loss: 0.0600 - val_accuracy: 0.4474\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x2f759280c70>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "custom_model = tf.keras.Sequential([\n",
    "    MyRNN(LNSimpleRNNCell(32), return_sequences=True, input_shape=[None, 1]),\n",
    "    tf.keras.layers.Dense(10)\n",
    "])\n",
    "\n",
    "custom_model.compile(optimizer=\"adam\", loss=\"mse\", metrics=[\"accuracy\"])\n",
    "\n",
    "custom_model.fit(\n",
    "    X_train, Y_train, validation_data=(X_valid, Y_valid), epochs=5\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iWRiw0ypFpaq"
   },
   "source": [
    "## Advanced preprocessing considerations\n",
    "\n",
    "\n",
    "### Avoiding  bias in the test set\n",
    "- Sometimes, e.g., in **natural language processing (NLP)**, the training, validation, and test sets should not be overlapped, and even, it is a good idea to leave a gap between them, e.g., to avoid paragraph overlapping.\n",
    "- On the other hand, for time series we may split in other dimensions rather than times, e.g., across different companies. But it is highly possible that they (in this case, companies) are correlated, making the test sets very biased. Therefore, generally splitting over time is more preferred, e.g., from 2012 to 2014 and from 2014 to 2016.\n",
    "\n",
    "### Considering stationary and nonstationary data\n",
    "- In *stationary* time series, the mean, variance, and ***autocorrelations*** (i.e., correlations between its separated intervals inside that time series) do not change over time, at least in the wide sense. E.g., chemical reactions (as their chemical laws do not change over time).\n",
    "- To ensure that the time series is sufficiently stationary, plot the models’ errors on the validation set across time which should not change significantly.\n",
    "- Time-series with cyclical patterns or particular trends are *nonstationary*, e.g., financial markets. For them, we may work on shorter time spans.\n",
    "\n",
    "### Chopping sequences\n",
    "- We can use `window()` to chop the dataset into smaller datasets (e.g., substrings) to avoid very large sequences.\n",
    "- We may use `shift`, `drop_remainder`, `stride` arguments. The latter defines the stride between elements of each window.\n",
    "- This result in a nested dataset, in which each window dataset can be operated seperately but should be converted to flat dataset (as training on nested datasets is not possible); thus, `flat_map()` can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TBKeQgH_Fpaq",
    "outputId": "cc9b37ae-8a9a-44cf-e9a7-0c321af97b04"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Window ds 0:\n",
      "tf.Tensor(0, shape=(), dtype=int32)\n",
      "tf.Tensor(1, shape=(), dtype=int32)\n",
      "tf.Tensor(2, shape=(), dtype=int32)\n",
      "tf.Tensor(3, shape=(), dtype=int32)\n",
      "tf.Tensor(4, shape=(), dtype=int32)\n",
      "\n",
      "Window ds 1:\n",
      "tf.Tensor(2, shape=(), dtype=int32)\n",
      "tf.Tensor(3, shape=(), dtype=int32)\n",
      "tf.Tensor(4, shape=(), dtype=int32)\n",
      "tf.Tensor(5, shape=(), dtype=int32)\n",
      "tf.Tensor(6, shape=(), dtype=int32)\n",
      "\n",
      "Window ds 2:\n",
      "tf.Tensor(4, shape=(), dtype=int32)\n",
      "tf.Tensor(5, shape=(), dtype=int32)\n",
      "tf.Tensor(6, shape=(), dtype=int32)\n",
      "tf.Tensor(7, shape=(), dtype=int32)\n",
      "tf.Tensor(8, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "# e.g., for demonstration\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# sequence 0 to 14 into windows of length 5, each shifted by 2:\n",
    "n_steps = 5\n",
    "dataset = tf.data.Dataset.from_tensor_slices(tf.range(10))\n",
    "dataset = dataset.window(n_steps, shift=2, drop_remainder=True)\n",
    "\n",
    "# here batch flat_map() makes each window dataset a batch.\n",
    "dataset = dataset.flat_map(lambda window: window.batch(5))\n",
    "\n",
    "for idx, window in enumerate(dataset):\n",
    "    print('\\n'+'Window ds '+str(idx)+':')\n",
    "    for element in window:\n",
    "        print(element)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L3ExEUy3Fpar",
    "outputId": "d83d406a-4811-4417-9182-3ffe74e1b57b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "____________________ Batch 0 \n",
      "X_batch\n",
      "[[0 1 2 3]\n",
      " [4 5 6 7]\n",
      " [2 3 4 5]]\n",
      "===== \n",
      "Y_batch\n",
      "[[1 2 3 4]\n",
      " [5 6 7 8]\n",
      " [3 4 5 6]]\n"
     ]
    }
   ],
   "source": [
    "# now windows are not batchs not datasets.\n",
    "dataset = dataset.shuffle(10).map(lambda window: (window[:-1], window[1:]))\n",
    "dataset = dataset.batch(3).prefetch(1)\n",
    "for index, (X_batch, Y_batch) in enumerate(dataset):\n",
    "    print(\"_\" * 20, \"Batch\", index, \"\\nX_batch\")\n",
    "    print(X_batch.numpy())\n",
    "    print(\"=\" * 5, \"\\nY_batch\")\n",
    "    print(Y_batch.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hGf8y1FVFpar"
   },
   "source": [
    "## Shakespearean text by a character RNN\n",
    "In a blog post [6], it is shown that an RNN model can predict the next character, i.e., ***Char-RNN***.\n",
    "\n",
    "### Data preparation\n",
    "- We first download all the all of Shakespeare’s work of that blog using `keras.utils.get_file()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 376,
     "status": "ok",
     "timestamp": 1669904988829,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "6jVqbJX1Fpar",
    "outputId": "fdfcbd18-58dc-4ac6-f90a-59aef79b154b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "shakespeare_url = \"https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt\"\n",
    "filepath = keras.utils.get_file(\"shakespeare.txt\", shakespeare_url)\n",
    "\n",
    "with open(filepath) as f:\n",
    "    shakespeare_text = f.read()\n",
    "\n",
    "# sample of the text\n",
    "print(shakespeare_text[:148])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G0szUCvvFpar"
   },
   "source": [
    "- Then, every character must be encoded as integers\n",
    "    - using a preprocessing layer, or\n",
    "    - as a simple alternative, using the `keras.preprocessing.text.Tokenizer` class, which supports the `char_level` argument.\n",
    "- Next, we split the data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 18,
     "status": "ok",
     "timestamp": 1669904993274,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "pU8A6CLCFpar",
    "outputId": "ccd49221-0627-4f08-e49e-3b026480019e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"\\n !$&',-.3:;?abcdefghijklmnopqrstuvwxyz\""
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\".join(sorted(set(shakespeare_text.lower())))  # finding the unique letters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1669904995335,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "mstYVtYoFpar",
    "outputId": "666c8f47-4924-4bab-d3b3-86320f6a2da4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'y', 'z', ':', 'm', 'p', 'u', 'i', 't', 's', 'l', 'f', ',', '\\n', 'a', 'k', 'd', '.', 'b', 'w', 'v', 'r', 'c', 'h', '?', 'o', 'n', ' ', 'e'}\n"
     ]
    }
   ],
   "source": [
    "print(set(shakespeare_text[:148].lower()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4KR1L3vhFpas"
   },
   "outputs": [],
   "source": [
    "tokenizer = keras.preprocessing.text.Tokenizer(char_level=True)\n",
    "tokenizer.fit_on_texts(shakespeare_text)\n",
    "\n",
    "# it maps to IDs 1, 2, 3, ... ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 13,
     "status": "ok",
     "timestamp": 1669905001686,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "5H7UB5t8Fpas",
    "outputId": "c811ce42-3b3f-4e98-848e-a6323610a83f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[20, 6, 9, 8, 3]]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.texts_to_sequences([\"First\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 11,
     "status": "ok",
     "timestamp": 1669905001687,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "Xc0T8pN6Fpas",
    "outputId": "4989b9d7-796f-4228-b63d-6112d0793c30"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['f i r s t']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.sequences_to_texts([[20, 6, 9, 8, 3]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vFcCMecZFpas"
   },
   "outputs": [],
   "source": [
    "max_id = len(tokenizer.word_index) # number of distinct characters\n",
    "dataset_size = tokenizer.document_count # total number of characters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "J9UHWMw6Fpas"
   },
   "outputs": [],
   "source": [
    "# encoding the full text\n",
    "\n",
    "[encoded] = np.array(tokenizer.texts_to_sequences([shakespeare_text])) - 1\n",
    "\n",
    "# Splitting the dataset\n",
    "\n",
    "train_size = dataset_size * 90 // 100\n",
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KYahRtWFFpas"
   },
   "source": [
    "### Stateless RNN\n",
    "- For simplification, we first use an stateless RNN (which starts each training iteration with a hidden state full of zero, and update it until only the last time step).\n",
    "\n",
    "#### Preprocessing and training\n",
    "- We should chop the datasets into window, on which the RNN is unrolled (i.e., the truncated backpropagation through time). To get the largest training set, use the `shift=1` argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ayyb8yObFpas"
   },
   "outputs": [],
   "source": [
    "n_steps = 100\n",
    "window_length = n_steps + 1 # target = input shifted 1 character ahead\n",
    "dataset = dataset.window(window_length, shift=1, drop_remainder=True)\n",
    "# now windows have the same length\n",
    "\n",
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "batch_size = 32\n",
    "dataset = dataset.shuffle(10000).batch(batch_size)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "S9Xuxpm3Fpat"
   },
   "source": [
    "- Use embeddings or `tf.one_hot` encoders for the categorical features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "E_M6lWbPFpat"
   },
   "outputs": [],
   "source": [
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1517,
     "status": "ok",
     "timestamp": 1669905016940,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "QHK-SHRSFpat",
    "outputId": "b2781ef3-45e2-4a89-c451-a93228b02be1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(32, 100, 39) (32, 100)\n"
     ]
    }
   ],
   "source": [
    "dataset = dataset.prefetch(1)\n",
    "\n",
    "for X_batch, Y_batch in dataset.take(1):\n",
    "    print(X_batch.shape, Y_batch.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2ylAKce8Fpat"
   },
   "source": [
    "- In the RNN model, the number of outputs is equal to the number of unique characters. These output are probabilities, summing to 1.\n",
    "- Once trained, the model can generate the next character by passing the preprocessed inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4881533,
     "status": "ok",
     "timestamp": 1669909901994,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "t9XURSz_Fpat",
    "outputId": "3124c57a-7997-43b2-8dc1-c2981f9acc51"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "31368/31368 [==============================] - 488s 15ms/step - loss: 1.6214\n",
      "Epoch 2/10\n",
      "31368/31368 [==============================] - 461s 15ms/step - loss: 1.5325\n",
      "Epoch 3/10\n",
      "31368/31368 [==============================] - 456s 14ms/step - loss: 1.5106\n",
      "Epoch 4/10\n",
      "31368/31368 [==============================] - 430s 14ms/step - loss: 1.4988\n",
      "Epoch 5/10\n",
      "31368/31368 [==============================] - 434s 14ms/step - loss: 1.4919\n",
      "Epoch 6/10\n",
      "31368/31368 [==============================] - 444s 14ms/step - loss: 1.4868\n",
      "Epoch 7/10\n",
      "31368/31368 [==============================] - 459s 15ms/step - loss: 1.4832\n",
      "Epoch 8/10\n",
      "31368/31368 [==============================] - 507s 16ms/step - loss: 1.4804\n",
      "Epoch 9/10\n",
      "31368/31368 [==============================] - 450s 14ms/step - loss: 1.4782\n",
      "Epoch 10/10\n",
      "31368/31368 [==============================] - 488s 16ms/step - loss: 1.4760\n"
     ]
    }
   ],
   "source": [
    "# some parts are commented out as non-default arguments may not work on GPU,\n",
    "# which can result in beyond 24h training.\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id],\n",
    "                     # recurrent_dropout=0.2,\n",
    "                     dropout=0.2\n",
    "                    ),\n",
    "    keras.layers.GRU(128, return_sequences=True,\n",
    "                     # recurrent_dropout=0.2,\n",
    "                     dropout=0.2\n",
    "                    ),\n",
    "    keras.layers.TimeDistributed(\n",
    "        keras.layers.Dense(max_id, activation=\"softmax\")\n",
    "        )\n",
    "    ])\n",
    "\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "history = model.fit(dataset, epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pS0yt33yFpat"
   },
   "source": [
    "#### text generation\n",
    "- For *text generation*, if we simply predict the next letters one by one, it will gradually lead to the same results.\n",
    "- Instead, we used the predicted probability to pick the word randomly, e.g., using `tf.random.categorical()`, taking the class ***log probabilities (logits)***, which can be tweaked by the `temperature` parameter.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "U1-fCDLEGTIi"
   },
   "outputs": [],
   "source": [
    "def preprocess(texts):\n",
    "    X = np.array(tokenizer.texts_to_sequences(texts)) - 1\n",
    "    return tf.one_hot(X, max_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 13,
     "status": "ok",
     "timestamp": 1669910100927,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "VSgoV8v0Fpau",
    "outputId": "4a410641-50cd-4974-c2a8-e9bedd2516fd"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first preprocess to pass it to the model.\n",
    "X_new = preprocess([\"How are yo\"])\n",
    "\n",
    "# Y_pred = model.predict_classes(X_new)  # depricated\n",
    "Y_pred = np.argmax(model(X_new), axis=-1)\n",
    "tokenizer.sequences_to_texts(Y_pred + 1)[0][-1] # 1st sentence, last char\n",
    "\n",
    "# it predicted correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 351,
     "status": "ok",
     "timestamp": 1669910103702,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "QmUTIVd3Gb_A",
    "outputId": "4cefeece-75b2-4ea7-f036-416021a62d04"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 1, 0, 0, 0, 0, 1, 1, 0]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for demonstration\n",
    "\n",
    "tf.random.set_seed(42)\n",
    "tf.random.categorical([[np.log(0.5), np.log(0.1)]], num_samples=10).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oI5kusGaGcWP"
   },
   "outputs": [],
   "source": [
    "def next_char(text, temperature=1):\n",
    "    X_new = preprocess([text])\n",
    "    y_proba = model(X_new)[0, -1:, :]\n",
    "    rescaled_logits = tf.math.log(y_proba) / temperature\n",
    "    char_id = tf.random.categorical(rescaled_logits, num_samples=1) + 1\n",
    "    return tokenizer.sequences_to_texts(char_id.numpy())[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 10,
     "status": "ok",
     "timestamp": 1669910107461,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "-rAyRYFCGcYP",
    "outputId": "8eca99dc-3b08-4825-9c32-f0e4b03a306c"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'u'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "next_char(\"How are yo\", temperature=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tHJhHXzKGcaR"
   },
   "outputs": [],
   "source": [
    "def complete_text(text, n_chars=50, temperature=1):\n",
    "    for _ in range(n_chars):\n",
    "        text += next_char(text, temperature)\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 471,
     "status": "ok",
     "timestamp": 1669910111816,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "bX0YQ4ecG4oU",
    "outputId": "a3914f1c-ad7f-4bf3-ced2-59d577908840"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the more than he will command me to her than a toot\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\", temperature=0.2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1257,
     "status": "ok",
     "timestamp": 1669910115785,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "xDcZ2RgEG_nQ",
    "outputId": "b55205e3-896a-4b92-a7b8-5ee385f91218"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "toke the\n",
      "dove execute to be ofr at aol best crase.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1532,
     "status": "ok",
     "timestamp": 1669910122062,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "GcYX7KcuG_pd",
    "outputId": "190483f5-df92-416a-eec6-7deb9cbaa9ad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tpeniomem ly\n",
      "wren? mazala,\n",
      "letvis the involli-faaru\n"
     ]
    }
   ],
   "source": [
    "print(complete_text(\"t\", temperature=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZE9L_ZCkFpau"
   },
   "source": [
    "### Stateless RNN\n",
    "The ***stateful RNNs*** reserves the state for the next batch, enabling to learn long-term patterns. For the simplest implantation:\n",
    "- avoid shuffling,\n",
    "- do not overlap sequences, using `shift=n_steps`,\n",
    "- each batch should contain only one window,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jesXuIPDHRnL"
   },
   "outputs": [],
   "source": [
    "# for demonstration with batch size 1\n",
    "\n",
    "tf.random.set_seed(42)\n",
    "dataset = tf.data.Dataset.from_tensor_slices(encoded[:train_size])\n",
    "dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "dataset = dataset.batch(1)\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch)\n",
    "    )\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XHL5whLJJq9B"
   },
   "source": [
    "- in a batch size > 1, the n<sup>th</sup> input sequence of one batch follows the n<sup>th</sup> input sequence of the previous batch (as shown in figure below),\n",
    "\n",
    "<img src=\"img/stateful.jpg\" width=\"300px\">\n",
    "<center>Preprocessing for a stateful RNN.</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w-1l4UoJz7sX",
    "outputId": "f36ff447-d8f9-42b7-c384-a6e6f295e33a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# first gether the training samples, and split it to get the batch splits\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "encoded_parts = np.array_split(encoded[:train_size], batch_size)\n",
    "\n",
    "len(encoded_parts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7_h3DbRtz7sY",
    "outputId": "2b34c003-69e7-492e-861d-5fd025dd02a0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# then make a dataset from each batch split for preprocessing, make the \n",
    "# windows for each, and append them to a list save the order of datasets\n",
    "\n",
    "datasets = []\n",
    "for encoded_part in encoded_parts:\n",
    "    dataset = tf.data.Dataset.from_tensor_slices(encoded_part)\n",
    "    dataset = dataset.window(window_length, shift=n_steps, drop_remainder=True)\n",
    "    dataset = dataset.flat_map(lambda window: window.batch(window_length))\n",
    "    datasets.append(dataset)\n",
    "    \n",
    "len(datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZeFNMRjsz7sb",
    "outputId": "1bb80d78-1ec4-4b7b-d136-be8212ce770a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<MapDataset shapes: (32, None), types: tf.int32>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# next, make them correctly consecutive,\n",
    "dataset = tf.data.Dataset.zip(tuple(datasets)).map(\n",
    "    lambda *windows: tf.stack(windows)\n",
    "    )\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Xt6IbAuhHRpV"
   },
   "outputs": [],
   "source": [
    "# and the rest of preprocessing\n",
    "dataset = dataset.map(lambda windows: (windows[:, :-1], windows[:, 1:]))\n",
    "dataset = dataset.map(\n",
    "    lambda X_batch, Y_batch: (tf.one_hot(X_batch, depth=max_id), Y_batch))\n",
    "dataset = dataset.prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "H7P9kVkHKCAS"
   },
   "source": [
    "- with the `stateful=True` argument, we can make the model, requiring us to define the batch size,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "06Tanh4PHRrY"
   },
   "outputs": [],
   "source": [
    "# Again, some arguments commented out for GPU acceleration.\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     # recurrent_dropout=0.2,\n",
    "                     dropout=0.2,\n",
    "                     batch_input_shape=[batch_size, None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True, stateful=True,\n",
    "                     # recurrent_dropout=0.2,\n",
    "                     dropout=0.2),\n",
    "    keras.layers.TimeDistributed(\n",
    "        keras.layers.Dense(max_id, activation=\"softmax\")\n",
    "        )\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IgKj7-OuKdeW"
   },
   "source": [
    "- at the end of each epoch, the states should be reset by a custom callback,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pnxlxIlsHRti"
   },
   "outputs": [],
   "source": [
    "class ResetStatesCallback(keras.callbacks.Callback):\n",
    "    def on_epoch_begin(self, epoch, logs):\n",
    "        self.model.reset_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QlZQEiClKkrG"
   },
   "source": [
    "- during training, we typically use more epochs, as they are shorter than the stateless epochs,"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 257638,
     "status": "ok",
     "timestamp": 1669910422420,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "biQDs1pTHRyp",
    "outputId": "e1f850da-fa55-42c6-f5f3-0df1160cb242"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/50\n",
      "313/313 [==============================] - 7s 14ms/step - loss: 2.6211\n",
      "Epoch 2/50\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 2.2422\n",
      "Epoch 3/50\n",
      "313/313 [==============================] - 4s 14ms/step - loss: 2.1109\n",
      "Epoch 4/50\n",
      "313/313 [==============================] - 6s 18ms/step - loss: 2.0358\n",
      "Epoch 5/50\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 1.9854\n",
      "Epoch 6/50\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 1.9480\n",
      "Epoch 7/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.9206\n",
      "Epoch 8/50\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 1.8991\n",
      "Epoch 9/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8817\n",
      "Epoch 10/50\n",
      "313/313 [==============================] - 4s 14ms/step - loss: 1.8661\n",
      "Epoch 11/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8533\n",
      "Epoch 12/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8419\n",
      "Epoch 13/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8338\n",
      "Epoch 14/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8235\n",
      "Epoch 15/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8160\n",
      "Epoch 16/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8100\n",
      "Epoch 17/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.8012\n",
      "Epoch 18/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7960\n",
      "Epoch 19/50\n",
      "313/313 [==============================] - 5s 14ms/step - loss: 1.7923\n",
      "Epoch 20/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7885\n",
      "Epoch 21/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7840\n",
      "Epoch 22/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7787\n",
      "Epoch 23/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7739\n",
      "Epoch 24/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7714\n",
      "Epoch 25/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7683\n",
      "Epoch 26/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7639\n",
      "Epoch 27/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7626\n",
      "Epoch 28/50\n",
      "313/313 [==============================] - 6s 20ms/step - loss: 1.7589\n",
      "Epoch 29/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7567\n",
      "Epoch 30/50\n",
      "313/313 [==============================] - 6s 19ms/step - loss: 1.7535\n",
      "Epoch 31/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7490\n",
      "Epoch 32/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7480\n",
      "Epoch 33/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7451\n",
      "Epoch 34/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7456\n",
      "Epoch 35/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7407\n",
      "Epoch 36/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7402\n",
      "Epoch 37/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7354\n",
      "Epoch 38/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7376\n",
      "Epoch 39/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7342\n",
      "Epoch 40/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7310\n",
      "Epoch 41/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7307\n",
      "Epoch 42/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7302\n",
      "Epoch 43/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7282\n",
      "Epoch 44/50\n",
      "313/313 [==============================] - 5s 17ms/step - loss: 1.7274\n",
      "Epoch 45/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7254\n",
      "Epoch 46/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7209\n",
      "Epoch 47/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7232\n",
      "Epoch 48/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7200\n",
      "Epoch 49/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7222\n",
      "Epoch 50/50\n",
      "313/313 [==============================] - 5s 15ms/step - loss: 1.7188\n"
     ]
    }
   ],
   "source": [
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")\n",
    "history = model.fit(dataset, epochs=50, callbacks=[ResetStatesCallback()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "stYeUf49JJ8n"
   },
   "source": [
    "- also, as these models are restricted to specific batch size, we should make them stateless after training, and we also should get rid of the dropout."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NPMLNvK9JLeA"
   },
   "outputs": [],
   "source": [
    "stateless_model = keras.models.Sequential([\n",
    "    keras.layers.GRU(128, return_sequences=True, input_shape=[None, max_id]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.TimeDistributed(\n",
    "        keras.layers.Dense(max_id, activation=\"softmax\")\n",
    "        )\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Tgfar6K6JLtM"
   },
   "outputs": [],
   "source": [
    "# To set the weights, we first need to build the model\n",
    "stateless_model.build(tf.TensorShape([None, None, max_id]))\n",
    "\n",
    "# Then\n",
    "stateless_model.set_weights(model.get_weights())\n",
    "model = stateless_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1249,
     "status": "ok",
     "timestamp": 1669910439271,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "-ohL1j6kJLvf",
    "outputId": "57f32c33-280e-40a2-9c83-b408187913b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thing addidness of a jisuve,\n",
      "by his paunt too stand\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "print(complete_text(\"t\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2_iaRPJ2JdCa"
   },
   "source": [
    "## Sentiment analysis by word-level methods\n",
    "### data preparation and preprocessing "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6377,
     "status": "ok",
     "timestamp": 1669991059951,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "dpGGiHqbOQdb",
    "outputId": "b3072d72-5a59-4606-a89f-3e447c3aa1b0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17464789/17464789 [==============================] - 1s 0us/step\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the famous dataset of the IMDB reviews with binary labels (negative and positive views)\n",
    "\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "(X_train, y_train), (X_test, y_test) = keras.datasets.imdb.load_data()\n",
    "\n",
    "X_train[0][:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CxFxP-6QOW-_"
   },
   "source": [
    "This dataset is alreathy proprocessed: all punctuation was removed, and then words were lowercased, split by spaces, and finally indexed by frequency, except for:\n",
    "\n",
    "- The integers 0 ==>> padding token\n",
    "- The integers 1 ==>> start-of-sequence\n",
    "- The integers 2 ==>> unknown words\n",
    "\n",
    "If we preferred to preprocess by ourselves, we may use the (default) argument `char_level=False`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 70
    },
    "executionInfo": {
     "elapsed": 392,
     "status": "ok",
     "timestamp": 1669991086327,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "DjoxsP5zOWpm",
    "outputId": "66948553-b3b0-4445-e00c-ae6f3d758d20"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1641221/1641221 [==============================] - 0s 0us/step\n"
     ]
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'<sos> this film was just brilliant casting location scenery story'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_index = keras.datasets.imdb.get_word_index()\n",
    "id_to_word = {id_ + 3: word for word, id_ in word_index.items()}\n",
    "for id_, token in enumerate((\"<pad>\", \"<sos>\", \"<unk>\")):\n",
    "    id_to_word[id_] = token\n",
    "\" \".join([id_to_word[id_] for id_ in X_train[0][:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l3wStt0HOeb-"
   },
   "source": [
    "----------\n",
    "For mobile or web browser deployment, we can use TF Transform (in this case, we can also use `collections.Counter()` to construct the vocabulary)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136,
     "referenced_widgets": [
      "c656e3e19c88426ba2274f9ad6f7593d",
      "345e089df7bb4a708ce9af7fd02f2550",
      "96f2561a16454bda9411a84fd62dc9c1",
      "8f6a9ecf93ed4406a84e4cb386aa627f",
      "acad49e817f046069d5c41f0c57392a1",
      "c9e9e702f51349d7bd24ca2afbb34e30",
      "d89056f2ef3947ecabfdb8f414821916",
      "49f9b321dbae469399eb9ade1a3cf2ec",
      "601bc3b00b484699ba289b354ed37a81",
      "8ece172075644ab383c83565f17ed662",
      "0b32c8300f8c43b59385e833c8f1d14f",
      "0428269d43174d6a8999e52eacf0b3df",
      "2cce0ab2d7d8440d878b601cba37bd3d",
      "3f752b431ca5448c93ecd10717951d18",
      "8b28dca3460a40e4873c3593b472290a",
      "67b93635f5fb4fb0bb346518c1410fa4",
      "c55ea7c2c70b46ac91f06f64bb083fea",
      "d28878c5599e4656af0fc7c04c6e9ab6",
      "1335953a3e8642679e8e2b6dcf4178c0",
      "3ebd4bd4641d424f96a511c586a55330",
      "9c7df8e6470148c2ad4a0ed2d50c7faa",
      "9f0e39f78ea543288e7a887a25cd5996",
      "92f8306c13f643f2a0ea9593449741de",
      "1988f5bb327d404fa16bcb23add04c51",
      "7ae7b2022c4946a2b36ebaf1db93b1da",
      "b1e4df2309574659b8295e0fd4b6f932",
      "9b70efb93ff149258053d0bb88e83b90",
      "4f24f1d28d144c1eb4ae165c12422788",
      "d1775d20113b4eda97c79a787b7aefa9",
      "7c3bc52999644cefbd2e2a1c80acf870",
      "11c5b2c9800e405f816ce77e17d2fd09",
      "194ff23579c344b982a720b5ccd66ce0",
      "f5c625672ff347e9aa8654fcf6c4e8a7",
      "bbac88fda8dd4e62a522f8d3f6049e03",
      "2484a41d0cb84776a9b41253ed4dee9c",
      "c6e19a0129984fef8099664ac826eda5",
      "a7591f8255d64affa8633b7265cc6685",
      "c378d6ce02d54985bbdc07926a480aea",
      "c467d9d0438543eea173ec2a0508edd1",
      "a63674b6a2094038bb50f627cfcc1df9",
      "004cfe137aed42ee91b67eccaf9a6c80",
      "c4295bc4592a43c583e626adc6e5e538",
      "a7d124e655d747ba90d3974215dfcfb9",
      "f385dc04949b4cd48732f35c316e76ac",
      "8e6b13758edc4bbabf8a0c18988c8c65",
      "e2cb78b056de4d21b732d02c4882363e",
      "4e2c552130094e55ab157f5ff99e9572",
      "29d3633bb4b84b4999ad980185d019e4",
      "1f88d6a0092445e8accbf3c0f6e3658f",
      "d66f399c9002437b9bee01932b83c82b",
      "ab02b18ab5114634ac4333bb4fef6584",
      "315e4088282545858818d869ab84f6cf",
      "6156f594196044f792d0c4b46278966c",
      "bc01c4866a3e441f889ece72bf12d98a",
      "c45e34d1b47942b39c2478225901dc98",
      "5b220cd7e2b44885b384dc3eeefa51b9",
      "abf40f3a508c436bbf7bb0b971886aa7",
      "5484f083f29646739026f23f273e2dd4",
      "8300c7b4a8004ebaa21e2600f8e13dd3",
      "cfd0a061e7ad4599a5aec67d7c658422",
      "11ac0a906a2f4dee908a2c5a34e71d32",
      "7a497484c0aa4019afaf46915b861447",
      "1a30c30df15b4c5aabc9bc970a6430a5",
      "402e10e3af474c8a9136edeb36614607",
      "6d3cef428c9840ffa9953f5e764e893f",
      "e6d88215fc8c4e75af0cd4079b9044a0",
      "24d27baa3de648ea8a1ab8ff3c39585c",
      "8120c921568c4986ac25940ab7570661",
      "13a4c68005e44188afe1433f2e748a76",
      "548d5d72690740829094f470de2aca6e",
      "04f3850729e149709343df40449662ce",
      "2d3d1904a204429381c78c582a7bfd0e",
      "13662985bc6643cea402f81f32337421",
      "b5c4e89ca6a64d3f91c60a21a168fafb",
      "6e86db15586b412dac66232041248a21",
      "d3c1d64fd84d4b15838eca22adcb59c4",
      "4c1bcd6ede3d4aae8cd235590cc7da92",
      "711083870fc9494c87f43c8b2d4ef840",
      "37e42ef6ce13464cb4924511a62712a6",
      "28064cce020c48f4a82dd2e4fc593495",
      "5eb299ccedda45678053f9fc9b6fa90f",
      "cdc8e700879b4573b6cffac9838bffde",
      "76ab9c4a636b460a8a757e18781ee69f",
      "6cfedefadfde432286ef09fb4da7ae5d",
      "f9f36b064a1e4858bf4d374f33d92fb9",
      "2ac2dbb64d064ff1837968220a00701e",
      "ac82bbe16c8e4945b4985dacfdae86d7",
      "9123a4b7fd18451eb5a7e16665968252",
      "55fdbaf1e5d84ee081b01359b51fb3ed",
      "6df89ebc143945ac8601c26d76ec105e",
      "198b8c1c8fbb4055b25ceaab843fc060",
      "c7aae61c0ca9490a9c7e0a9d7e763be8",
      "8470cc9b81394da1bda380fe6a49b528",
      "e5dd3a037c3b41ed849fecf223e190d5",
      "6cc320d62c12425da70b687c68fbc54e",
      "41a3251130634aca8d1cf5bf0ddd2a6e",
      "36fb86d798cb4b1e9e87aa4615edac4a",
      "c8dc362bd3e84c0ca39bb79da762a968",
      "4c3be37f55ea4b85bb2ecdcb906e9692"
     ]
    },
    "executionInfo": {
     "elapsed": 69841,
     "status": "ok",
     "timestamp": 1669991176937,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "p23ESo-wOQfp",
    "outputId": "00aef0ab-506a-459a-b461-ce1fed502319"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset 80.23 MiB (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c656e3e19c88426ba2274f9ad6f7593d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Completed...: 0 url [00:00, ? url/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0428269d43174d6a8999e52eacf0b3df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Size...: 0 MiB [00:00, ? MiB/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "92f8306c13f643f2a0ea9593449741de",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating splits...:   0%|          | 0/3 [00:00<?, ? splits/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bbac88fda8dd4e62a522f8d3f6049e03",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train examples...:   0%|          | 0/25000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8e6b13758edc4bbabf8a0c18988c8c65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-train.tfrecord*...…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5b220cd7e2b44885b384dc3eeefa51b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test examples...:   0%|          | 0/25000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24d27baa3de648ea8a1ab8ff3c39585c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-test.tfrecord*...:…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "711083870fc9494c87f43c8b2d4ef840",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating unsupervised examples...:   0%|          | 0/50000 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "55fdbaf1e5d84ee081b01359b51fb3ed",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-unsupervised.tfrec…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset imdb_reviews downloaded and prepared to ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 28,
     "status": "ok",
     "timestamp": 1669991176938,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "dzE-x8X_OQhx",
    "outputId": "51b2889d-ef5d-468f-a203-82439d6486e7"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([Split('train'), Split('test'), Split('unsupervised')])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "datasets.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 21,
     "status": "ok",
     "timestamp": 1669991176940,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "vCHbMxkMOQjz",
    "outputId": "593e42bf-1688-46aa-aec0-fe03c6a4630c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 25000)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_size = info.splits[\"train\"].num_examples\n",
    "test_size = info.splits[\"test\"].num_examples\n",
    "\n",
    "train_size, test_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 469,
     "status": "ok",
     "timestamp": 1669991177397,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "Km6AIjuDOQoz",
    "outputId": "fc76e160-c2d4-4707-dd4f-c01648a1a002"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: This was an absolutely terrible movie. Don't be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting  ...\n",
      "Label: 0 = Negative\n",
      "\n",
      "Review: I have been known to fall asleep during films, but this is usually due to a combination of things including, really tired, being warm and comfortable on the sette and having just eaten a lot. However  ...\n",
      "Label: 0 = Negative\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in datasets[\"train\"].batch(2).take(1):\n",
    "    for review, label in zip(X_batch.numpy(), y_batch.numpy()):\n",
    "        print(\"Review:\", review.decode(\"utf-8\")[:200], \"...\")\n",
    "        print(\"Label:\", label, \"= Positive\" if label else \"= Negative\")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KO3iEVwIOsdG"
   },
   "outputs": [],
   "source": [
    "def preprocess(X_batch, y_batch):\n",
    "    \n",
    "    # truncate the reviews (keep only the first 300 characters)\n",
    "    X_batch = tf.strings.substr(X_batch, 0, 300)\n",
    "    \n",
    "    # use regular expression to replace <br /> tages with space.\n",
    "    X_batch = tf.strings.regex_replace(X_batch, rb\"<br\\s*/?>\", b\" \")\n",
    "    \n",
    "    # replace any any characters other than letters and quotes with spaces\n",
    "    X_batch = tf.strings.regex_replace(X_batch, b\"[^a-zA-Z']\", b\" \")\n",
    "    \n",
    "    # split the reviews by spaces \n",
    "    X_batch = tf.strings.split(X_batch)\n",
    "    \n",
    "    # the ragged tensor is converted to the dense tensor,\n",
    "    # while padding them to make the same lenghts\n",
    "    return X_batch.to_tensor(default_value=b\"<pad>\"), y_batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1669991177398,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "Eu54ln9-Osfm",
    "outputId": "36efb736-fc3c-4182-d71a-08ac4a5bcf2b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2, 53), dtype=string, numpy=\n",
       " array([[b'This', b'was', b'an', b'absolutely', b'terrible', b'movie',\n",
       "         b\"Don't\", b'be', b'lured', b'in', b'by', b'Christopher',\n",
       "         b'Walken', b'or', b'Michael', b'Ironside', b'Both', b'are',\n",
       "         b'great', b'actors', b'but', b'this', b'must', b'simply', b'be',\n",
       "         b'their', b'worst', b'role', b'in', b'history', b'Even',\n",
       "         b'their', b'great', b'acting', b'could', b'not', b'redeem',\n",
       "         b'this', b\"movie's\", b'ridiculous', b'storyline', b'This',\n",
       "         b'movie', b'is', b'an', b'early', b'nineties', b'US',\n",
       "         b'propaganda', b'pi', b'<pad>', b'<pad>', b'<pad>'],\n",
       "        [b'I', b'have', b'been', b'known', b'to', b'fall', b'asleep',\n",
       "         b'during', b'films', b'but', b'this', b'is', b'usually', b'due',\n",
       "         b'to', b'a', b'combination', b'of', b'things', b'including',\n",
       "         b'really', b'tired', b'being', b'warm', b'and', b'comfortable',\n",
       "         b'on', b'the', b'sette', b'and', b'having', b'just', b'eaten',\n",
       "         b'a', b'lot', b'However', b'on', b'this', b'occasion', b'I',\n",
       "         b'fell', b'asleep', b'because', b'the', b'film', b'was',\n",
       "         b'rubbish', b'The', b'plot', b'development', b'was', b'constant',\n",
       "         b'Cons']], dtype=object)>,\n",
       " <tf.Tensor: shape=(2,), dtype=int64, numpy=array([0, 0])>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocess(X_batch, y_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jjmj1V5IOshy"
   },
   "outputs": [],
   "source": [
    "# constructing the vocabulary\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "vocabulary = Counter()\n",
    "for X_batch, y_batch in datasets[\"train\"].batch(32).map(preprocess):\n",
    "    for review in X_batch:\n",
    "        vocabulary.update(list(review.numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1669991192085,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "0Ld6iOJROsj5",
    "outputId": "a3319cf8-0a11-4748-9051-35f63315a316"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(b'<pad>', 214309), (b'the', 61137), (b'a', 38564)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary.most_common()[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 316,
     "status": "ok",
     "timestamp": 1669991196362,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "d3DyTwsdOsmF",
    "outputId": "63677b8f-7f9b-4489-b7f0-77d0a240b690"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53893"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KwODgJVIOsoZ"
   },
   "outputs": [],
   "source": [
    "# we do not need all the words\n",
    "\n",
    "vocab_size = 10000\n",
    "truncated_vocabulary = [\n",
    "    word for word, count in vocabulary.most_common()[:vocab_size]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 3,
     "status": "ok",
     "timestamp": 1669991210930,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "qG3i-Ru-Osqx",
    "outputId": "db911ea8-eebc-4550-8d92-90ef1b6f55e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "22\n",
      "12\n",
      "11\n",
      "10000\n"
     ]
    }
   ],
   "source": [
    "word_to_id = {word: index for index, word in enumerate(truncated_vocabulary)}\n",
    "for word in b\"This movie was faaaaaantastic\".split():\n",
    "    print(word_to_id.get(word) or vocab_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qH9gsTBaOss0"
   },
   "outputs": [],
   "source": [
    "# replace every word with its ID\n",
    "\n",
    "words = tf.constant(truncated_vocabulary)\n",
    "word_ids = tf.range(len(truncated_vocabulary), dtype=tf.int64)\n",
    "vocab_init = tf.lookup.KeyValueTensorInitializer(words, word_ids)\n",
    "num_oov_buckets = 1000\n",
    "table = tf.lookup.StaticVocabularyTable(vocab_init, num_oov_buckets)\n",
    "\n",
    "# Alternatively, in TF transform, we may use compute_and_apply_vocabulary()."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 333,
     "status": "ok",
     "timestamp": 1669991228713,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "vL_HBWPIO9YX",
    "outputId": "2d855be4-04c2-47e1-dcaf-9152f4473753"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(1, 4), dtype=int64, numpy=array([[   22,    12,    11, 10053]])>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.lookup(tf.constant([b\"This movie was faaaaaantastic\".split()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Q8ezkES2O9an"
   },
   "outputs": [],
   "source": [
    "def encode_words(X_batch, y_batch):\n",
    "    return table.lookup(X_batch), y_batch\n",
    "\n",
    "train_set = datasets[\"train\"].batch(32).map(preprocess)\n",
    "train_set = train_set.map(encode_words).prefetch(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 573,
     "status": "ok",
     "timestamp": 1669991241851,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "llapn_N-O9cp",
    "outputId": "6718cd89-9059-406e-aebd-dbd88a4add6a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(\n",
      "[[  22   11   28 ...    0    0    0]\n",
      " [   6   21   70 ...    0    0    0]\n",
      " [4099 6881    1 ...    0    0    0]\n",
      " ...\n",
      " [  22   12  118 ...  331 1047    0]\n",
      " [1757 4101  451 ...    0    0    0]\n",
      " [3365 4392    6 ...    0    0    0]], shape=(32, 60), dtype=int64)\n",
      "tf.Tensor([0 0 0 1 1 1 0 0 0 0 0 1 1 0 1 0 1 1 1 0 1 1 1 1 1 0 0 0 1 0 0 0], shape=(32,), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for X_batch, y_batch in train_set.take(1):\n",
    "    print(X_batch)\n",
    "    print(y_batch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iwiZ-luFPD5c"
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 128042,
     "status": "ok",
     "timestamp": 1669991377243,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "Oq2JV4_BO9e8",
    "outputId": "8c0cade3-3429-4b77-da81-5ad45374553b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 30s 22ms/step - loss: 0.5305 - accuracy: 0.7281\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 19s 24ms/step - loss: 0.3459 - accuracy: 0.8549\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 20s 25ms/step - loss: 0.1934 - accuracy: 0.9313\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 17s 22ms/step - loss: 0.1361 - accuracy: 0.9503\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 22s 28ms/step - loss: 0.1032 - accuracy: 0.9634\n"
     ]
    }
   ],
   "source": [
    "embed_size = 128\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size,\n",
    "                           mask_zero=True,\n",
    "                           input_shape=[None]),\n",
    "    keras.layers.GRU(128, return_sequences=True),\n",
    "    keras.layers.GRU(128),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YLqrvsVDz7sg"
   },
   "source": [
    "------\n",
    "Also, you may use `TensorBoard()` callback to see how words are located next each others.\n",
    "\n",
    "#### Making\n",
    "- Using ***making***, the padding tokens are ignored by all downstream layers (so, the network does not need to learn it by itself). The mask tensor can be propagated to next layers, until the time dimension exists. If the mask reaches to the loss function, its loss is 0.\n",
    "- For implementation either\n",
    "    - Use the layers that can support it (check `supports_masking` attribute), and implement them by `mask_zero=True`, but note that they may not work with the Keras GPU optimization.\n",
    "    - Use custom layers by adding a `mask` argument to its `call()`, and adding `self.supports_masking = True` in the constructor.\n",
    "    - Use `keras.layers.Masking` if the layer does not begin with an Embedding layer: it mask out time steps where the last dimension is full of specifice values, e.g., zeros (equivalent to the implementation by`K.any(K.not_equal(inputs, 0), axis=-1)`).\n",
    "    - It may not handle complex models, e.g., a combination of conv1D and RNN. For them explicitly compute the mask, and pass it the layer, using Functional or Subclassing API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 124308,
     "status": "ok",
     "timestamp": 1669991501548,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "rpkF4BW7O9hJ",
    "outputId": "4317ad0a-0137-4d5b-c1c1-bfa80304f71d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 29s 28ms/step - loss: 0.5426 - accuracy: 0.7156\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 18s 23ms/step - loss: 0.3469 - accuracy: 0.8572\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 20s 25ms/step - loss: 0.1760 - accuracy: 0.9387\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 18s 23ms/step - loss: 0.1329 - accuracy: 0.9514\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 19s 24ms/step - loss: 0.1103 - accuracy: 0.9597\n"
     ]
    }
   ],
   "source": [
    "K = keras.backend\n",
    "embed_size = 128\n",
    "inputs = keras.layers.Input(shape=[None])\n",
    "\n",
    "# not_equal(inputs, 0)) return a (Boolean) mask tensor\n",
    "mask = keras.layers.Lambda(lambda inputs: K.not_equal(inputs, 0))(inputs)\n",
    "\n",
    "z = keras.layers.Embedding(vocab_size + num_oov_buckets, embed_size)(inputs)\n",
    "z = keras.layers.GRU(128, return_sequences=True)(z, mask=mask)\n",
    "z = keras.layers.GRU(128)(z, mask=mask)\n",
    "\n",
    "# here the masks are not propogated, as no sequence is returned.\n",
    "outputs = keras.layers.Dense(1, activation=\"sigmoid\")(z)\n",
    "model = keras.models.Model(inputs=[inputs], outputs=[outputs])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\", metrics=[\"accuracy\"])\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3PM7hoicPL7_"
   },
   "source": [
    "### Reusing Pretrained Embeddings\n",
    "\n",
    "- Generally, we may download any model components (or ***modules***), e.g., embeddings by `tensorflow_hub` (such as a sentence encoder).\n",
    "- By default a `hub.KerasLayer` is not trainable (otherwise set `trainable=True`).\n",
    "- Not all modules support TF2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-gXpPqBhO9jW"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "# The downloading directory is set to a location (that is not temporary)\n",
    "\n",
    "TFHUB_CACHE_DIR = os.path.join(os.curdir, \"my_tfhub_cache\")\n",
    "os.environ[\"TFHUB_CACHE_DIR\"] = TFHUB_CACHE_DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZWWbW0BhPQ83"
   },
   "outputs": [],
   "source": [
    "import tensorflow_hub as hub\n",
    "\n",
    "model = keras.Sequential([\n",
    "    hub.KerasLayer(\"https://tfhub.dev/google/tf2-preview/nnlm-en-dim50/1\",\n",
    "                   dtype=tf.string, input_shape=[], output_shape=[50]),\n",
    "    keras.layers.Dense(128, activation=\"relu\"),\n",
    "    keras.layers.Dense(1, activation=\"sigmoid\")\n",
    "])\n",
    "model.compile(loss=\"binary_crossentropy\", optimizer=\"adam\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 45,
     "status": "ok",
     "timestamp": 1669991508533,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "tu9GxxzRPQ_N",
    "outputId": "46b88e46-d2dd-4ebb-fce6-126233f39a80"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe.descriptor.txt\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/saved_model.pb\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/assets/tokens.txt\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.data-00000-of-00001\n",
      "./my_tfhub_cache/82c4aaf4250ffb09088bd48368ee7fd00e5464fe/variables/variables.index\n"
     ]
    }
   ],
   "source": [
    "for dirpath, dirnames, filenames in os.walk(TFHUB_CACHE_DIR):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirpath, filename))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 28137,
     "status": "ok",
     "timestamp": 1669991536631,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "gxJsxylYPRBW",
    "outputId": "7b889d43-8507-4ed6-d223-d8b46c8fdaca"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "782/782 [==============================] - 5s 6ms/step - loss: 0.5461 - accuracy: 0.7267\n",
      "Epoch 2/5\n",
      "782/782 [==============================] - 5s 6ms/step - loss: 0.5130 - accuracy: 0.7495\n",
      "Epoch 3/5\n",
      "782/782 [==============================] - 6s 7ms/step - loss: 0.5081 - accuracy: 0.7532\n",
      "Epoch 4/5\n",
      "782/782 [==============================] - 5s 6ms/step - loss: 0.5047 - accuracy: 0.7540\n",
      "Epoch 5/5\n",
      "782/782 [==============================] - 7s 9ms/step - loss: 0.5018 - accuracy: 0.7566\n"
     ]
    }
   ],
   "source": [
    "# E.g.,\n",
    "\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "datasets, info = tfds.load(\"imdb_reviews\", as_supervised=True, with_info=True)\n",
    "train_size = info.splits[\"train\"].num_examples\n",
    "batch_size = 32\n",
    "train_set = datasets[\"train\"].batch(batch_size).prefetch(1)\n",
    "history = model.fit(train_set, epochs=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CzlmdTU-PcCK"
   },
   "source": [
    "## An Encoder–Decoder Network for Neural Machine Translation\n",
    "\n",
    "We start with a simple neural machine translation [7], as shown below:\n",
    "\n",
    "<img src=\"img/translation.jpg\" width=\"400px\">\n",
    "<center>A simple NLP method.</center>\n",
    "\n",
    "- The encoder converts the varible-lenght input sequents to fixed-size ***context*** vector, i.e., the last encoder's hidden state containing the input's *contextual representation*.\n",
    "- During training, the ground-trought target translations are fed (with one step delay) to the decoder (i.e., the ***Teacher Forcing*** algorithm [12]).\n",
    "- At inference time, we use the predicted translations (as the true target translations are not available). This training-prediction discrepancy may cause ***Exposure Bias*** [13], i.e., when the training data distribution is different from the corresponding data distribution after traininig.\n",
    "- The very first word is a *start-of-sequence* (**SOS**) token, and the decoder ends the sentence with an *end-of-sequence* (**EOS**) token.\n",
    "- English sentences are reversed as verbs are more important.\n",
    "- The decoder outputs a score, which turn into a probability by a softmax layer.\n",
    "\n",
    "For implementation use the **TF Addons** project that has many *sequence-to-sequence* tools for production-ready encoder–decoders:\n",
    "- Use the TF Addons project that has many sequence-to-sequence tools for production-ready encoder–decoders. Here, we may use `TrainingSampler()` that provides the decoder with the pretended  previous output (based on the predicted, or target values, as stated above). However, It is also suggested to start training with the target of the previous time step and gradually shift to using the actual values. For this, you may use `ScheduledEmbeddingTrainingSampler` that  randomly selects between the target or the predicted output, with a probability that can be changed gradually.\n",
    "- As the sentences have varying lengths, we should group them into buckets of similar lengths, e.g., one bucket for 1- to 5-word sentences, another bucket for 6- to 10-word sentences, etc. Smaller sentences are simply padded to fill each bocket. Check `tf.data.experimental.bucket_by_sequence_length()`.\n",
    "- Mask out (to ignore) any output pass the EOS token.\n",
    "- As the output is too large, use `tf.nn.sampled_softmax_loss()` [37] that approximates the loss based on merely the logits outputs for the correct word and for some random incorrect words. However, at inference time (as targets are not available), use the regular softmax function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "vS3MYc9jz7si"
   },
   "outputs": [],
   "source": [
    "import tensorflow_addons as tfa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "HojrayKIPRDa"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(42)\n",
    "\n",
    "vocab_size = 100\n",
    "embed_size = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f9hAy4iqPRFo"
   },
   "outputs": [],
   "source": [
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)\n",
    "\n",
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fyHvyOjDz7si"
   },
   "outputs": [],
   "source": [
    "# set return_state=True to give the final hidden state to the decoder.\n",
    "encoder = keras.layers.LSTM(512, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]\n",
    "\n",
    "sampler = tfa.seq2seq.sampler.TrainingSampler()\n",
    "\n",
    "decoder_cell = keras.layers.LSTMCell(512)\n",
    "output_layer = keras.layers.Dense(vocab_size)\n",
    "decoder = tfa.seq2seq.basic_decoder.BasicDecoder(decoder_cell, sampler,\n",
    "                                                 output_layer=output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(\n",
    "    decoder_embeddings, initial_state=encoder_state,\n",
    "    sequence_length=sequence_lengths)\n",
    "Y_proba = tf.nn.softmax(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.models.Model(\n",
    "    inputs=[encoder_inputs, decoder_inputs, sequence_lengths],\n",
    "    outputs=[Y_proba])\n",
    "\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\", optimizer=\"adam\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 9676,
     "status": "ok",
     "timestamp": 1669991640619,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "FK3VFZ2HPirx",
    "outputId": "310a912a-e994-496a-c7c2-d1fe0e3f058d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "32/32 [==============================] - 5s 46ms/step - loss: 4.6052\n",
      "Epoch 2/2\n",
      "32/32 [==============================] - 2s 50ms/step - loss: 4.6037\n"
     ]
    }
   ],
   "source": [
    "X = np.random.randint(100, size=10*1000).reshape(1000, 10)\n",
    "Y = np.random.randint(100, size=15*1000).reshape(1000, 15)\n",
    "X_decoder = np.c_[np.zeros((1000, 1)), Y[:, :-1]]\n",
    "seq_lengths = np.full([1000], 15)\n",
    "\n",
    "history = model.fit([X, X_decoder, seq_lengths], Y, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tUIPKl8qPrxi"
   },
   "source": [
    "### Bidirectional Recurrent Layers\n",
    "Two sentences “the queen of hearts” and “the queen bee” shows that we need to bidirectional RNNs, in which we run two RNN layers that read in different directions the same inputs, and their outputs are combined. `keras.layers.Bidirectional` implements this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 13,
     "status": "ok",
     "timestamp": 1669991640623,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "y0MEkDm0Pipy",
    "outputId": "ad1f431b-8d67-4f16-cd50-eff6d74c7be2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " gru_4 (GRU)                 (None, None, 10)          660       \n",
      "                                                                 \n",
      " bidirectional (Bidirectiona  (None, None, 20)         1320      \n",
      " l)                                                              \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 1,980\n",
      "Trainable params: 1,980\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.GRU(10, return_sequences=True, input_shape=[None, 10]),\n",
    "    keras.layers.Bidirectional(keras.layers.GRU(10, return_sequences=True))\n",
    "])\n",
    "\n",
    "model.summary()\n",
    "\n",
    "# note that we have twice outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oEJUY6JuPvvz"
   },
   "source": [
    "### Beam Search\n",
    "\n",
    "- The above codes can *greedily* search for the correct predictions, meaning that when it predicts one word, then another, it is unable to return back to change already predicted words (In other words, it does not see the whole sentences and focuses on already predicted values).\n",
    "- *Beam search* addresses this issue with its parameter $k$, i.e., *beam width* that refers to keeping the $k$ most likely prediction of each word (figure below).\n",
    "\n",
    "<img src=\"img/beam_search.jpg\" width=\"400px\">\n",
    "<center>Beam search [11].</center>\n",
    "\n",
    "- Each time $k$ models are generated for each prediction that finds a conditional probability (i.e., on the condition of a previous translations). The probability comes from the multiplications of all the probability, and this way, only the top predictions are kept in each step (of predicting the new word in the following).\n",
    "- TF Addons can implement it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "F-172UwtQ_-o"
   },
   "outputs": [],
   "source": [
    "# for demonstration:\n",
    "\n",
    "beam_width = 10\n",
    "\n",
    "# BeamSearchDecoder() wraps all the decoder clones.\n",
    "decoder = tfa.seq2seq.beam_search_decoder.BeamSearchDecoder(\n",
    "    cell=decoder_cell, beam_width=beam_width, output_layer=output_layer)\n",
    "\n",
    "# tile_batch() creates one copy of the encoder’s final state for each decoder clone\n",
    "decoder_initial_state = tfa.seq2seq.beam_search_decoder.tile_batch(\n",
    "    encoder_state, multiplier=beam_width)\n",
    "\n",
    "# pass all these states to the decoder clones\n",
    "outputs, _, _ = decoder(\n",
    "    embedding_decoder, start_tokens=start_tokens, end_token=end_token,\n",
    "    initial_state=decoder_initial_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tv__F9A8P8sU"
   },
   "source": [
    "## Attention mechanisms\n",
    "\n",
    "- Attention mechanisms (firstly introduced by Graves et al. [25]), which can be with or without RNNs, can address the issue RNN short memory by focusing on the right word (instead of the regular RNN's encoders, that emphasises more on the input's hidden state).\n",
    "- They are also helps to improve *explanaibility* by e.g., **visual attention** as used in ***image captions generation*** [22] (shown below).\n",
    "<img src=\"img/attention_vis.jpg\" width=\"400px\">\n",
    "<center>E.g., for the visual attention generating the caption “A woman is throwing a frisbee in a park”: an input image (left) and the model’s focus when it was close to the it was about to output the word “frisbee”.</center>\n",
    "- They can be categorized to **inter-attention** and **intra-attention** mechanisms depending whether they consider the information of both inputs and outputs or just one of them.\n",
    "\n",
    "\n",
    "### Attention mechanisms with RNNs or CNNs\n",
    "As show in figure below:\n",
    "<img src=\"img/attention.jpg\" width=\"400px\">\n",
    "<center>An attention mechanism with RNNs.</center>\n",
    "\n",
    "- In addition to transferring the final encoder’s hidden states (not shown here), the outputs are also sent to the decoder. A weighted sum of these outputs are computed using the weights $\\alpha_{\\left(t,i\\right)}$ corresponding to the i$^\\text{th}$ encoder output at the t$^\\text{th}$ decoder time step.\n",
    "- While the rest of the model works like other RNNs, weights are computed by another small NN, i.e., an ***alignment model*** (or an ***attention layer***) that is trained jointly with the encoder-decoder model.\n",
    "- This is the **Bahdanau** attention mechanism [9]:\n",
    "    - This alignment model starts with a time-distributed layer with a single neuron, receiving the concatenation of the previous decoder’s hidden layer and encoder’s outputs. Then, this layer outputs a ***score*** (or *energy*), i.e., $e$, for each, which measures how well each e is aligned with the decoder’s previous state. With the softmax, the weights sum to 1.\n",
    "    - A simplification is to use $\\mathbf{h}_{\\left(t\\right)}$ instead of $\\mathbf{h}_{\\left(t-1\\right)}$.\n",
    "    - This attention is also called ***concat*** [10] or ***additive*** [21] attention.\n",
    "    - It is considered as the a ***global attention***  model, since it considers all inputs, \n",
    "    - It is ***soft attention*** as considers all the input hidden states by summation (in contrary to *hard attention* that strictly works based one hidden state).\n",
    "- An alternative is the **Luong** attension [10]:\n",
    "    - it measures the similarity by the *dot product* (thus, also callled ***multiplicative*** attention) to benefit from the hardware acceleration.\n",
    "    - it has two implemenation, namely ***dot product*** and ***general dot product*** (for which, the outputs go through a linear transformation prior to the dot product).\n",
    "    - The original implementation is a a ***local attention***  model, as it attends around only ***alignment position*** (instead of the whole sequence), to accelate model. The aligment position for $t^{th}$ decoder hidden state $p_t$ can be detemined by\n",
    "        - ***monotonic alignment*** that assumes input and output sequences are relatively monotonically aligned, thus $p_t = t$;\n",
    "        - ***predictive alignment*** that uses Gaussian distribution to favor alignment weights around the alignment position, which is then defined by a learable method: $L\\cdot \\text{sigmoid}\\left ( v_{p}^{\\text{T}} \\text{tanh}\\left ( W_p h_t \\right )\\right )$, where $L$ is the sequence length, and $W_p$ and $v_{p}$ are learnable parameters. \n",
    "- These attenstion mechanisms are compared bolow:\n",
    "\n",
    "\\begin{equation}\n",
    "{\\widetilde{\\mathbf{h}}}_{\\left(t\\right)}=\\sum_{i}{\\alpha_{\\left(t,i\\right)}\\mathbf{y}_{\\left(i\\right)}}\n",
    "\\quad \\text{with} \\quad\n",
    "\\alpha_{\\left(t,i\\right)}=\\frac{exp\\left(e_{\\left(t,i\\right)}\\right)}{\\sum_{i} e x p\\left(e_{\\left(t,i^\\prime\\right)}\\right)}\n",
    "\\quad \\text{and} \\quad\n",
    "e_{\\left(t,i\\right)}=\\left\\{\\begin{matrix}{\\mathbf{h}_{\\left(t\\right)}}^T\\mathbf{y}_{\\left(i\\right)}&\\text{dot}\\\\{\\mathbf{h}_{\\left(t\\right)}}^T\\mathbf{W}\\mathbf{y}_{\\left(i\\right)}&\\text{general}\\\\\\mathbf{v}^T\\mathbf{tanh}\\left(\\mathbf{W}\\left[\\mathbf{h}_{\\left(t\\right)};\\mathbf{y}_{\\left(i\\right)}\\right]\\right)&\\text{concat}\\\\\\end{matrix}\\right.\n",
    "\\end{equation}\n",
    "\n",
    "- Here, $\\mathbf{v}$ is a rescaling parameter vector.\n",
    "- Generally, the dot products variants may work better and can be implemented by TF Addons."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "76cxojO_RDG-"
   },
   "outputs": [],
   "source": [
    "# for demonstration\n",
    "\n",
    "# Simply wrap the decoder cell in an AttentionWrapper, after providing an attention mechanism.\n",
    "\n",
    "attention_mechanism = tfa.seq2seq.attention_wrapper.LuongAttention(\n",
    "    units, encoder_state, memory_sequence_length=encoder_sequence_length)\n",
    "\n",
    "attention_decoder_cell = tfa.seq2seq.attention_wrapper.AttentionWrapper(\n",
    "    decoder_cell, attention_mechanism, attention_layer_size=n_units)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lHG6TnIWz7sk"
   },
   "source": [
    "- By the concept of **query**, **key**, **value**, the attention mechanism can be seen as a function outputing with weigthed sum of the *values*, where each weight is computed by a ***compatibility function*** of the query with the assosiated key [21].\n",
    "- For illustration, in the above *encoder-docoder attention*, the queries are from each decoder's layer, which attents to all the key/values (which are usually the same) from the encoder [21].\n",
    "- Acoordingly, the `Attention` layer implements the dot-product attention by this concept (that might be used as the base class of other attension layers):\n",
    "    - The query, key, and value tensors are inputs to this layer with shapes `[batch_size, Tq or Tv or Tv, dimension]` (considering that values and keys are usually the same);\n",
    "    - it calculate ***scores*** with shape `[batch_size, Tq, Tv]` by `scores = tf.matmul(query, key, transpose_b=True)`.\n",
    "    - it then calculates a ***distribution*** with the same shape of scores by `distribution = tf.nn.softmax(scores)`.\n",
    "    - it finally creates a linear combination of values of the shape `[batch_size, Tq, dim]` by `return tf.matmul(distribution, value)`.\n",
    "- E.g., a CNN+Attention network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dZ4L0gHUz7sk"
   },
   "outputs": [],
   "source": [
    "# For demonstration (from TensorFlow documentation)\n",
    "\n",
    "# Variable-length int sequences.\n",
    "query_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
    "value_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
    "\n",
    "# Embedding lookup.\n",
    "token_embedding = tf.keras.layers.Embedding(input_dim=1000, output_dim=64)\n",
    "\n",
    "# Query embeddings of shape [batch_size, Tq, dimension].\n",
    "query_embeddings = token_embedding(query_input)\n",
    "\n",
    "# Value embeddings of shape [batch_size, Tv, dimension].\n",
    "value_embeddings = token_embedding(value_input)\n",
    "\n",
    "# CNN layer.\n",
    "cnn_layer = tf.keras.layers.Conv1D(\n",
    "    filters=100,\n",
    "    kernel_size=4,\n",
    "    # Use 'same' padding so outputs have the same shape as inputs.\n",
    "    padding='same')\n",
    "\n",
    "# Query encoding of shape [batch_size, Tq, filters].\n",
    "query_seq_encoding = cnn_layer(query_embeddings)\n",
    "\n",
    "# Value encoding of shape [batch_size, Tv, filters].\n",
    "value_seq_encoding = cnn_layer(value_embeddings)\n",
    "\n",
    "# Query-value attention of shape [batch_size, Tq, filters].\n",
    "query_value_attention_seq = tf.keras.layers.Attention()(\n",
    "    [query_seq_encoding, value_seq_encoding])\n",
    "\n",
    "# Reduce over the sequence axis to produce encodings of shape\n",
    "# [batch_size, filters].\n",
    "query_encoding = tf.keras.layers.GlobalAveragePooling1D()(\n",
    "    query_seq_encoding)\n",
    "query_value_attention = tf.keras.layers.GlobalAveragePooling1D()(\n",
    "    query_value_attention_seq)\n",
    "\n",
    "# Concatenate query and document encodings to produce a DNN input layer.\n",
    "input_layer = tf.keras.layers.Concatenate()(\n",
    "    [query_encoding, query_value_attention])\n",
    "\n",
    "# Add DNN layers, and create Model ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "491LGaE-z7sk"
   },
   "source": [
    "### Transformers\n",
    "- Google researchers suggested that “Attention Is All You Need.” [21], as their proposed method (figure below), i.e., transformers, can learn sequentional data mostly by attenstion mechanisms and related components without RNNs, which permits easier parallelization without the need for extremely deep CNN-based implementation for large sequences.\n",
    "- The vanialla transformer is show below, which, among its components, it contains N sequential encoders and decoders.\n",
    "\n",
    "<img src=\"img/transformer.jpg\" width=\"300px\">\n",
    "<center>A transformer [21].</center>\n",
    "\n",
    "- Since transformers do not have recurrence and convolusion operators, they use **positional encoding (PE)** [29], e.g., by the concept of Fourier series [28], to add positional information into fed data:\n",
    "    - The vannilla transformer uses the following PE method of sequences:\n",
    "    \\begin{equation}\n",
    "    {PE}_{\\left(pos, 2i\\right)}= \\text{sin}(pos/10000^{2i/d_{\\text{model}}}) \\\\\n",
    "    {PE}_{\\left(pos, 2i+1\\right)}= \\text{cos}(pos/10000^{2i/d_{\\text{model}}})\n",
    "    \\end{equation}\n",
    "    - Here,\n",
    "        - $0\\leq pos<L$ is the position of fed object in the sequence of length $L$,\n",
    "        - Position function for mapping a position  in the input sequence to index  of the positional matrix,\n",
    "        - the arbitrary number 10000 controls the frequency and wavelenght of of the trigonometric functions,\n",
    "        - $0\\leq i<d/2$ is an indice of used to for mapping the functions outputs to the components of the output embeddings with the dimension of $d$ (as illustrated below).\n",
    "    <img src=\"img/PE3.png\" width=\"500px\">\n",
    "    <center>E.g., of a positional encoding matrix for an input sentence [24].</center>\n",
    "    - Mathematically speaking,\n",
    "        - each dimension then corresponds to a sinusoid with different wavelengths, and\n",
    "        - it may allow the optimizer to easily learn to attend when considering relative positions, as it can be proved that ${PE}_{\\left(pos\\right)}$, is linearly corrolated to ${PE}_{\\left(pos+k\\right)}$, for any arbitraty offset $k$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1671396893281,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "KozwQEYMz7sl"
   },
   "outputs": [],
   "source": [
    "def positional_encoding(length, depth):\n",
    "    depth = depth/2\n",
    "    \n",
    "    positions = np.arange(length)[:, np.newaxis]     # (seq, 1)\n",
    "    depths = np.arange(depth)[np.newaxis, :]/depth   # (1, depth)\n",
    "    \n",
    "    angle_rates = 1 / (10000**depths)         # (1, depth)\n",
    "    angle_rads = positions * angle_rates      # (pos, depth)\n",
    "    \n",
    "    pos_encoding = np.concatenate(\n",
    "        [np.sin(angle_rads), np.cos(angle_rads)],\n",
    "        axis=-1)\n",
    "    \n",
    "    return tf.cast(pos_encoding, dtype=tf.float32)\n",
    "\n",
    "class PositionalEmbedding(tf.keras.layers.Layer):\n",
    "    def __init__(self, vocab_size, d_model):\n",
    "        super().__init__()\n",
    "        self.d_model = d_model\n",
    "        self.embedding = tf.keras.layers.Embedding(vocab_size, d_model, mask_zero=True) \n",
    "        self.pos_encoding = positional_encoding(length=2048, depth=d_model)\n",
    "    \n",
    "    def compute_mask(self, *args, **kwargs):\n",
    "        return self.embedding.compute_mask(*args, **kwargs)\n",
    "    \n",
    "    def call(self, x):\n",
    "        length = tf.shape(x)[1]\n",
    "        x = self.embedding(x)\n",
    "        # This factor sets the relative scale of the embedding and positonal_encoding.\n",
    "        x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))\n",
    "        x = x + self.pos_encoding[tf.newaxis, :length, :]\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 290
    },
    "executionInfo": {
     "elapsed": 2909,
     "status": "ok",
     "timestamp": 1671396900132,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "FVKH-JPcz7sl",
    "outputId": "13dc7bcf-43ea-4c30-86eb-25209cf50e90"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0, 1.0)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEACAYAAAC08h1NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdd3xUVfr48c8zmdRJ7wGSQOgdpCogKNi77lqwu4prd91113Vty7pfV3ct6xaVn12x7K4idl1XUCnSCR2khEgJKZDek/P7485MJkPKhCQkMM/79coL5pxzb87c3Hnm3HPPPUeMMSillPIftq6ugFJKqaNLA79SSvkZDfxKKeVnNPArpZSf0cCvlFJ+RgO/Ukr5GQ38SinlZ3wK/CJyu4isFJEqEXm1lbK/EJEcESkWkZdFJNgjr7eILBCRchHZIiIz2ll/pZRSbeRri38f8CjwckuFROQM4D5gOpAOZAC/9yjyNrAGiAN+B/xHRBLaWGellFLtIG15cldEHgV6GWOuayb/LSDLGHO/8/V0YK4xJllEBgDrgXhjTIkz/ztn/vPtextKKaV8Ze/g/Q0F5nu8zgSSRCTOmbfTFfQ98oc2tSMRmQXMAnA4HGMGDRrUwVVV6nBbt24FYODAgYflHSyrZm9hBakxYUSHBbZpW6WOtlWrVuUbY5rsUenowB8OFHm8dv0/ook8V37PpnZkjJkDzAEYO3asWblyZcfWVKkmTJs2DYCFCxcelveXL7by9wXbuWPGAO6a0b9N2yp1tInI7ubyOnpUTykQ6fHa9f+SJvJc+SUodQwItlsfl8raui6uiVLt09GBfyMw0uP1SOCAMabAmZchIhFe+Rs7uA5KdQp7gPVxqa/XGW3Vsc3X4Zx2EQkBAoAAEQkRkaa6iV4HfiYiQ0QkGngAeBXAGLMNWAs87Nz+ImAE8F4HvA+llFI+8rWP/wHgYY/XVwG/F5GXgU3AEGNMtjHmcxF5AlgAhGIFdc/tLsf6IjgEZAM/Mcbkte8tKKX8XU1NDXv27KGysrKrq3LUhYSE0KtXLwIDDx9w0ByfAr8x5hHgkWayw73KPgU81cx+soBpvlZOKaV8sWfPHiIiIujduzci0tXVOWqMMRQUFLBnzx769Onj83Y6ZYNSPjJo3353VVlZSVxcnF8FfQARIS4urs1XOhr4lfJRXZ0G/u7M34K+y5G8bw38Svmouq4egHpdp1od4zTwK+UjV+Cvrq3v4pqo7mbevHmMGjWq0Y/NZuOzzz7r6qo1qaOf3FXquOUK+NXa5aO8XHTRRVx00UXu13PmzGHu3LmcccYZXVir5mmLXykfuQO/tvhVC7Zt28bs2bN54403EBHuvfdehg0bxvDhw3n33XcBazROU+kLFy5k6tSpXHDBBWRkZHDfffcxd+5cxo8fz/Dhw9mxY0eH1FFb/Er5qMbV1VOngb87u/vuu1m7dm2H7nPUqFE888wzrZarqalh5syZPPnkk6SlpfHee++xdu1aMjMzyc/PZ9y4cZx88sksWbKkyXSAzMxMNm/eTGxsLBkZGdx4440sX76cv/71r/ztb3/zqR6t0Ra/Uj5ytfRrtMWvmvHggw8ydOhQLrvsMgAWLVrEFVdcQUBAAElJSUydOpUVK1Y0mw4wbtw4UlJSCA4Opm/fvpx++ukADB8+nKysrA6pp7b4lfJRjbNvX1v83VtHtIiPxMKFC3nvvfdYvXp1u/YTHOxetBCbzeZ+bbPZqK2tbde+3fvtkL0o5QeqXC1+DfzKy6FDh7j++ut5/fXXiYhomIdyypQpvPvuu9TV1ZGXl8e3337L+PHjm00/WrTFr5SPXC39Ku3qUV6ef/55cnNzueWWWxql//a3v2XEiBGMHDkSEeGJJ54gOTmZiy66iKVLlx6WvmXLlqNS3zYtvdhVdCEWdbS0tJjKFXO+Z+nOAkalRvPBbZPatK3qXJs3b2bw4MFdXY0u09T7F5FVxpixTZXXrh6lfORq8WtXjzrWaeBXykc6jl8dLzTwK+UjHcffvR0L3dad4Ujet68rcMWKyDwRKROR3SIys5lyn4lIqcdPtYis98jPEpEKj/wv21xjpbqIjuPvvkJCQigoKPC74O+ajz8kJKRN2/k6qucfQDWQBIwCPhGRTGNMo/VyjTFneb4WkYXA1177Os8Y81WbaqlUN1BZYy2yri3+7qdXr17s2bOHvDz/W9DPtQJXW7Qa+EXEAVwCDDPGlAKLRORD4Grgvha26w1MAa5rU42U6qZKq6yHZ3Q4Z/cTGBjYphWo/J0vXT0DgFrnYukumcDQVra7BvjOudyip7kikiciX4rIyOY2FpFZIrJSRFb647e46l7q64078OvNXXWs8yXwhwPFXmlFQEQTZT1dg7Wwuqcrgd5AOtaC7F+ISHRTGxtj5hhjxhpjxiYkJPhQTaU6z4qsg9QbGN4ziqraen48WN7VVVLqiPkS+EuBSK+0SKCkuQ1EZDKQDPzHM90Ys9gYU2GMKTfGPAYUYnUHKdWt7cwvA+DO6f0B64tAqWOVL4F/G2AXkf4eaSOBjc2UB7gWeN95T6AlBvDPhTLVMSW/pAqACRmxAOwvatvi1kp1J60GfmNMGfA+MFtEHCIyCbgAeKOp8iISClyKVzePiKSJyCQRCRKREBG5F4gHFrfzPSjV6fJKq4gMsRMZEogjKICDZdVdXSWljpivD3DdCoQCucDbwC3GmI0iMkVEvFv1F2J14SzwSo8AngMOAXuBM4GzjDEFR1p5pY6WgtJq4iOs6XFjHEEa+NUxzadx/MaYg1gB3Tv9O6ybv55pb2N9OXiX3QiMOLJqKtW1CsqqiHMEARCrgV8d43TKBqV8UFBaTZzD2eIPC+JQuQZ+dezSwK9UK3KKKvkht5TkKOux+JSoEDbvLya/tKqLa6bUkdHAr1Qr1v5YCMD0wYkAXDS6JzV1hsXb87uyWkodMQ38SrViZ741fmFUqvWs4ai0aGwCO3JbG62sVPekgV+pVuzKKyMhIpiIkEAAgu0BpMWGsSOvrItrptSR0cCvVAvq6g0Lt+XRJ97RKL1fYjhfbT5AXb1/TQOsjg8a+JVqwcZ9ReSVVHFy//hG6RMz4qiqrWf5Lp26QR17NPAr1QLXjd0LR/dslH7JCdb855l7Co96nZRqLw38SjXDGMPnG3JIiAimZ3Roo7wYRxDpcWH8a8WPfrfqkzr2aeBXqhkb9hazZEcBV05IQ+TwuQSnD0piZ34ZG/Z6z1quVPemgV+pZizeYY3Tv2xcapP5d07vh90mfLxu39GsllLtpoFfqSaUVtXy3MIdZMQ7SIkKbbJMdFgQk/rF89bybPd6vEodCzTwK9WELzfmUFRRw6/PHNhiuSsnpFFSWcvnG3KOUs2Uaj8N/Ep5qTeGP36ymThHEKcOSmqx7LSBiSREBPPwhxup15u86hihgV8pL/sKKykoq+aPFw0jyN7yRyTIbmP2+UMpqqght1gnbVPHBp8Cv4jEisg8ESkTkd0iMrOZco+ISI2IlHr8ZHjkjxKRVSJS7vx3VEe9EaU6QnVtPXsLKxjWM5LThyT7tM2MIUkMSAon+2A5NXX1nVxDpdrP1xb/P4BqIAm4EnhORIY2U/ZdY0y4x89OABEJAuYDbwIxwGvAfGe6Ut3C9txSMIZnLhuFzebbctCBATaeunQU9cawNadEx/Wrbq/VwC8iDuAS4EFjTKkxZhHwIXB1G3/XNKwVv54xxlQZY57FWmj91DbuR6lO8Y8F2ymurKFnTBj9EiPatO2wnlGkxYZRWlXLn7/Y2kk1VKpj+NLiHwDUGmO2eaRlAs21+M8TkYMislFEbvFIHwqsM42bQ+ua24+IzBKRlSKyMi8vz4dqKnXkPszcx5+/2EpYUAA9okOOaB8pUaFEhQbyz4U7mL92bwfXUKmO40vgDwe8H00swlo83du/gMFAAnAT8JCIXOGxnyIf94MxZo4xZqwxZmxCQoIP1VTqyCzcmstd76yhT7yDwSmR2Jp4StcXItasncmRIdz1zlpdqEV1W74E/lIg0istEijxLmiM2WSM2WeMqTPGLAH+CvykrftR6mj5dlse172ygsAAG3OuHkNgQPsGugUG2Hjzxgk4ggK48sVlfLtNr1ZV9+PLWb4NsItIf4+0kcBGH7Y1WP34OMuPkMaTnozwcT9KdbivNh3g+ldXEBliZ96tJ9E/qW39+s3plxjOuzefSFRoINe8vJwP1mi3j+peWg38xpgy4H1gtog4RGQScAHwhndZEblARGLEMh64E2skD8BCoA64U0SCReR2Z/rXHfA+lPJZdW09D83fwI2vryQhPJiP7pjM0B5RHfo7hvWMYt6tJxEfHszd767lnnfX6lBP1W34el17KxAK5AJvA7cYYzaKyBQR8Vx49HJgO1b3zevA48aY1wCMMdXAhcA1QCFwA3ChM12po+LHg+VMf2ohry/dzaDkCD68YxLpcY7WNzwCGQnh/PcXJ3NCWjTvr9nLtD8vpKBUH/JSXc/uSyFjzEGsoO2d/h3WTVvX6yu8y3iVXwOMaWMdlWo3YwwvL87iDx9vAuCqiWn84YJhTU633JFiHEH86+YTeXD+Rt5ens2YR7/i3jMGctsp/Tr19yrVEp2yQR33Fv2QT5/ffuoO+n+fOZpHLxze6UHfxR5g47GLh/PiNWMB+PMXW+l93ycs2aGjflTX8KnFr9SxxhjD5v0lXPzcYiprrL71Kf3jee6qMYQHd81pP2NIEpkPn86Nr61gRdYhZv6/ZYQFBfDOrIkM7xl11L6IlNLAr44rpVW1vLYk67CnZ1+9fhzTBiZ2Ua0aRIUG8u+fn8SS7flc+dIyyqvrOP/viwF48NwhXDE+lbAg/ViqzqVnmDrm1Ncbvvkhj2+35VFbZ9hzqJxtB0rZW1hxWNl7zxjILVP7+jzvztFyUr94tv/xbF78biePfbYFgD98vMndHdUn3kGP6BD6JYQTGGBjUv94pg1I0KsC1SE08KtjSnZBOWf+9VvKq1te8eqcESk8dvFwIkMCj1LN2i7AJtw8tS9XTUzn3v9k8un6hsVcduWXsSu/jMXbCwB4cdEuQgMD+PpXU5tdEUwpX2ngV93eh5n7WJl1kE/X55DfynDIiBA782+bREZCeIvluhNHsJ1/XjmG7bklXD5nWbPvsaKmjhMf+5qIEDszBicxuV88l4zpdZRrq44HGvhVt2GMobKmnrLqWj7bkMNby6y1bHfll/m0/eOXDOfSsanHbHdIv8QIlt8/nbnLs3nwgw3NliuprGXemr3MW7OXxz/fQkxYEDMnpHHuiBQcwXZCAgOOYq3VsUgDv+pSdfWGvYcqqK6rY+HWPB79ZPMR7eebe6d12oNYR5PNJlw9MZ2T+sYx/clvWi2fW1JFbkkVD3+4kYc/tGY/ufeMgZw5LBm7TUiNCet29zdU19PAr466fYUVFJbX8Nv311FTZ9i033vyV9+d1DeO564aQ1Ro9+3LPxJ9E8JZ+9BpzHpjFct3HWzTtn/+Yqt7VFO/xHCiQgP5/flDSYgIJinyyKacVscXDfyqQ1XX1pNVUEZyVAg788oIDw6gsqaeipo6bAKfrs/h0/X72V9U2a7fM7RHJLMvGMqY9NgOqnn3Ex1mPfW7Iusgsz/axPq93rOat257rjWjyrl/W0TP6FDOH9WDGYMTERGC7TbCg+3kFFUypEck2QfLGZgUgb2dM5Sq7k8Dv2qXqto6KqrrWLqjgFcWZ9ErJpT31+wlI97BTh/75n0xOi2apy8dRXpc2DHbh3+kxvWO5aM7JgPWUNac4kpue2s1a7IL27SfvYUVPLdwB88t3HFYXs/oUPYWVnDR6J7szC/jlqkZTOgThyPY3uqC8+rYo4Ff+aS8upawIDtvfr+bUanRvLd6D+HBdkoqa3l1SRaJEcHkllSxPMsq3xFB/6djejHr5Ax6RIfi6KKnbbsbm03oER3KvFsnUVFdx55D5cxdls17q/ZQUlV7xPt1PQMxzzmF9M/fXA3AzVMzMAZsIpw5LJltOSVcOi7VfT6oY5P+5RQA2w6UEB0aSEhQAE//dxu3ndKPJTsKOFhaRaDdxu/mbeCtGyfwwAcbGJISeVi/fG5J+2edvHxcKjMGJzEwOYLU2LB27+94FxoUQP+kCB45fyiPnD+UXfllbDtQwuLt+by+dHeH/I4Xvtnp/v//Nh/gh9xSekSHctVLy3jq0pEcLKumV0wYo9OimfPtTu45bQDFlTWUVdW2ed1idfRo4D9OVdXWsedQBX0TwjlYVs3KrIOcNiSJHXmlLN15kKsnpvP60ix25pXxyPlDOf3pbwkPtnPbKf14ZXEWKVEh/N+n1hOlkSHWaTLzxWUA7boZ6zIxI5b+iRFcOjaVxMhgyqvr6BN/7I/K6Up94h30iXdwxtBkrj2pNxEhdvYXVvLWsmwKK6r5YuOBdu3/B+f9gqtess6Dv3yxlX3OezW/PG0ALy3aRWpMKM9/s5Oc4kp2PXY298/bwKjUKC4dm8ori7M4ZVAi6bFhfLkphxMz4okKC+SHAyX0jne0e/Uz5TufAr+IxAIvAacD+cBvjTFvNVHuXuBaIN1Z7p/GmD975GcBSVgLsgAsMcac3p434O9WZx+isqaOk/rGsyWnmOW7rKD+8qIsHv98C1//ciovLtrFW8uyee+Wk7j5jZXkl1YzJCWSh+Zbw/8yEqyAW1pVy+OfW8H+r1/94P4dxZVH3oXg7Y5T+yHAndP7603ETtTX+QBbYkQII1Ojqa2r55mvfiA0KOCweYyO1D6PG/RvfG9dYTzy0SZ32vPf7OTt5dm8vRz6xIcz++NNvPDtDp6+dBQ/f3M1N07uw0Un9OScZxfxwDmD+dnkPryyOIuTByTQLzGcr7ccICIkkHG9j98b+F1FjDGtFxJ5G2sK558Bo4BPgJOMMRu9yv0a+ApYB/QFvgR+Y4x5x5mfBdxojPmqLZUcO3asWblyZVs26fay8suoqq1nYLJ1OfzNtjxiw4IY3isKYwyvL93N+SN7EOMIYkdeKev3FHHh6J6UV9dy0+sr+c2ZgxjRK5re930CwA9/PIupTyxgX1Elf7tiNHe8vQaAMekxrNp9qMPrHx8e3OpTtADje8dyzogUJmTEEucIJiEiuMPr0pGmTZsGwMKFC4/qtkeLMYaCsmr2HqpgRdZBPli7lw17W7+C6xUTyp5Dh8+F1F4TM2L5fqc1XPWvl4/irnfW0j8xnM/vPpm+938KQNafzmFN9iGe/uoHnr/qBMKC7HywZi8npMWQFhdGXkkVX27KYeb4NESE1dmHKKuqZUr/BAA27SsmIsTud92HIrLKGDO2ybzWAr+IOIBDwDBjzDZn2hvAXmPMfa1s+6zzd9zhfJ2FnwT+zzfk0CsmlGE9o9yviyqquWxcGoA7YGf96ZzDXq/fU8R5f1/EaUOSeP6qMZz/90Vs3FfMR7dP5q531rAzv4y02DDiw4NY7RzZMTglks0d0AXj6cSMOJbuLGiUNqFPLMt2HeSUgQks2Nr0QuIn9Y3jhLQYrpyYRkRIYJdNg3wkjvfA762ksoaK6jpe+HYn23NL+cZrcfiIEOsG/jnDU/hk/X4GJUewJaekUZnOaFyM7xPrfn5hdFo0VTX1bNpfTHx4MO/ePJHpT35DrCOI5fdP58531vDp+hw+un0yw3tFNfnZCgwQfvjj2QC8v3oPMY4gTnHO1rpuTyH5pVWcOiipQ99DV2sp8PtyrT0AqHUFfadMYGgrv1SAKRy+mPpcEckTkS9FZGQL288SkZUisjIvr+kA0xlq6uqpr2/8Zfj9zgIqaxomBdtfVMHSHQ0BsbiyhpvfWMmOvIZVKH/+5irO/duiRq9/8956vL9o6+pNo7VY80qq+GT9fvfvGfX7L9m4zwrot8xd5R4tc7Cs2h30gXYH/Sn9493/dwXqE/vGHVauV4zVahrgvFK5/+xBpESFcMOkPjx/1Qn85+cn8tZNE/nVGQNJiQo9poK+P4oICSQxMoQHzx3CazeM5+2bJvLiNWO5YnwqGfEO7preH8B9ZdpUq3lyP+vc6RXTMHnc8J7tW8PY86G1NdmFbD1gfdnkl1bx8zdWAdZn4IQ//Jdd+eUAfL5xPwfLGlZyraiuc39ua+oaPnf3/CuT619Z4X59/t8Xc8OrDQ3LLTnF/PyNVZR6jJJavD2f3OKGrq2yqlqWeTWK6usN1bXHxrrKvgT+cMA7qhQBrd2yf8S5/1c80q4EemPdA1gAfCEi0U1tbIyZY4wZa4wZm5CQ4EM1W7Ylp5iSyppGabsLyiiqaJzW/3ef8fM3V7lfb9hbxOVzvufJLxv6RX/6/FKu+H/fU+Y8MZbvPMgXGw/wJ+f0umUeJ0xNXT3bcxtaSKt2H+L3HzV8F36xMYeL/7nE/fq376/j+W+scdYBNlujIXqel9ql7Ri6lx5nfXjvP3uQO+2qiekAJHp0xUzqZwX+YT0j+ctPRzJjcBJ3Te/P5H7xzJqSwfY/nsWsk/uy5L5Teei8IZw5LIWx2h97TDuxbxwzhiTx2MUj+N8vp3LjlAx2PXY2V05IY1K/OH5//lCmDkjgb1eMdp9Hk52NBmMazq3rJ/V27/Ohc4cAEB8edMT1qvNojLluMoN1/8n1uf7Hgh1c/8pyd95F/1zMgi257tePfLiRnR6Ns20HSqj1aHS5YsGfPtvC5xtz3I274soarnxxGZfN+d5d9vlvdnDZnO9Zt6eh8XXnO2sY+OBnjepdWF7NjwfLG6UVldew1euq6WjzpTlWCkR6pUViLajeJBG5HWtR9SnGGHdHsDFmsUexx0TkWqyrgo98rvERqKyp48xnvmNK/3je+NkEwArIU/+8kMEpkXx21xSgIWB/ualh9MMmZ2t7R17DuHRXAM4+WE5YUADbnSeTAL/8Vyb/3dQwve7f/vcDz3693f361++tY6fHvv729fZGrfWvNjecqJk/tu0BHWh4EAfg/JE9+DBzHz2iQrhifBpP/ncbZw9Ppr4edheUExUa6O4iOrl/AqcMTOCK8Vb3zLsrshmVGsOjFw5jVGo0w3pG8RPnTJBv3jih0e/0tweq/IXr7yoixIUHM/fGiQC8dsN4wGrhbztQwqjUaM4dkcL1k3pzoLiKD9bsZcYQq9tkTHqM+77O4JRIkiND+PeqPdx+Sj/+u+kAWw+UcOnYXvxr5R4ARvaKInNP255Q9mwQeW67Jaek0dxPry7JYtuBhrB1+tPfcu8ZA92vR/7+S64Yn4rN+b437StmQFK4uxXvOVmgaz9bc0oY0ctqu368zrpSL6qocU8hcsE/FrO7oJyd/3e2e86k++et55P1+8l8+HR3uYLSKm54dQW/OXMQJ/VruPruLL60+LcBdhHp75E2ksO7cAAQkRuA+4Dpxpg9rezbYMXLTuXqk1y0vWGNU1dQdQXdrPwyFnr0WdfXGx7/fAtzl2cDVvfH9zsLuOyFpe4yG/YWMfXPC90t/fLqOt5bvafRKBjvPnLPoO/5+1vTP7FhmuHrTurt/v+9ZwxkcIr1vfzEJSOYOcG6h/CTMb24YFQPAOLCg+mfZG1fWVPPuSNTABiZGs2cq8cw/7ZJhAYF8Mr14zl9aDIn9o3jmctHE2ATrpqY7r5PoZSn0WkxXDYujcAAG3+feQJj0mM5e3gKc64ZS2RIIO/dchLPXjGaAUlW58BPxvSiwtn10j8pnKgwK+idNiSZ80Za5+pPx6by9GVWD3DvuDCe/GlDb/CNk/u4/z+0h3dbtGnei/Ms2dH487jU6/Xby3+k2Nnyf/qrbUz7y0I27Gv4Mrnon4tZsiMfh/PhtbeWZ/PUl1sbdeF+uy3P3crfXWD96zkE+rsfrDjjalQCrN9bROaeIh7voBFXrWk18BtjyoD3gdki4hCRScAFwBveZUXkSuD/gNOMMTu98tJEZJKIBIlIiHPoZzyw2Hs/Hem7H/L498ofAUiODMEYw1+/+sE9lBGsb+hpf1nIbW+tdqc99d9tPLdwh/sL4sPMfdw2dzXLPPoe7/3Puka/y/OLxWVFlnXTa1zvmEbppw9puJGUkeDgpWsb7sH8bHIfNs8+0/364zsmM+vkDMD6MDx83hB33vWTehMVap2EkaGBnOwcyTB1QIL7C+HMYclMcbboHzhnMOeO6MG6R05nUHIkqbFhjExtsrdNqXYZkx5Dz+hQBiZHsO6R07lgVE9uP7UfMwYncdqQJKYPsm6uDusZycnO7qKJGbHEhFldQlGhgVw4uqd7fw+cO4R+zgbQzVP7svBX09x52x49iztP7ed+/cp14xjWs+HL4Sde6xb0dQ5hbuozu9LjRrUx8It3M92v12QXcvc7a3nf+YTzmuxCnv16O894DH++4+01TP3zAorKG7qRH5y/gWf/9wPGGBKdE+W9umQXC7daV/i5xVbHyL7CCuav3UuxV7d0R/N1IPWtQCiQC7wN3GKM2SgiU0Sk1KPco0AcsEJESp0/zzvzIoDnsEYI7QXOBM4yxjT+yu1An2/I4eqXljN3mdVq319Uyd3vruXpr7Y1+gYe+fsvD9v27wu2H5ZW4HHjyFPfBAcpUQ2zHl53Um+e+MkI9+sZg5P4180nul9nxDt44eox7vHzveMcTB+c5G71pESFEBrUMKd6clSI+3IyyG5DRLhren9OG5JEWJCdu2cMIM4RxLjeMQzvFUXmw6dz7ogUekSHsuJ3M7h1Wl8cwXZeuX68e4GS7rwylTr+uM63QcmRvHjtWMKC7Nw0JYPVD55GSlQol5zQi3WPnE6/xAiG94wiOiyQ35w1iACbcMrABG6eajV87M7uktGp0SR7fOaC7DZ6RFs3l08bksQpgxLdrfJRqdH85acjGw0l/uLuk933sMAaSup5JR0fHsTApKZvYzb1lPpf//dDo9f1BkbObogra7ILeeq/27j73bXuifO+2HiA615ZwWfr95PtvELIK6nirnfWcvc7a1s6nO3m05ALY8xB4MIm0r/Duvnret3Hu4xH3kZgRHP5neH1pVmHpc1fu69N+xiYFOEeUQBWy+Ge0wZw0p++dqf95+cnUVpVy5QnFgDWF8GlY1PZkVfKC9/sJCYsEBHhhavHcPMbq6iqrUdEuHVaP37170z3ZbfPqmEAACAASURBVKKrz8s1EiY1NpQfD1YQGxZEbFgQt5/Sj4tPsFpAvzhtgPv3T8yIY9WDp7lfe05R3N3HzSv/ZbMJsY4g9/9dXw5x4cGsfajhuc5Xrh/v/v/fZ452D5V23YNw3TR2NZZcnyPXaDnX1bIr/cVrxmIPsBHtvLK4aUofLhjVs9GAiQ9vn0ysI4hBD37uTlt2/3Se+Wobby//0Z3WPzG80c3m1rjij02sLweAW+auPqzc11tyuW3uam6Y3Icx6TGH5bfXcf3opOedc8/pAOLDg7nmxHQeOGewO80mcOWEtEajEcamx/DFL05m9gUNI1eHpETSIzqUV64f506LDgskNTbM3e8Y4zyZezsXBql1/oVdVwVVzptFrqAcF279e+1J6TiCApgywOquee+Wk/jqnpOx2QSbTfjVGQOPqSUFlepo/RIjuP3U/u6g/+UvTubLX0wFrLH/ADc5A73r85UUaf3rumnrSq9y3m9wdYkmOD+HvePC6BEd2mgls3dnTSQpMsT9RDTA788fyn/vmcqJGQ1XDldNTOPqiekEe8xoes9pA5yDJhra2a6Hy1ryyfr9XPLcklbLHYnjNvDX1NVzyZhePPGTEfzzyhO4yKOvMCPewewLhjHN+QAHWHOf//Gi4fx8al93musPdc2Jvd19iz2ireA92ePOu+skdE1f62pxu8Y+n5AW7dzWdSlq/d4TM+K4e0Z/fnm61Xofkx7Lxtln0tNZLjEiRCe6UqoFA5Ii3FcNKVGhZP3pHPcUDw+cM4RfnjaAkc5u0pMHWJ9ZV4PM9UXhCvyu/TS1dKWr1e3qnx+SEsm1zq4hz27Zu2cM4A8XDmt01X3eyB48dvFwBiU3fJan9I/n3VkT3YMxwLrSv/nkDMYfhSHRx+3TNYEBNu4/u6FFX1Rew+cbcti0v5hwZ0DvlxjO05eN5BfvZhLknDcmKTKEpy4dyT3/ymx0Arjyk6NC3fv3dtOUDG6du5phPaxRMCekxbD4vlPp4Wzpx4cH8+UvTnZffQTZbdw9Y8Bh+1FKtV9qbBh3TG8YjDj7gmHMOrmvezTRTVMyOGdED3dDy3U1fdf0/oftyzWvlKtbKcwj2Ac47zs8fdlI4p1XDa7Y8cLVY9yf92C7lTa8ZxSXj08jPNhOpsdzAO/fehIDkiKoqq1j4ANWF9OMwUkYYzp8yPRxG/i9RYUFcsu0vu45bFzSYq0/SqC94cC65n7v7dE99JuzBjHn2x2NvrVfvm5so6Xszh6e4n5M3MV1UrkMaOaGkVKqc4UEBriv3MG6Uvf8fMY6gg77/H58x2QKPUbnjOwVzcSMWO70+HJw7cMV9K3fZWuU5+nmqRnu+3iuXoARvaLcscH1BQHw4LmDO+U5Gb8J/NBwiXfKoIYunjTnI+jXndRwX3r6oEQeu3i4+0YqWMMjpw5o3C93vM3toZRqzPsZFkewnXdmndgo7TdnDmJQcgST+jZ0/141MZ2H5m9sNNpv2sAEFm3Pb9T4czUcm5syzfPLpCP5VeAf3iuK7359SqP5RhIigtnyhzMbdevYA2xcMT6tqV0opVQjoUEBXO4VL66emM7l49IaLVv5s8l9OH1IMmlxDfFneM8oThuSxK89niD21Fkrz/lV4IemJ5lq6maOUkodKREhyC6HpXkGfbBiz/+75vAJND+6fTL7ijp+GmwXvwv8SinV3Q3vFcXwXp03VcpxO5xTKaVU0zTwK6WUn9HAr5RSfkYDv1JK+RkN/Eop5Wc08CullJ/RwK+UUn7Gp8AvIrEiMk9EykRkt4jMbKaciMjjIlLg/HlcPCaaEJFRIrJKRMqd/47qqDeilFLKN762+P8BVANJwJXAcyIytIlys7AWbBmJtejKecDNACISBMwH3gRigNeA+c50pZRSR0mrgV9EHMAlwIPGmFJjzCLgQ+DqJopfCzxpjNljjNkLPAlc58ybhvWk8DPGmCpjzLNYi+Kc2u53oZRSyme+TNkwAKg1xmzzSMsEpjZRdqgzz7PcUI+8dcY0modunTP9c7yIyCysKwiAUhE50uXn44HDV1RW3vQ4eWhhKtxWj1NnTKN7jNJzyjeddZzSm8vwJfCHA8VeaUVYi6c3VbbIq1y4s5/fO6+l/WCMmQPM8aF+LRKRlcaYw2dBUo3ocfKNHiff6bHyTVccJ1/6+EuBSK+0SKDEh7KRQKmzld+W/SillOokvgT+bYBdRDzXIxsJbGyi7EZnXlPlNgIjpPF18Ihm9qOUUqqTtBr4jTFlwPvAbBFxiMgk4ALgjSaKvw7cIyI9RaQH8EvgVWfeQqAOuFNEgkXkdmf61+17C61qd3eRn9Dj5Bs9Tr7TY+Wbo36cxDS35pdnIZFY4GXgNKAAuM8Y85aITAE+M8aEO8sJ8Dhwo3PTF4HfuG7oishoZ9oQYDPwM2NM40VwlVJKdSqfAr9SSqnjh07ZoJRSfkYDv1JK+ZnjNvD7Or+QPxCRhSJSKSKlzp+tHnkzncenTEQ+cN7PceUd18dQRG4XkZUiUiUir3rlTReRLc55pRaISLpHXrCIvCwixSKSIyL3+Lrtsai54yQivUXEeJxXpSLyoEe+vx2nYBF5yflZKRGRtSJylkd+9zmnjDHH5Q/wNvAu1oNjk7EeFhva1fXqomOxELixifShWM9RnOw8Tm8B7/jLMQQuxppb6jngVY/0eOd7/SkQAvwZ+N4j/zHgO6w5pwYDOcCZvmx7LP60cJx6AwawN7Odvx0nB/CI87jYgHOdn6/e3e2c6vKD1Yl/gGpggEfaG8CfurpuXXQ8mgv8/we85fG6r/O4RfjTMQQe9Qpos4AlXudTBTDI+XofcLpH/h9cX5itbXss/zRxnFoL/H55nLyOwTqsuc661Tl1vHb1NDe/UFMzivqLx0QkX0QWi8g0Z1qjuZWMMTtwBnv8+xh6H5cyYAcwVERigBRanpOqyW07uc5dabeI7BGRV0QkHkCPE4hIEtbnaCPd7Jw6XgN/W+YX8ge/ATKAnlgPi3wkIn1pef4kfz6GrR0XOHxOKtdxadOcVMe4fGAc1mRgY7De41xnnl8fJxEJxDoWrxljttDNzilfJmk7Fum8QB6MMcs8Xr4mIlcAZ9PycapvIe9419JxKfV4XemV19q2xxVjTCmw0vnygPNp/P0iEoEfHycRsWF1i1YDrhkKutU5dby2+Nsyv5A/MlhrITSaW0lEMoBgrOPnz8fQ+7g4sO5/bDTGHAL20/KcVE1u28l17g5cT4Pa/PU4OWcveAlr0apLjDE1zqzudU519c2PTryp8g7WqBQHMInjbERKG45DNHAG1mgAO9YKamVYfY9DsbpzpjiP05s0HtVzXB9D5/EIwRpR8YbHMUpwvtdLnGmP03gExp+Ab7BGYAxyfmhdIzBa3PZY/GnhOE0ABmI1IOOwRoAt8Nfj5HxfzwPfA+Fe6d3qnPL1zdyOdUlXhcdd/WbK/gJrKFIx1vw+wR55vYEFQDmwBZjRiX+AWOADZ5DLBmZ29UnRRSdiArAC67Kw0HlSnuaRP9N5fMqwlsaM9ZdjiDX0znj9POLMm+E8RyuwRkX19tgu2HluFwMHgHu89tvstsfiT3PHCbgC2OU8P/ZjTdKY7MfHKd15bCqxumdcP1d2t3PK10naLsbq8z0DCDXGXNdMuTOcf/xTsYYnzcP6ZrrPmb8UWAr8DquP+SWgvzEmr9VKKKWU6hBtmqRNRB4FerUQ+N8Csowx9ztfTwfmGmOSRWQAsB6IN8aUOPO/c+Y/3763oZRSylcdPapnKFZ3gUsmkCQicc68na6g75Hf5FhU8Vhz1+FwjBk0aFAHV1WpI1NTV8/ugnIqauoACAsKID3Ogd2ma+2q7mPVqlX5xpiEpvI6OvA3teYuNIxVbWosas+mdmQ81twdO3asWblyZVPFlDqqluzI5/a31tCrtp6nLxtFRU0dv/5PJsGhgTx31RhOSIvp6ioqBYCI7G4ur6OHcza15i40jFU9LsftKv+wPbeEa15aTkxYIPNvn8RpQ5I4f2QP5t06iWB7AFfM+Z49h8q7uppKtaqjA39Ta+4eMMYUOPMynA93eOYf0+N2lf949n/bCbLb+NfNJ9I3IdydPjglkndmTcQY+OfCHV1YQ6V841PgFxG7iIQAAUCAiISISFPdRK8DPxORISISDTyAc81dY835shZ42Ln9RViLrb/XAe9DqU61PbeEj9bt49qTehMXHnxYfo/oUC4bl8q/V/6orX7V7fna4n8Aa/zofcBVzv8/ICJpzjm40wCMMZ8DT2CN1c8GdgMPe+zncmAscAjrgYWf6FBOdSx49n/bCQ0M4KYpGc2WuWVaXwTRVr/q9ny6uWuMeQTrgY2mhHu+MMY8BTzVzH6ygGm+Vk6p7sDV2r/55L7EOoKaLedq9b+zIptbp/WlV0zYUaylUr47XufqUarDNLT2+7RaVlv96liggV+pFrha+9ec2HTfvjft61fHAg38SrXghW92Emy3+dTad7llWl8AXlq0q7OqpVS7aOBXqhkFpVXMz9zHJSf08qm179IjOpRzhqfw75V7KKmsaX0DpY4yDfxKNeOtZdlU19Zz/aTebd72+kl9KK2q5T+r9nR8xZRqJw38SjWhuraeN77fzZT+8fRLbPsKdyNTozkhLZrXlmRRX+/7RIhKHQ0a+JVqwmcb9pNbUsUNk3zv2/d2/aQ+ZBWUs2BrbgfWTKn208CvVBNeXpxFRryDqQOanNzQJ2cOSyY5MoRXFmd1XMWU6gAa+JXysjr7EJk/FnLdpN7Y2jHVcmCAjatPTGfR9ny2HdC5CFX3oYFfKS8vL9pFRIidS07o1e59zRyfRrDdxss6tFN1Ixr4lfKQ+WMhH6/bz1UT03EEt3+5ihhHkPVA16o9/KCtftVNaOBXyskYw+yPNxEfHsytzoewOsJd0/sTFhTA7I830ZalTpXqLBr4lXL6MHMfq3Yf4t4zBhAREthh+40LD+buGQP47od8vt6iI3xU19PArxRQXl3Lnz7bwrCekfxkTGqH7/+aE9PJSHDw6Cebqa6t7/D9K9UWGviVwpqTZ39RJQ+dO5SATlg0PTDAxoPnDmFXfhmvL83q8P0r1Ra+rsAVKyLzRKRMRHaLyMxmyn3mXJjF9VMtIus98rNEpMIj/8uOeiNKHallOwt44dsdnDMihfF9Yjvt95wyMJFpAxP461c/sPbHwk77PUq1xtcW/z+AaiAJuBJ4TkSGehcyxpxljAl3/QBLgH97FTvPo8zp7am8Uu315ve7ufLFZfSIDuXBc4Z0+u/7wwXDiHYEcukLS3l/tc7jo7pGq4FfRBzAJcCDxphSY8wi4EPg6la26w1MwVqHV6lupbSqlt/NW88DH2xgSv94PrhtEslRIZ3+e1Njw5h/22TGpMVwz78y+eMnmyivru3036uUJ2lteJmIjAYWG2PCPNJ+BUw1xpzXwnYPAacaY6Z5pGUBoVhfOGuAe40xmc1sPwuYBZCWljZm9+7dPr4lpayhmVsPlPDN1jwKK2qoqK6jrKqWfUUV7MgtI6e4EoCfT+3LvWcM7JR+/ZbU1NXzh4838fpS67zuERVC38RwUqJCCAuyExYUQExYEKcMSjiiSeKUEpFVxpixTeb5EPinAP82xiR7pN0EXOkZ1JvYbjvwqDHmVY+0ScBqQIC7nD+DjDEtdniOHTvWrFy5ssV6KgWwr7CCt5dn88n6/ezMKwPAbhPCggIIC7KTFBlM38Rw+iaEMzY9hgkZcV1a38Xb81n7YyHbc0vZkVfKgeJKyqvrqKiuo9Y5q+fApAjOHp7CFRNSSYzo/KsSdXxoKfD78mhiKRDplRYJNPsYoohMBpKB/3imG2MWe7x8TESuxeoO+siHeijVrLp6w+tLs/jLF1upqKljQp84bpjUhzOGJpMQ4fsiKkfbpH7xTOoX32TegeJKPlu/n0/X5/DM/7bx4qKd/PaswVw+LrVdcwgp5Uvg3wbYRaS/MeYHZ9pIYGML21wLvG+MKW1l3war9a/UEduSU8xv3ltP5o+FTB2QwKMXDiM1Nqz1Dbu5pMgQrpvUh+sm9WFHXim/m7ee++et54O1e3ns4uH0TQjv6iqqY1SrXT0AIvIOVpC+ERgFfAqcZIw5LPiLSCiQA1xkjPnaIz0NSAVWYPXx3wH8Gqurp6Cl369dPaopVbV1/OPr7fxz4Q6iQgN56LwhnD+yByLHZ1vCGMO/V+7hj59upqKmjrtn9GfWlAzsAfo4jjpce7t6AG4FXgZygQLgFmPMRmf//2fOoZsuFwKFwAKvfUQAzwF9gUpgLXBWa0Ffqaas/bGQX/8nk20HSrl4dE8ePHcIMY6grq5WpxIRLh2XyimDEnn4ww088flWPl2/nycuGcmQHt69sUo1z6cWf1fTFr8CayTM/zbn8s6KbL7ZlkdyZAj/d9FwThmU2NVV6xKfrd/Pg/M3cqi8mlMGJjJzQipTByQe9RFKqnvqiBa/Up2quLKGrPwy9h6qoLqunnpjqK0zFJbXkFNcSU5xJct3HSSvpIrkyBDuOLU/N07pQ2QHTqZ2rDlreAon9o3jhW938u+Ve/hq8wFSokIY2zuWpIhgkqNCiA4LIsAGNhGC7TZ6xYTRO95BeAdMOa2OXdriV12isqaOr7fk8lHmPlZkHSK/tKrZsqGBASRHhTAgKZyfjkll2sAE7df2Yl0NHeA/q/awPbeUnOJKKmuanwwuMSKY8X1iOX9kD6YOTCDYHnAUa6uOBm3xq27BGMPq7EPMXZbNFxtyKKuuIz48mKkDEuiXGE6feAepsaEE2wMIsAkBIkQ7AokIth+3N2w7SmCAjTOHpXDmsBTAOtbFFbUUVdRQZwx19YbKmjp+PFjOzvwytueWsnBrLh+v209EiJ2zh6Uwc0IaI1Oju/idqKNBA7/qdLkllXyxIYe5y7LZklNCRLCdc0f04PxRPZjQJ1Zb751ARIgKCyQqrHFX2LCeUe7/19TVs3h7Ph9m7uOjdft4d+WPDO8ZxcwJaZw+JIm48O77/INqH+3qUUfMGEN5dR31xmAAUw+FFdXklVSRV1LF+r1FLNyax6b9xQAM6xnJVRPSOW9kjw5Z1lB1nJLKGj5Yu4+53+9mS04JIjC8ZxRTByQwtEcUCRFBJISHEBUWiIj18E2ATQgNDNCrsW6qXVM2dAca+LtWRXUdmXsKWZ19iNW7C8k+WMbBshoKy6vd0wo0xW4TxqTHMHVgAqcMTGRwig457O6MMWzcV8yCLbl8sy2P1dmHaOFPjN0mxDiCiA0LIj0ujBPSYzghLYYRvaIICdT7Bl1JA79qE2MMWQXlfL0llwVbclm2q4CaOus8yYh30D8pnFhHMLGOQCJCAgkQwdXoiw4LIiEimPjwINJiwzp0CUN19BVX1pBdUE5+qXUVV1RRA4AxUGcMxRU1HCqvpqC0mq0HSthdUA5AkN3GiRlxnDookVMHJR4XT1IfazTwqybV1xuKKmqsD3VpFVv2l7Aq+xCrsg65Z6/slxjOqYMSmZgRy+jUmOP+ISnVPvmlVazJLmTpjgIWbM1lV741UV7P6FDGpMcwJj2GAUkRJEQEER8eTFRooHYVdRIN/H4kp6iStT8ecs72WMbO/DIOllVRUV1HeXUdVbX1GFeffBN/+p7RoZyQHsO43jGcMlBbaqp9duWXsXBrLiuzDrFy90EOFB8+bNd1z8D1rEFokJ3QIBvx4cH0iXfQNyGcfonhjE6L1tlJ20AD/3GqpLKGTfuK2bCvmMwfC1m1+xB7Cyvc+SlRIWQkOEiMCCE0KICwwACC7DZszq4ZESE6NJB4Z9dMn3gHKVGhXfiO1PHMGMPewgqyC8rJK60iv7SaoooaqyFioN4YqmrrqaixpqU+UFzpnKq64csiLTaMMekxjOwVxbCeUQxOidSBAs3QcfzdWE1dPVn5ZewrqiSnqIKcoirKqmupqaunts5QZ4z7g1FbbzhYVk1+aRX5JVXsK6p07ycpMpgx6THcMLkPJ6RFMzA5grAg/fOq7kNE6BUTRq+Ytl1FllbVsjWnmNW7C1m5+yDf/ZDPvDV7nfuEHlGhxIdbXUcxjiDstoaGjd0m2G02AgMER7Cd5KgQUqJCSIkKpXdcmN8OJdYW/1FijCG3pIotOSVsyylhc04xm/eXsD23xH3j1CXYbiMwwIY9wDpxXS30ALFGUMSHBxMXHkTvOAfDe0YxtEckiZF6Caz8g+uztGFvEev3Flk3n8uqyS+porC82tlYsq4gauutqT9q6uqpqm38JHOQ3caApHAGJ0cyMDmCQcmRDEgOJyE8+Li476BdPR3AGKu1vSu/jOyD5Rwsq+ZgWTWHymuoqavHGKtMvTHUOU+6mtp6Cjxa6GXVde79JUYEMzglksEpkQxKjqBnTCjJkSEkRgbr4/NKdYKq2jpyi6vYX1TJnkPlbMkpYfP+YjbvLya/tNpdLjzYTkJEMHGOIGIdQQQ6u0cDBGcjzGqIBdltxIQFEhMWRFx4EGmxDjLiHd1mAIR29bTCuB5pr62noLTKPXRtV345O/KsJfF25pW5h7K5BNiEmLBAggJs7pPBJkKATbAJ2G02Yh1BjOgVTbyzhT4wOYIBSRHEdpOTQyl/EWwPIDU2jNTYMMb3iW2Ul19axbacEveQ1HxnHNhdUE5NvdWwq6u3GnauRl5VbT2FFTXUeT3oEBMWSEZCOH0TrBvT6XEOEiKCSXBeqQfbbQTYpEuvKnxdiCUWeAk4HcgHfmuMeauJco8AvwM8b92PMMbsdOaPcu5nMLAZ+JkxZm1rv9/XFn9JZQ27C8rJKrBa5XsPVbC3sIKcokpKq2opq6qlrMp60tTFwGF/OE9JkcH0TbDmkclICCcjwUF6bBhx4cFEhugcMkr5s/p6Q0llLXmlVWQfLGNnXhk78srYlW+NqssraX7ywQCbNFp+0GYTHEEBhIfYCQ8OJDkymF4xYfSMCSUtNoz0uDDS43yfWbUjWvz/AKqBJKwVuD4RkcymVuAC3jXGXNVEJYKA+cAzwD+Bm4H5ziUdq73Le9pfVMkfPt6ETaCuHmrr66mpM1RU15JfanWlHCiu5FB54xZ5TFig+2ZSZIgdR7CdsOAA7F7zldttNuw2IchuIy482H2jKD1OH0BSSjXPZmuYE8l65qVxflFFDT8edI5iKqmioKyamtp6695DfeN7DrX1hvKqOkqraimprGF/USWrswsP62mIc1gPSVoPSgYTEhhAYIB1EzvABvWm6aHanloN/CLiAC4BhjnX0F0kIh8CVwP3tXpkGkxz/r5njHWZ8ayI/Ao4Ffi8pQ0PllXzzvJs6o31LWl3vsmQQGusb2psGKPTYkiLDaN3nDXfeGpsmM45rpTqUlGhgUR5TIx3JEoqa8g+WM7ugnJ25Zex51A5eSXV5JVWsTOvjKraemrrnaMA6w0254imlvgSGQcAtcaYbR5pmcDUZsqfJyIHgf3A340xzznThwLrTOO+pXXO9MMCv4jMAmYBpKWlsXH2mT5UVSmlji8RIYEM7RHF0B5t+wKR2c3n+TKINRwo9korwlpD19u/sPrvE4CbgIdE5AqP/RT5uB+MMXOMMWONMWMTEhJ8qKZSSilf+BL4SwHvaRUjgRLvgsaYTcaYfcaYOmPMEuCvwE/auh+llFKdx5fAvw2wi0h/j7SRQFM3dr0ZcN+43giMkMadTyN83I9SSqkO0mrgN8aUAe8Ds0XEISKTgAuAN7zLisgFIhIjlvHAnVgjeQAWAnXAnSISLCK3O9O/7oD3oZRSyke+TlRxKxAK5AJvA7cYYzaKyBQRKfUodzmwHav75nXgcWPMawDOIZsXAtcAhcANwIWtDeVUSinVsXwa72iMOYgVtL3Tv8O6aet6fYV3Ga/ya4AxbayjUkqpDuSfU9MppZQf08CvlFJ+RgO/Ukr5GQ38SinlZzTwK6WUn9HAr5RSfkYDv1JK+RkN/Eop5Wc08CullJ/RwK+UUn5GA79SSvkZDfxKKeVnNPArpZSf0cCvlFJ+RgO/Ukr5GZ8Cv4jEisg8ESkTkd0iMrOZcveKyAYRKRGRXSJyr1d+lohUiEip8+fLjngTSimlfOfTQizAP4BqIAkYBXwiIpnGGO/1cgVrha11QF/gSxH50RjzjkeZ84wxX7Wz3koppY5Qqy1+EXEAlwAPGmNKjTGLgA+Bq73LGmOeMMasNsbUGmO2Yq23O6mjK62UUurI+dLVMwCoNcZs80jLBIa2tJGICDAF8L4qmCsieSLypYiMbGH7WSKyUkRW5uXl+VBNpZRSvvAl8IcDxV5pRUBEK9s94tz/Kx5pVwK9gXRgAfCFiEQ3tbExZo4xZqwxZmxCQoIP1VRKKeULXwJ/KRDplRYJlDS3gYjcjtXXf44xpsqVboxZbIypMMaUG2MeAwqxrgqUUkodJb4E/m2AXUT6e6SN5PAuHABE5AbgPmC6MWZPK/s2WDeElVJKHSWtBn5jTBnwPjBbRBwiMgm4AHjDu6yIXAn8H3CaMWanV16aiEwSkSARCXEO9YwHFnfEG1FKKeUbXx/guhUIBXKBt4FbjDEbRWSKiJR6lHsUiANWeIzVf96ZFwE8BxwC9gJnAmcZYwo64o0opZTyjU/j+I0xB4ELm0j/Duvmr+t1nxb2sREYcQR1VEop1YF0ygallPIzGviVUsrPaOBXSik/o4FfKaX8jAZ+pZTyMxr4lVLKz2jgV0opP6OBXyml/IwGfqWU8jMa+JVSys9o4FdKKT+jgV8ppfyMBn6llPIzGviVUsrPaOBXSik/41PgF5FYEZknImUisltEZjZTTkTkcREpcP48LiLikT9KRFaJSLnz31Ed9UaUUkr5xtcW/z+AaiAJuBJ4TkSGNlFuFtaCLSOxFl05D7gZQESCgPnAm0AM8Bow35mulFLqKGk18IuIA7gEeNAYU2qMWQR8CFzdRPFrgSeNMXuMMXuBJ4HrnHnTfHM08gAABodJREFUsFb8esYYU2WMeRZrofVT2/0ulFJK+cyXpRcHALXGmG0eaZnA1CbKDnXmeZYb6pG3zhhjPPLXOdM/996RiMzCuoIAKBWRrT7UtTPFA/ldXIfuQo9FAz0WDfRYNOgOxyK9uQxfAn84UOyVVoS1eHpTZYu8yoU7+/m981raD8aYOcAcH+p3VIjISmPM2K6uR3egx6KBHosGeiwadPdj4UsffykQ6ZUWCZT4UDYSKHW28tuyH6WUUp3El8C/DbCLSH+PtJHAxibKbnTmNVVuIzDCc5QP1g3gpvajlFKqk7Qa+I0xZcD7wGwRcYjIJOAC4I0mir8O3CMiPUWkB/BL4FVn3kKgDrhTRIJF5HZn+tftewtHTbfpduoG9Fg00GPRQI9Fg259LKTxvdZmConEAi8DpwEFwH3GmLdEZArwmTEm3FlOgMeBG52bvgj8xnVDV0RGO9OGAJuBnxlj1nTsW1JKKdUSnwK/Ukqp44dO2aCUUn5GA79SSvkZvw/8IjJYRL4WkSIR2S4iF3nkhYnIP0Uk35n/rUdei/MSHYtaORaXishmESkRkU0icqHXtr8QkRwRKRaRl0Uk+Oi/gyMnIreLyEoRqRKRV73ypovIFuccUwtEJN0j7/+3d3YhVlVRHP8tJmemDyIza8oeoiLG9MEhGskefGi4GkJQUo9RBOrDMNJYDxWBg5KmvQRBQUEFMUiPvUS9lCaWpT0UmmYlRh+jUgM3LVJq9bD24J7DmXM/pjPnnjnrBxvuPvusy1n/2Xefs/dZe01P8Lce/B9t1rZTmYUWL4nIidBHjonIownb0uXqaleL6JxrReSsiOxv1TZXVLWyBdvA9i0wCnRh6SPOA3eE9neAPcDi0H5XZLsROA7cDCwBjgKbivYpDy2CfxeA+7E0G+uAP4Hrg+0a4DS2C3shFsG1s2ifWvT/ISzP1KvAW9Hx67CNhg8DvcBu4LOofQfwSfB7KTABrG3GtlPLLLQYA/qxB8qVwCSwKrR1A6eAJ4EeYCTUu4v2Nw8tovNeB/YB+1u1zdWvooUt+I+6HNtYJtGxD4FtoQPXgatnsD0AbIjqT5ThR92mFiuBM4nzzwL3hM/jwAtR233ARNE+tanD9sQPfANwIKpfCfwF9If6L0Atat8G7GnGttNLq1qk2L8HbAmfa8DPif71I+Em2emlHS2AVcCnwOOJgb/wflH5pZ4UBBsEB7EnkrGw1PO1iKyPzsvKSzRfmNLiEPCNiDwgIl1hmedvLNcSpGtxg4gsmtOrzYdpvqnta/keWCYiC4Ebyc5PlWqb8zXnRdP+iMjlwN1c2qCZlaurjGRqISJdwCvAMJAMnSy8X1R94D8OnAGeFpEFIlLDks9dgS3hLMemZDdhf8C3RWRpsM3KS1RGZtRCVf/BNueNYwP+OLAxdFhI1wJmyMNUMrJyTF0V1ZNtjWzLSCv+vIYNbh+0YVsGGvkzAhxU1cNt2OZOpQd+Vb2Ird+tw9ZmtwDvAj9hU6+LwHZVvaCqe4GPsCkrZOclKh1ZWojIELALS63djd0Q3ohezqVpAfMjD1NWjqlzUT3Z1si2jDTlj4jsxh6aHol+D5XRImQtGAGea9X2f73CDCo98AOo6lequlpVF6nqGuBW4HMuLWNMOz36nJWXqJRkaLEC2Keqh1T1X1X9AjgIDAXTNC1Oq+pvc3n9OTHNN7H/T3EbcERVJ4Ffyc5PlWqb8zXnRUN/RGQMCwKoqWo9YTufcnVlaTGILQEeFZEJ4GVgMER9dTWwnRuKfmlSdME6Xy+2vPMUcBKLOlgAfAc8j0W83Ivdkade6m3C0k4swZaCjlDiqJ4GWqzGcouvCOcNYKk7aqG+Fpsl3Alcg+VfKltUz2XB9x1YHqrecGwxNg1fH469yPRIlp3AXiyqpx+7EUxF9WTadmqZhRbPACeAvpTvnIrq2Rz61DDliOppWYvgX19UNmMPSn2d0i8KF7bogoVSTWLTr/eB26O2Zdhb+fNYuOaDUZtgyx+/h7KLKGKhjKWBFsPYjfAP4AdCtEbUPoqFdNaBN4Geov1p0fet2IwuLltD2xBwDFv++xi4JbLrwfJY1YP/o4nvndG2U8sstFDsHdC5qDwbtQ8Ah4Ptl8BA0b7mpUXiOx4jiurphH7huXocx3EqRuXX+B3HcaqGD/yO4zgVwwd+x3GciuEDv+M4TsXwgd9xHKdi+MDvOI5TMXzgdxzHqRg+8DuO41SM/wCb98vtoAIysQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# For illustration, we can show that by dot-product comparison,\n",
    "# the nearby PE vectors are more alighned with the multiplied vector.\n",
    "\n",
    "pos_encoding = positional_encoding(length=2048, depth=512)\n",
    "\n",
    "pos_encoding/=tf.norm(pos_encoding, axis=1, keepdims=True)\n",
    "p = pos_encoding[1000]\n",
    "dots = tf.einsum('pd,d -> p', pos_encoding, p)\n",
    "plt.subplot(2,1,1)\n",
    "plt.plot(dots)\n",
    "plt.ylim([0,1])\n",
    "plt.plot([950, 950, float('nan'), 1050, 1050],\n",
    "         [0,1,float('nan'),0,1], color='k', label='Zoom')\n",
    "plt.legend()\n",
    "plt.subplot(2,1,2)\n",
    "plt.plot(dots)\n",
    "plt.xlim([950, 1050])\n",
    "plt.ylim([0,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wbxXpHtEQEW5"
   },
   "source": [
    "- Next, it uses the ***Intra-attention*** [26]:\n",
    "    - It is also known as ***self-attention*** [21], with the first implemenation by the modified LSTMs, to see the inter-relationship between tokens (rather than considering both inputs and outputs as the above-mentioned general or **inter attention**):\n",
    "    - This allows simulatation of the human reading, since we can find important context rather than memorizing everything by LSTM.\n",
    "- As a basis for all their attention mechanisms, they defined their own attension mechanisms:\n",
    "    1. **Scaled dot-product attention**:\n",
    "\\begin{equation}\n",
    "\\text{Attention}\\left ( Q, K, V \\right ) = \\text{softmax}  \\left ( \\frac{QK^\\text{T}}{\\sqrt{d_k}} \\right )V\n",
    "\\end{equation}\n",
    "        - Here,\n",
    "            - $Q$, $K$, and $V$ are respectively query, key, value vectors packed into seperate matrices.\n",
    "            - $d_k$ is the dimension of the query or key space.\n",
    "        - Thus, its difference with dot-product attention is:\n",
    "            - the ***scaling*** by the *scale factor* $1 / \\sqrt{d_k}$ that counteract the issue of vanishing gradient of the softmax function (figure below);\n",
    "<img src=\"img/scaled.png\" width=\"100px\">\n",
    "<center>Scaled dot-product attention [21].</center>\n",
    "            - the optional mask that is used for decoder self-attention to avoid leftward flow of information and therefore keeping its auto-regressive property by setting all the values of the illegal connectin to −∞  (e.g., the self.attention with masking as shown below).\n",
    "<img src=\"img/causal2.png\" width=\"200px\">\n",
    "<center>Simplified representation of auto-regressive self-attention (image from tensorflow.org/responsible_ai).</center>\n",
    "    2. **Multi-head attention**:\n",
    "\\begin{equation}\n",
    "\\text{MultiHead}(Q, K, V ) = \\text{Concat} \\left ( \\text{head}_1, ..., \\text{head}_\\text{h} \\right )W^O\\text{,}\\quad \\text{where}\\quad \\text{head}_\\text{i} = \\text{Attention}\\left ( QW_{i}^{Q}, KW_{i}^{K}, VW_{i}^{V} \\right )\n",
    "\\end{equation}\n",
    "        - Here,\n",
    "            - $Q$, $K$, $V$, and $O$ are respectively the superscripts query, key, value, and output;\n",
    "            - $W$ is the learnable linear projection of each input' or output' vector to the attention model.\n",
    "        - Thus, each head gives a diverse representation depending on the projection, which by concatenation of with h heads, it can jointly attend to information from different prespectives to then average them (figure below).\n",
    "<img src=\"img/multihead.png\" width=\"160px\">\n",
    "<center>Multi-head attention [21].</center>\n",
    "        - Keras supports `layers.MultiHeadAttention`.\n",
    "- Now, all the attention mechanism can be defined by using a base attention class, that also has methods for layer noramlization (by `layers.LayerNormalization()`) and resedual connections (by `layers.Add()` instead of `+` that cannot propogated the Keras masks)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "executionInfo": {
     "elapsed": 505,
     "status": "ok",
     "timestamp": 1671396900629,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "Y_YMY-tVz7sl"
   },
   "outputs": [],
   "source": [
    "class BaseAttention(tf.keras.layers.Layer):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__()\n",
    "        self.mha = tf.keras.layers.MultiHeadAttention(**kwargs)\n",
    "        self.layernorm = tf.keras.layers.LayerNormalization()\n",
    "        self.add = tf.keras.layers.Add()\n",
    "\n",
    "\n",
    "# the cross attention (i.e., between encoder and decoder)\n",
    "\n",
    "class CrossAttention(BaseAttention):\n",
    "    def call(self, x, context):\n",
    "        attn_output, attn_scores = self.mha(\n",
    "            query=x,\n",
    "            key=context,\n",
    "            value=context,\n",
    "            return_attention_scores=True)\n",
    "    \n",
    "        # Cache the attention scores for plotting later.\n",
    "        self.last_attn_scores = attn_scores\n",
    "    \n",
    "        x = self.add([x, attn_output])\n",
    "        x = self.layernorm(x)\n",
    "        \n",
    "        return x\n",
    "\n",
    "\n",
    "# The global self attention layer (i.e., without masking)\n",
    "\n",
    "class GlobalSelfAttention(BaseAttention):\n",
    "    def call(self, x):\n",
    "        attn_output = self.mha(\n",
    "            query=x,\n",
    "            value=x,\n",
    "            key=x)\n",
    "        x = self.add([x, attn_output])\n",
    "        x = self.layernorm(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# The causal self attention layer (i.e., with masking)\n",
    "\n",
    "class CausalSelfAttention(BaseAttention):\n",
    "    def call(self, x):\n",
    "        attn_output = self.mha(\n",
    "            query=x,\n",
    "            value=x,\n",
    "            key=x,\n",
    "            use_causal_mask = True)\n",
    "        x = self.add([x, attn_output])\n",
    "        x = self.layernorm(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1671396902844,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "nFiYO40oz7sl"
   },
   "outputs": [],
   "source": [
    "# other components:\n",
    "\n",
    "# the pointwise ffnn.\n",
    "\n",
    "class FeedForward(tf.keras.layers.Layer):\n",
    "    def __init__(self, d_model, dff, dropout_rate=0.1):\n",
    "        super().__init__()\n",
    "        self.seq = tf.keras.Sequential([\n",
    "          tf.keras.layers.Dense(dff, activation='relu'),\n",
    "          tf.keras.layers.Dense(d_model),\n",
    "          tf.keras.layers.Dropout(dropout_rate)\n",
    "        ])\n",
    "        self.add = tf.keras.layers.Add()\n",
    "        self.layer_norm = tf.keras.layers.LayerNormalization()\n",
    "  \n",
    "    def call(self, x):\n",
    "        x = self.add([x, self.seq(x)])\n",
    "        x = self.layer_norm(x) \n",
    "        return x\n",
    "\n",
    "# 1 encoder layer\n",
    "\n",
    "class EncoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self,*, d_model, num_heads, dff, dropout_rate=0.1):\n",
    "        super().__init__()\n",
    "    \n",
    "        self.self_attention = GlobalSelfAttention(\n",
    "            num_heads=num_heads,\n",
    "            key_dim=d_model,\n",
    "            dropout=dropout_rate)\n",
    "    \n",
    "        self.ffn = FeedForward(d_model, dff)\n",
    "    \n",
    "    def call(self, x):\n",
    "        x = self.self_attention(x)\n",
    "        x = self.ffn(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "# 1 decoder layer\n",
    "\n",
    "class DecoderLayer(tf.keras.layers.Layer):\n",
    "    def __init__(self,\n",
    "                 *,\n",
    "                 d_model,\n",
    "                 num_heads,\n",
    "                 dff,\n",
    "                 dropout_rate=0.1):\n",
    "        super(DecoderLayer, self).__init__()\n",
    "    \n",
    "        self.causal_self_attention = CausalSelfAttention(\n",
    "            num_heads=num_heads,\n",
    "            key_dim=d_model,\n",
    "            dropout=dropout_rate)\n",
    "    \n",
    "        self.cross_attention = CrossAttention(\n",
    "            num_heads=num_heads,\n",
    "            key_dim=d_model,\n",
    "            dropout=dropout_rate)\n",
    "    \n",
    "        self.ffn = FeedForward(d_model, dff)\n",
    "    \n",
    "    def call(self, x, context):\n",
    "        x = self.causal_self_attention(x=x)\n",
    "        x = self.cross_attention(x=x, context=context)\n",
    "    \n",
    "        # Cache the last attention scores for plotting later\n",
    "        self.last_attn_scores = self.cross_attention.last_attn_scores\n",
    "    \n",
    "        x = self.ffn(x)  # Shape `(batch_size, seq_len, d_model)`.\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8eDuePwNz7sm"
   },
   "source": [
    "- The encoder and encoder layers are connected other custom layers, as clarified in the illuestrated e.g. below.\n",
    "<img src=\"img/4layer.png\" width=\"300px\">\n",
    "<center>A simplified representation of a 4-layer transformer (image from tensorflow.org/responsible_ai).</center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "executionInfo": {
     "elapsed": 1,
     "status": "ok",
     "timestamp": 1671396903453,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "MrMka4fPz7sm"
   },
   "outputs": [],
   "source": [
    "class Encoder(tf.keras.layers.Layer):\n",
    "    def __init__(self, *, num_layers, d_model, num_heads,\n",
    "                 dff, vocab_size, dropout_rate=0.1):\n",
    "        super().__init__()\n",
    "    \n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "    \n",
    "        self.pos_embedding = PositionalEmbedding(\n",
    "            vocab_size=vocab_size, d_model=d_model)\n",
    "    \n",
    "        self.enc_layers = [\n",
    "            EncoderLayer(d_model=d_model,\n",
    "                         num_heads=num_heads,\n",
    "                         dff=dff,\n",
    "                         dropout_rate=dropout_rate)\n",
    "            for _ in range(num_layers)]\n",
    "        self.dropout = tf.keras.layers.Dropout(dropout_rate)\n",
    "  \n",
    "    def call(self, x):\n",
    "        # `x` is token-IDs shape: (batch, seq_len)\n",
    "        x = self.pos_embedding(x)  # Shape `(batch_size, seq_len, d_model)`.\n",
    "    \n",
    "        # Add dropout.\n",
    "        x = self.dropout(x)\n",
    "    \n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x)\n",
    "    \n",
    "        return x  # Shape `(batch_size, seq_len, d_model)`.\n",
    "\n",
    "\n",
    "class Decoder(tf.keras.layers.Layer):\n",
    "    def __init__(self, *, num_layers, d_model, num_heads, dff, vocab_size,\n",
    "                 dropout_rate=0.1):\n",
    "        super(Decoder, self).__init__()\n",
    "    \n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "    \n",
    "        self.pos_embedding = PositionalEmbedding(vocab_size=vocab_size,\n",
    "                                                 d_model=d_model)\n",
    "        self.dropout = tf.keras.layers.Dropout(dropout_rate)\n",
    "        self.dec_layers = [\n",
    "            DecoderLayer(d_model=d_model, num_heads=num_heads,\n",
    "                         dff=dff, dropout_rate=dropout_rate)\n",
    "            for _ in range(num_layers)]\n",
    "    \n",
    "        self.last_attn_scores = None\n",
    "    \n",
    "    def call(self, x, context):\n",
    "        # `x` is token-IDs shape (batch, target_seq_len)\n",
    "        x = self.pos_embedding(x)  # (batch_size, target_seq_len, d_model)\n",
    "    \n",
    "        x = self.dropout(x)\n",
    "    \n",
    "        for i in range(self.num_layers):\n",
    "            x  = self.dec_layers[i](x, context)\n",
    "    \n",
    "        self.last_attn_scores = self.dec_layers[-1].last_attn_scores\n",
    "    \n",
    "        # The shape of x is (batch_size, target_seq_len, d_model).\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xg69AyIpz7sm"
   },
   "source": [
    "- Now, the transformer custom model can be defined with a special attenstion on the possible effect of `_keras_mask` on the loss function, which should be deleted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1671396908332,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "NXUMJlhgz7sm"
   },
   "outputs": [],
   "source": [
    "class Transformer(tf.keras.Model):\n",
    "    def __init__(self, *, num_layers, d_model, num_heads, dff,\n",
    "                 input_vocab_size, target_vocab_size, dropout_rate=0.1):\n",
    "        super().__init__()\n",
    "        self.encoder = Encoder(num_layers=num_layers, d_model=d_model,\n",
    "                               num_heads=num_heads, dff=dff,\n",
    "                               vocab_size=input_vocab_size,\n",
    "                               dropout_rate=dropout_rate)\n",
    "    \n",
    "        self.decoder = Decoder(num_layers=num_layers, d_model=d_model,\n",
    "                               num_heads=num_heads, dff=dff,\n",
    "                               vocab_size=target_vocab_size,\n",
    "                               dropout_rate=dropout_rate)\n",
    "    \n",
    "        self.final_layer = tf.keras.layers.Dense(target_vocab_size)\n",
    "    \n",
    "    def call(self, inputs):\n",
    "        # To use a Keras model with `.fit` you must pass all your inputs in the\n",
    "        # first argument.\n",
    "        context, x  = inputs\n",
    "    \n",
    "        context = self.encoder(context)  # (batch_size, context_len, d_model)\n",
    "    \n",
    "        x = self.decoder(x, context)  # (batch_size, target_len, d_model)\n",
    "    \n",
    "        # Final linear layer output.\n",
    "        logits = self.final_layer(x)  # (batch_size, target_len, target_vocab_size)\n",
    "    \n",
    "        try:\n",
    "            # Drop the keras mask, so it doesn't scale the losses/metrics.\n",
    "            # b/250038731\n",
    "            del logits._keras_mask\n",
    "        except AttributeError:\n",
    "            pass\n",
    "    \n",
    "        # Return the final output and the attention weights.\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "scEqO55tz7sm"
   },
   "source": [
    "E.g., using [Portuguese-English translation dataset](https://www.tensorflow.org/datasets/catalog/ted_hrlr_translate#ted_hrlr_translatept_to_en)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 168,
     "referenced_widgets": [
      "d069f92b086441f99ed23a1aefcfbd60",
      "a23e10b75632416a831d67a22e1d18bb",
      "26e450f753ff4c5296abc1ab46d282bd",
      "aff31d1ccb204103972f2ca7dd016d96",
      "fc05cddd40ab4076a5e1fa79fb50468d",
      "0d408ecb99354ade8ebad91d18702705",
      "b31dce9c8e074b99832accc016c8b0f1",
      "326b326fe9df4122a58b9d974a65953d",
      "ec56928587114c4d9a071ddeb396a6e4",
      "03a511833ccb4513bface70d1e62042e",
      "833914562fec4da99e4a178c43b85c8e",
      "d47a155b8c9146d1a8eb80c228a8caf3",
      "f52e89e54b624ed8923a63aac94b5fe4",
      "ca5186021ba844ce9b9b7bcdf0a3e9ec",
      "f827dcae35604af492d0984ad0ee9c97",
      "4a3688bcb22c47e2ac6e6660ea369f71",
      "e38ee97623474b60900b1110402b62ca",
      "e3a1ff7cbd4b470588f20c5aea5c2812",
      "157002f4e0f24e479f4ba497e8ba2b76",
      "717f5951a5bb4e39babe308912dc391f",
      "32c6f8706db54aed9031efb314fa10b1",
      "2c531649f8e74016a8c8529c7cf2ba64",
      "fba171f398da41a19ca1ec7d870db662",
      "768d778dea1f47ed8a0aab960e384da4",
      "e1750ad0e52540669da5affce95aca74",
      "f9148891a3664014a2a8f4a2ab64a220",
      "a194453ab86d4a06ad3f1353bc7a2cec",
      "74466042f43b4f23a78aee2a4639f808",
      "fbed69c140c044baa60bf37c2bec3857",
      "dfaccc081ff743d7899658021fd21318",
      "af6c0ad3bb0b45358fc284b4dc765e7a",
      "56a9ea20d9c04820bd2243efa1c8a0ff",
      "361ffbd5f2b74f5ba58ddae9a9fcadb5",
      "16a4e508013c4b93acbb291cb77c5ce8",
      "9e09e6206da948788e6529bcaee79b00",
      "37d112806dad488ca624a99a2e4e2785",
      "41841b7a60a64791ba1e97d601fe43b3",
      "00cd33f6c452418db7a15306114b9f05",
      "45595c8ff42d4837836d7d7cfecd9aa3",
      "1e6dfb42441c4f17a6b64324cf62e765",
      "aa31e8b550d044af8ad4b8b9e3480292",
      "da2b89043e2f4c24b4ff42a5376f4371",
      "4607f99cd4e64cc594db6d00fc4bee7a",
      "2da098d7744f44eb81a8eacb3f8cb711",
      "a9d091eeeea545fa891cea89abd69f58",
      "662f4379bf2b473eaeb72ee31f144e5a",
      "0d8fcd67e58c406a9b093f681dcc7a77",
      "8d49812246354321b1f65421c9baaa5c",
      "a9ea451184f1469d9b56cbdcd51ba579",
      "522ef828d4a744e78892b5641b99988e",
      "26ea44ffac52464ea1f3d1d4de67da91",
      "0e5ee5b41e1248e99810003836896473",
      "8acc26bb1e564c89aa05506eced82cc8",
      "66edae33ebb84ab6a1ba3bb79b091542",
      "e992c27b3eb8463fb03ba08cc31ad0d1",
      "d7b94d523f2a4bbd91fb465178ac6e0b",
      "d4a1cbacfc3d4dc39df02c7b6da95f40",
      "bfadf2a622a74dafa8e844383e1010e1",
      "75ea4dd373294c8bac2757a61dea2bc5",
      "45bac909d1dc4e16a297e25a0edb5a26",
      "a1df0ae0c7c24c66b00556c56c123f28",
      "b2e300f3b3124049827dd331dc1966a9",
      "b0671f5dfad045dd8cc22525bd8148ef",
      "43b2355f92684db19b8597b012a1917a",
      "04c99482102e4ce799991de3308d8089",
      "973469e2dfc84062a90c87aed43310be",
      "ffa91a31b4ad444780645507b3d56453",
      "6a839391facd410fb9ce3fbc194efede",
      "5abc8a5b299f488ab905e60be7cd0012",
      "5b4bd79e8c7b4b6b8e6d16732a47e050",
      "28d32b1732b74a859725717db2749f98",
      "24ba73502d364163af4eaff8f989199c",
      "bca7023d538c4978b401874b15717087",
      "d1960ffbcbc3461d8baf9076bd2c2f67",
      "f51df18a6d794ec38ec7cc0fb470fea9",
      "88e7206a195640ff9eeeefac0cf86c2f",
      "cc458249511d4b8e917bde34a9213f4c",
      "89f05f0c418042338acbe160f007d9f2",
      "4cbf9b42f7be4b6298cee16ec037697e",
      "6562ce5c53bd4555afeceaee989c1e9d",
      "da4a94ac24a14348a32ac14a18e6a9a4",
      "f142ff7976ef4c6ebdefcfffae8ada71",
      "10957d9c84244089b7700dc7448a4265",
      "525c7216bae048b0b5447f21e2bbb2ad",
      "836e4a996cfc4486a1fb3e72dab946cf",
      "50ed4a177c5140dd9c958a74418e90bb",
      "1a82f261d2d94520b631cfe320e47992",
      "1cc12d9744d5424b9a23f0b9d4c5a07f",
      "bb2ac279c8ae42b5a82bf9caf36b3818",
      "a0ad7da1aa904b3793781e82347839d6",
      "25ac286850d248a7bba9d2286d1b2b58",
      "00e6339db2cd4fa4805fa52656856946",
      "da3501a7bae049f5908ba83c546d5c21",
      "199f5dde766640d78f8bdb4bcb018f80",
      "805ff42ca705407aae5851dd0c4d311f",
      "b7e62b474add4ec997b819826deed3f1",
      "dec0aafa5c544c0fa9d9ade8a1b5ea39",
      "8443ca78174a4bd3ab46d014568c0459",
      "5de1f721afc345faab1488846d847b4d",
      "f24d47724f8744a3bcd643e6660efe45",
      "dbb6fa23119443a5bea25bc2c5e560e0",
      "904e62bf412c422d893f1256f494e0a0",
      "f1430049a4934fb4b6e6bd4ffa19ff14",
      "e745d8bf7da04409a025204d44961bb0",
      "3fbfd0e2483d4cc69ef982f604eda87a",
      "355605003a024848abe079a2a383ffb3",
      "18153dfd563549c78770122912d7d8c3",
      "f176ee46da264916890c84776263b8d6",
      "081e224382ed46de89ddbfc02442dd42",
      "6b3c0db62591435c9752f9d38653e703"
     ]
    },
    "executionInfo": {
     "elapsed": 28749,
     "status": "ok",
     "timestamp": 1671396939540,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "0SzAvbMWz7sm",
    "outputId": "31299537-a5cb-43de-d561-eeb27f51ebe7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset 124.94 MiB (download: 124.94 MiB, generated: Unknown size, total: 124.94 MiB) to ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d069f92b086441f99ed23a1aefcfbd60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Completed...: 0 url [00:00, ? url/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d47a155b8c9146d1a8eb80c228a8caf3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dl Size...: 0 MiB [00:00, ? MiB/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fba171f398da41a19ca1ec7d870db662",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extraction completed...: 0 file [00:00, ? file/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "16a4e508013c4b93acbb291cb77c5ce8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating splits...:   0%|          | 0/3 [00:00<?, ? splits/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a9d091eeeea545fa891cea89abd69f58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train examples...:   0%|          | 0/51785 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d7b94d523f2a4bbd91fb465178ac6e0b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-train.tf…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ffa91a31b4ad444780645507b3d56453",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating validation examples...:   0%|          | 0/1193 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "89f05f0c418042338acbe160f007d9f2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-validati…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bb2ac279c8ae42b5a82bf9caf36b3818",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test examples...:   0%|          | 0/1803 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f24d47724f8744a3bcd643e6660efe45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-test.tfr…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset ted_hrlr_translate downloaded and prepared to ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "# downloding and loading the dataset\n",
    "examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en',\n",
    "                               with_info=True,\n",
    "                               as_supervised=True)\n",
    "\n",
    "train_examples, val_examples = examples['train'], examples['validation']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 22,
     "status": "ok",
     "timestamp": 1671396939540,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "A-wx1KiUz7sn",
    "outputId": "f8c33085-9e87-4c5b-beac-62ae5f488722"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> Examples in Portuguese:\n",
      "e quando melhoramos a procura , tiramos a única vantagem da impressão , que é a serendipidade .\n",
      "mas e se estes fatores fossem ativos ?\n",
      "mas eles não tinham a curiosidade de me testar .\n",
      "\n",
      "> Examples in English:\n",
      "and when you improve searchability , you actually take away the one advantage of print , which is serendipity .\n",
      "but what if it were active ?\n",
      "but they did n't test for curiosity .\n"
     ]
    }
   ],
   "source": [
    "# for demonstration, sample e.g.,\n",
    "\n",
    "for pt_examples, en_examples in train_examples.batch(3).take(1):\n",
    "    print('> Examples in Portuguese:')\n",
    "    for pt in pt_examples.numpy():\n",
    "        print(pt.decode('utf-8'))\n",
    "    print()\n",
    "    \n",
    "    print('> Examples in English:')\n",
    "    for en in en_examples.numpy():\n",
    "        print(en.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1381,
     "status": "ok",
     "timestamp": 1671396940902,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "DSCID-n0z7sn",
    "outputId": "7648154b-1b05-42e4-e8ec-d8dc5b9a4e52"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/download.tensorflow.org/models/ted_hrlr_translate_pt_en_converter.zip\n",
      "184801/184801 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "# downloading and loading the tokenizers of each language\n",
    "\n",
    "model_name = 'ted_hrlr_translate_pt_en_converter'\n",
    "tf.keras.utils.get_file(\n",
    "    f'{model_name}.zip',\n",
    "    f'https://storage.googleapis.com/download.tensorflow.org/models/{model_name}.zip',\n",
    "    cache_dir='.', cache_subdir='', extract=True\n",
    ")\n",
    "\n",
    "tokenizers = tf.saved_model.load(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1671396940902,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "oonCfGIYz7sn",
    "outputId": "0069f199-cc1a-4b77-eeb8-1e9f9e388b13"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['detokenize',\n",
       " 'get_reserved_tokens',\n",
       " 'get_vocab_path',\n",
       " 'get_vocab_size',\n",
       " 'lookup',\n",
       " 'tokenize',\n",
       " 'tokenizer',\n",
       " 'vocab']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for demonstration, listing its methods\n",
    "\n",
    "[item for item in dir(tokenizers.en) if not item.startswith('_')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1671396940903,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "KQN749Xg8oeU",
    "outputId": "afc3efde-a67b-47aa-acd8-0bcb0c83fa50"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "and when you improve searchability , you actually take away the one advantage of print , which is serendipity .\n",
      "but what if it were active ?\n",
      "but they did n't test for curiosity .\n"
     ]
    }
   ],
   "source": [
    "# for demonstration, e.g. of decoding, standardizing (although the the inputs\n",
    "# are already standardized) by spliting punctuation, lowercaseing and\n",
    "# unicode-normalizing the input (before tokenizing)\n",
    "for en in en_examples.numpy():\n",
    "    print(en.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 617,
     "status": "ok",
     "timestamp": 1671396941517,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "EBiyBBk78ogj",
    "outputId": "43857536-e6a3-4cbe-cfd7-675b16891f11"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> This is a padded-batch of token IDs:\n",
      "[2, 72, 117, 79, 1259, 1491, 2362, 13, 79, 150, 184, 311, 71, 103, 2308, 74, 2679, 13, 148, 80, 55, 4840, 1434, 2423, 540, 15, 3]\n",
      "[2, 87, 90, 107, 76, 129, 1852, 30, 3]\n",
      "[2, 87, 83, 149, 50, 9, 56, 664, 85, 2512, 15, 3]\n"
     ]
    }
   ],
   "source": [
    "# for demonstration, encoding\n",
    "encoded = tokenizers.en.tokenize(en_examples)\n",
    "\n",
    "print('> This is a padded-batch of token IDs:')\n",
    "for row in encoded.to_list():\n",
    "  print(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 15,
     "status": "ok",
     "timestamp": 1671396941518,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "44Kj7RYZ8oin",
    "outputId": "31e21f4a-3e0c-48de-b97c-b6c994cc4d8f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "and when you improve searchability , you actually take away the one advantage of print , which is serendipity .\n",
      "but what if it were active ?\n",
      "but they did n ' t test for curiosity .\n"
     ]
    }
   ],
   "source": [
    "# for demonstration, detokenizing\n",
    "round_trip = tokenizers.en.detokenize(encoded)\n",
    "\n",
    "for line in round_trip.numpy():\n",
    "  print(line.decode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 9,
     "status": "ok",
     "timestamp": 1671396941518,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "MRGGKwYn8oky",
    "outputId": "6b8893e4-61e0-4c8a-c65e-d504d0279821"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[b'[START]', b'and', b'when', b'you', b'improve', b'search', b'##ability',\n",
       "  b',', b'you', b'actually', b'take', b'away', b'the', b'one', b'advantage',\n",
       "  b'of', b'print', b',', b'which', b'is', b's', b'##ere', b'##nd', b'##ip',\n",
       "  b'##ity', b'.', b'[END]']                                                 ,\n",
       " [b'[START]', b'but', b'what', b'if', b'it', b'were', b'active', b'?',\n",
       "  b'[END]']                                                           ,\n",
       " [b'[START]', b'but', b'they', b'did', b'n', b\"'\", b't', b'test', b'for',\n",
       "  b'curiosity', b'.', b'[END]']                                          ]>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for demonstration, converting token IDs to token text to demonstrate subwords (as subword tokenizer is used)\n",
    "\n",
    "tokens = tokenizers.en.lookup(encoded)\n",
    "tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 301
    },
    "executionInfo": {
     "elapsed": 6245,
     "status": "ok",
     "timestamp": 1671396947759,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "k0BOFIG78onF",
    "outputId": "ff848e67-cb11-41e7-d982-a187621914b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "..................................................."
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAELCAYAAAAspXpuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAY70lEQVR4nO3dfdRdVWHn8e8PkhLNixATEalNBgoGgxNdYHEttVJREZDKMmNVUEEqVFmM02FZjS7QLMQB68yy46goCEbefKuAUpROXQVHcYoTZgSbMTClElGJBgwhCW+Ke/44+6Ent89OHvK85Xny/ax1Vu45++xz977nOfd3zz7n3qSUgiRJw9ljshsgSdp1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJDQiSbYkOWCy2zFaSVYlOW+y27E7S3JTkrdPdjs0MobENJDk7iSPJVkwsPz/JClJFo/2OUopc0op/zza7Yw13/R3X0kWJLk5yf1JHkjyP5O8uFd+cpJbkzyY5KdJ/jLJjF75/CTXJNmaZF2SEyenJ7s2Q2L6+DHwpqGZJM8Dnjp5zdF4Smd3P363AKcCC4F9gI8A1/WC4KnAnwMLgCOAo4B39+p/EngM2Bc4CbgwydKJafrUsbv/kU0nlwNv7c2fDFzWXyHJcfXs4sEk9yRZ2St7Q5IfJ5lX549Jsj7Jwjpfkvx+fbwqyaeSfLMOQ92c5JlJ/irJxiRrk7ygt+0n6vbqn1cfH1k/5b0nyS+T3JvkhCTHJrkzya+SvH+4Dic5ne7gfk9tx3V1+SF1SOOBJGuS/HGj/twkNyb5eH3TXZLk7+pz3pHkTwba/Mkk1yfZnOSWJAfWsiT5WG3/g0l+mOTQxnPelOT8JN+v634tyfxe+YuSfK+2/bYkRw7U/XCSm4GHgH81/JfkWUm+mmRD3Z/vqsvn19f5+Do/J8k/JXnrCP42Ftd9+LZatjHJO5K8MMntta2f6K1/Sv2b+ESSTfXv4ajhXo+6/qlJflS3+7dJFrXW7SulPFJKuaOU8lsgwON0YTG/ll9YSvlOKeWxUsrPgCuBF9fnnA0sB84ppWwppXwX+DrwlpE8926llOI0xSfgbuAVwB3AIcCewE+BRUABFtf1jgSeR/fh4N8CvwBO6G3nSmAV8HTg58BremUF+P36eBVwH3AYMAv4e7ozmbfW5z4PuHG4ur365/Xa9BvgA8BM4DRgA3AVMBdYCjwM/JtG35/YVp2fCfwT8H7gd4CXA5uB5/TXr338fq8ds4F7gLcBM4AX1D4+t1fvfuAPavmVwBdr2dHArcDedG9WhwD7Ndp7E/Az4ND6nF8Frqhl+9fnOLbuo1fW+YW9uj+pr8kMYObAtveo7fhA7fsBwD8DR9fyVwHrgWcAFwN/3at7JI2/DWBx3Yefrvv7VcAjwLV1W/sDvwReVtc/pe7T/1j3xxuATcD8Xj/eXh+/tu6vQ2qfzga+12vX3wArdvD3fzvdGUEBLt7OetcCF9THLwAeGih/N3DdZB/Pu9o06Q1wGoOd+C8hcTZwPvBq4O/qQfdESAxT76+Aj/Xm965vQj8EPjOw7mBIXNwr+/fAj3rzzwMeGK5ur34/JB4G9qzzc+v6R/TWv5VemA2064lt1fmX1jfCPXrLvgCs7K1/KfCPwF/01nkD8J2BbX8G+GCv3md7ZccCa+vjlwN3Ai/qP2+jvTcNvVHV+efWN7g9gfcClw+s/7fAyb26525n20cAPxlY9j7gc735/1b378+Ap29nW0/8bfAvIbF/r/x+4A29+a8Cf14fn0L3ISO98u8Db+n1Yygkvgn8aW+9PejOkhY9yWNgFt1w68mN8lPpPjgt6P+dDKxzGnDTWB6b02FyuGl6uRw4ke4gvWywMMkRdXhlQ5JNwDvoxmsBKKU8AHyF7lPuf9nBc/2i9/jhYebnPIl2319KebxXd7jtj3R7zwLuKd0QxJB1dJ92hxwHPIXuk/GQRcARdejkgSQP0A1lPbO3zvre44eG2lRK+XvgE3Rj3L9MctHQsF3DPQNtm0m3HxYBrx9ow0uA/Rp1By0CnjVQ//10Y+5DLqLbv6tKKfcPLdzR30b1ZPb5z0p95+3181mNNv/XXnt/RXc2tv8w6zaVbujpC8CKJMv6ZUlOoPvwdEwp5b66eAswuI/m0Z11qseQmEZKKevohn2OBa4eZpWr6MZdn11KeRrdm2SGCpM8n+4T1xeAj49h0x5i24voz2ytuBMGf8b458Czs+1F3d+j++Q85GLgBuAbdWwaujffb5dS9u5Nc0op7xxRI0r5eCnlMLozg4OBv9jO6s8eaNuv6Ya27qE7k+i3YXYp5YLt9LfvHuDHA/XnllKOBUiyJ11IXAac0b9OxA7+NnbC/kn69X+Pbt8M1+Y/G2jzU0op39vJ551J71pNklfT7e/jSyk/7K13JzAjyUG9ZcuANTv5vNOWITH9/Cnw8lLK1mHK5gK/KqU8kuQP6M46AEgyC7iC7pPn2+gO8jPGqE0/AE5Msmc9aF82RtuF7tNs/wLuLXSh9J4kM+uF3+OBLw7UO5PuGs51SZ5CN/Z9cJK31Hoz64XZQ3bUgLreEUlmAlvpxut/u50qb07y3CRPBc6luzbwON3rf3ySo+trNSvdhf3fHckLQTeksznJe5M8pW7j0CQvrOXvpwuZU4GPApfV4IDt/G3spGcA76qv4+vprjl8Y5j1Pg28L/WuoiRPq+vvUL3I/5Ikv1P7+166s6ZbavnL6a4dLS+lfL9ftx4fVwPnJpmd7tbZ19KdjavHkJhmSil3lVJWN4rPoDsoNtNd3Pxyr+x8umGaC0spjwJvBs4b+KS1s/4D3Rv10BDOtWOwzSGXAM+twxXXllIeq891DN2n808Bby2lrO1XqkMhp9ONU3+N7tP8q4A30n3iXU93S+VeI2jDPLpPqxvphlXup3sTbrmc7hrHerqx9HfVNt1D90b1frqL9/fQnZGM6DitQfMa4Pl0Z5T3AZ8FnpbkMOAsutfi8dq3Aqyo1bf3t7EzbgEOqm34MPDv+sNbvTZfU9vyxSQP0l0rOmaoPN0ddMPe3Ua3bz5J93r/jO4M+rhSytAZyznA0+jOGLfU6Zu9+mfQDTv+ku7s+Z2lFM8kBmTbYUNJ4ynJTXR3M312stsyXpKcQndh+iWT3RaNnmcSkqQmQ0KS1ORwkySpyTMJSVLTjB2vMrUsWLCgLF68eLKbIU1d9/2/7t8FY3Fjm6aKW2+99b5SysLB5dMuJBYvXszq1a07QCXt0OeO6/592/WT2w5NqCTrhlvucJMkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKlp2n3jejwsXvEv3zy9+4LjJrElkjSxPJOQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlq2mFIJNkrySVJ1iXZnOQHSY7plR+VZG2Sh5LcmGTRQN1LkzyYZH2Sswa2vdN1JUnjbyRnEjOAe4CXAU8Dzga+nGRxkgXA1cA5wHxgNfClXt2VwEHAIuCPgPckeTXAaOpKkibGjB2tUErZSveGPeRvkvwYOAx4OrCmlPIVgCQrgfuSLCmlrAVOBk4ppWwENia5GDgFuAF43SjqSpImwJO+JpFkX+BgYA2wFLhtqKwGyl3A0iT7APv1y+vjpfXxaOoOtun0JKuTrN6wYcOT7ZIkqeFJhUSSmcCVwOfrp/05wKaB1TYBc2sZA+VDZYyy7jZKKReVUg4vpRy+cOHCkXdIkrRdIw6JJHsAlwOPAWfWxVuAeQOrzgM21zIGyofKRltXkjQBRhQSSQJcAuwLLC+l/LoWrQGW9dabDRxId61hI3Bvv7w+XjMGdSVJE2CkZxIXAocAx5dSHu4tvwY4NMnyJLOADwC316EogMuAs5Psk2QJcBqwagzqSpImwEi+J7EI+DPg+cD6JFvqdFIpZQOwHPgwsBE4Anhjr/oH6S5GrwO+DXy0lHIDwGjqSpImxkhugV0HZDvl3wKWNMoeBU6t05jWlSSNP3+WQ5LUtMMzCW1r8Yrrn3h89wXHTWJLJGn8eSYhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqGlFIJDkzyeokjyZZ1Vu+OElJsqU3ndMr3yvJpUkeTLI+yVkD2z0qydokDyW5McmikdaVJI2/GSNc7+fAecDRwFOGKd+7lPKbYZavBA4CFgHPBG5M8n9LKTckWQBcDbwduA74EPAl4EU7qjvCNkuSRmlEZxKllKtLKdcC9z/J7Z8MfKiUsrGU8iPgYuCUWvY6YE0p5SullEfoQmFZkiUjqCtJmgBjdU1iXZKfJvlcPUMgyT7AfsBtvfVuA5bWx0v7ZaWUrcBdwNIR1N1GktPrcNjqDRs2jFGXJEmjDYn7gBfSDQkdBswFrqxlc+q/m3rrb6rrDJX3y/rlO6q7jVLKRaWUw0sphy9cuHAnuiFJGs5Ir0kMq5SyBVhdZ3+R5Ezg3iRzgS11+Tzgkd7jzfXxljrfN1S+o7qSpAkw1rfAlqHtllI2AvcCy3rly4A19fGaflmS2cCBdNcpdlRXkjQBRnoL7Iwks4A9gT2TzKrLjkjynCR7JHk68HHgplLK0DDRZcDZSfapF6RPA1bVsmuAQ5Msr9v+AHB7KWXtCOpKkibASM8kzgYeBlYAb66PzwYOAG6gGwb6R+BR4E29eh+kuxi9Dvg28NGhW1hLKRuA5cCHgY3AEcAbR1JXkjQxRnRNopSyku4W1eF8YTv1HgVOrdNw5d8CljTKtltXkjT+/FkOSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUNKrfbtrdLV5x/ROP777guElsiSSND88kJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWryy3QN/S/KSdLuyjMJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlqGlFIJDkzyeokjyZZNVB2VJK1SR5KcmOSRb2yvZJcmuTBJOuTnDVWdSVJ42+kZxI/B84DLu0vTLIAuBo4B5gPrAa+1FtlJXAQsAj4I+A9SV492rqSpIkxopAopVxdSrkWuH+g6HXAmlLKV0opj9C9sS9LsqSWnwx8qJSysZTyI+Bi4JQxqCtJmgCjvSaxFLhtaKaUshW4C1iaZB9gv355fbx0DOpKkibAaENiDrBpYNkmYG4tY6B8qGy0dbeR5PR6zWT1hg0bnlQHJEltow2JLcC8gWXzgM21jIHyobLR1t1GKeWiUsrhpZTDFy5c+KQ6IElqG21IrAGWDc0kmQ0cSHetYSNwb7+8Pl4zBnV3OYtXXP/EJEnTxUhvgZ2RZBawJ7BnkllJZgDXAIcmWV7LPwDcXkpZW6teBpydZJ96Qfo0YFUtG01dSdIEGOmZxNnAw8AK4M318dmllA3AcuDDwEbgCOCNvXofpLsYvQ74NvDRUsoNAKOpK0maGDNGslIpZSXdLarDlX0LWNIoexQ4tU5jWleSNP78WQ5JUpMhIUlqMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlSkyEhSWoyJCRJTYaEJKnJkJAkNc2Y7AZMR4tXXP/E47svOG4SWyJJo+OZhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJajIkJElNhoQkqcmQkCQ1GRKSpCZDQpLUZEhIkpoMCUlS05iERJKbkjySZEud7uiVnZhkXZKtSa5NMr9XNj/JNbVsXZITB7bbrCtJGn9jeSZxZillTp2eA5BkKfAZ4C3AvsBDwKd6dT4JPFbLTgIurHVGUleSNM7G+z8dOgm4rpTyPwCSnAP8KMlc4LfAcuDQUsoW4LtJvk4XCiu2V7eUsnmc2z1m/A+IJE1lY3kmcX6S+5LcnOTIumwpcNvQCqWUu+jOHA6u029KKXf2tnFbrbOjuttIcnqS1UlWb9iwYQy7JEm7t7EKifcCBwD7AxcB1yU5EJgDbBpYdxMwt5Y92ChjB3W3UUq5qJRyeCnl8IULF46mH5KknjEZbiql3NKb/XySNwHHAluAeQOrzwM20w03tcrYQV1J0gQYr1tgCxBgDbBsaGGSA4C9gDvrNCPJQb16y2oddlBXkjQBRh0SSfZOcnSSWUlmJDkJ+EPgBuBK4PgkL00yGzgXuLqUsrmUshW4Gjg3yewkLwZeC1xeN92sO9o2S5JGZiyGm2YC5wFLgMeBtcAJQxekk7yD7g3/6cC3gLf16p4BXAr8ErgfeGcpZQ1AKWXNDupKksbZqEOilLIBeOF2yq8CrmqU/Qo4YWfqSpLGnz/LIUlqMiQkSU2GhCSpyZCQJDUZEpKkpvH+gT/1+GN/kqYazyQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmgwJSVKTISFJavLLdJPEL9ZJmgo8k5AkNRkSkqQmQ0KS1GRISJKaDAlJUpMhIUlq8hbYXUD/dljwllhJuw7PJCRJTYaEJKnJkJAkNRkSkqQmL1zvgvxdJ0m7Cs8kJElNhoQkqcmQkCQ1eU1iF+f1CUmTyTMJSVKTZxJTiGcVkiaaIdEz+BtKuzIDQ9JEMCSmAQND0njZpUMiyXzgEuBVwH3A+0opV01uq3ZtBoaksbRLhwTwSeAxYF/g+cD1SW4rpayZ3GZNDa3hM8ND0kjtsiGRZDawHDi0lLIF+G6SrwNvAVZMauOmuLG69mLYSNPfLhsSwMHAb0opd/aW3Qa8bHDFJKcDp9fZLUnu2MnnXEA3rLU72ek+5yNj3JKJ434eiVMzPi2ZOLvbfh5tfxcNt3BXDok5wIMDyzYBcwdXLKVcBFw02idMsrqUcvhotzOV2Ofdg32e/sarv7vyl+m2APMGls0DNk9CWyRpt7Qrh8SdwIwkB/WWLQO8aC1JE2SXDYlSylbgauDcJLOTvBh4LXD5OD7tqIespiD7vHuwz9PfuPQ3pZTx2O6YqN+TuBR4JXA/sMLvSUjSxNmlQ0KSNLl22eEmSdLkMyQkSU2GBN21jyTXJNmaZF2SEye7TaOV5Mwkq5M8mmTVQNlRSdYmeSjJjUkW9cr2SnJpkgeTrE9y1oQ3fifUdl9S99/mJD9IckyvfNr1GSDJFUnurW2/M8nbe2XTss9DkhyU5JEkV/SWnVj/BrYmubZe1xwqm7LHeZKbal+31OmOXtn49rmUsttPwBeAL9F9ge8ldF/aWzrZ7Rpln14HnABcCKzqLV9Q+/d6YBbwUeAfeuXnA98B9gEOAdYDr57s/oygv7OBlcBiug8/r6H7Ts3i6drn2valwF718ZLa9sOmc597ffjvtQ9X9F6LzcAf1mP5KuCLvfWn7HEO3AS8vbH/x7XPk975yZ7qm8tjwMG9ZZcDF0x228aof+cNhMTpwPcG+v8wsKTO/xx4Va/8Q/0/uqk0AbfT/f7XbtFn4DnAvcCfTPc+A28Evkz3wWAoJP4TcFVvnQPrsT13qh/n2wmJce+zw03t34haOkntGW9L6foHPPF9lLuApUn2AfbrlzNFX4sk+9Lt2zVM8z4n+VSSh4C1dCHxDaZxn5PMA84FBofIBvt8F/VNkulxnJ+f5L4kNyc5si4b9z4bEk/iN6KmiTl0/esb6u+c3vxg2ZSRZCZwJfD5UspapnmfSyln0LX3pXRfQH2U6d3nDwGXlFJ+OrB8R32eysf5e4EDgP3pvjR3XZIDmYA+GxK7329Eba+/W3rzg2VTQpI96E6pHwPOrIundZ8BSimPl1K+C/wu8E6maZ+TPB94BfCxYYp31Ocpe5yXUm4ppWwupTxaSvk8cDNwLBPQZ0Ni9/uNqDV0/QOe+H87DgTWlFI20g1XLOutP2VeiySh+58M9wWWl1J+XYumbZ+HMYPaN6Znn4+kuxnhJ0nWA+8Glif53/zrPh8A7EV3jE+347wAYSL6PNkXZHaFCfgi3V0As4EXM4XuethOn2bQ3dVyPt0n61l12cLav+V12UfY9q6XC4Bv0931soTuzWRK3PUCfBr4B2DOwPJp2WfgGXQXcOcAewJHA1uBP57GfX4q8Mze9J+Bv679XUo3vPLSeixfwbZ3+kzJ4xzYu+7boWP4pLqfD56IPk/6C7ArTMB84Nr6wv8EOHGy2zQGfVpJ92mjP62sZa+gu8j5MN1dE4t79fai+72sB4FfAGdNdl9G2N9FtY+P0J1mD00nTeM+L6xv9A/Utv8QOK1XPu36PMxrsJJ6d1OdP7Eew1uBrwHze2VT8jiv+/l/0Q0TPUD3QeiVE9Vnf7tJktTkNQlJUpMhIUlqMiQkSU2GhCSpyZCQJDUZEpKkJkNCktRkSEiSmv4/R6CYI9Qv/VoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for demonstration, the distribution of tokens per example\n",
    "\n",
    "lengths = []\n",
    "\n",
    "for pt_examples, en_examples in train_examples.batch(1024):\n",
    "    pt_tokens = tokenizers.pt.tokenize(pt_examples)\n",
    "    lengths.append(pt_tokens.row_lengths())\n",
    "    \n",
    "    en_tokens = tokenizers.en.tokenize(en_examples)\n",
    "    lengths.append(en_tokens.row_lengths())\n",
    "    print('.', end='', flush=True)\n",
    "\n",
    "\n",
    "all_lengths = np.concatenate(lengths)\n",
    "\n",
    "plt.hist(all_lengths, np.linspace(0, 500, 101))\n",
    "plt.ylim(plt.ylim())\n",
    "max_length = max(all_lengths)\n",
    "plt.plot([max_length, max_length], plt.ylim())\n",
    "plt.title(f'Maximum tokens per example: {max_length}');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "executionInfo": {
     "elapsed": 23,
     "status": "ok",
     "timestamp": 1671396947761,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "1yKH1MPPAXpS"
   },
   "outputs": [],
   "source": [
    "# preprocessing by tf.data\n",
    "\n",
    "MAX_TOKENS=128 # i.e., one token before the [END] token in each sequence.\n",
    "def prepare_batch(pt, en):\n",
    "    pt = tokenizers.pt.tokenize(pt)      # Output is ragged.\n",
    "    pt = pt[:, :MAX_TOKENS]    # Trim to MAX_TOKENS.\n",
    "    pt = pt.to_tensor()  # Convert to 0-padded dense Tensor\n",
    "\n",
    "    en = tokenizers.en.tokenize(en)\n",
    "    en = en[:, :(MAX_TOKENS+1)]\n",
    "    en_inputs = en[:, :-1].to_tensor()  # Drop the [END] tokens\n",
    "    en_labels = en[:, 1:].to_tensor()   # Drop the [START] tokens\n",
    "\n",
    "    return (pt, en_inputs), en_labels\n",
    "\n",
    "\n",
    "BUFFER_SIZE = 20000\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "def make_batches(ds):\n",
    "    return (\n",
    "        ds\n",
    "        .shuffle(BUFFER_SIZE)\n",
    "        .batch(BATCH_SIZE)\n",
    "        .map(prepare_batch, tf.data.AUTOTUNE)\n",
    "        .prefetch(buffer_size=tf.data.AUTOTUNE))\n",
    "\n",
    "\n",
    "train_batches = make_batches(train_examples)\n",
    "val_batches = make_batches(val_examples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "executionInfo": {
     "elapsed": 21,
     "status": "ok",
     "timestamp": 1671396947762,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "EjOKaJI2AXrw"
   },
   "outputs": [],
   "source": [
    "# setting the hyperparameters\n",
    "num_layers = 4\n",
    "d_model = 128\n",
    "dff = 512\n",
    "num_heads = 8\n",
    "dropout_rate = 0.1\n",
    "\n",
    "# creating the Keras model\n",
    "transformer = Transformer(\n",
    "    num_layers=num_layers,\n",
    "    d_model=d_model,\n",
    "    num_heads=num_heads,\n",
    "    dff=dff,\n",
    "    input_vocab_size=tokenizers.pt.get_vocab_size().numpy(),\n",
    "    target_vocab_size=tokenizers.en.get_vocab_size().numpy(),\n",
    "    dropout_rate=dropout_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "86YFvnTjPdHk"
   },
   "source": [
    "- The original Transformer uses the following learning-rate scheduler that first linearly increase the the learning rate, and then gradually decrease it:\n",
    "\n",
    "\\begin{equation}\n",
    "lrate = d_{model}^{-0.5} * \\min(step{\\_}num^{-0.5}, step{\\_}num \\cdot warmup{\\_}steps^{-1.5})\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "executionInfo": {
     "elapsed": 727,
     "status": "ok",
     "timestamp": 1671396948469,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "pKjtKYFwz7sn",
    "outputId": "df94865a-8519-43de-eda1-75875ddff5c0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Train Step')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaMAAAENCAYAAACigwpqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9bn48c+ThYRskI2whk0WAVk0iiIqSmu11YraTbwuVYvV68/e6q3tvb12sZu211trtSptqVu11q1o3avigmsQBYKAZISwZ8ISmAkQkjy/P84ZGMbZksyS5Xm/XvNK5nu+58wzI86T7znf83xFVTHGGGPSKSPdARhjjDGWjIwxxqSdJSNjjDFpZ8nIGGNM2lkyMsYYk3ZZ6Q6guygrK9MRI0akOwxjjOlWlixZ0qCq5bH6WTKK04gRI6iurk53GMYY062IyPp4+tlpOmOMMWlnycgYY0zaWTIyxhiTdpaMjDHGpJ0lI2OMMWmX0mQkIiUi8qSI+EVkvYjMjdBPROQWEdnuPm4REQnaPlVElohIk/tzatC2U0XkVRFpFJF1UWI5RURURH6e0DdpjDGm3VI9MroTaAYqgAuBu0RkYph+84A5wBRgMnA2cCWAiPQBFgIPAsXAfcBCtx3ADywAvhcpCBHJBn4HvNv5t2SMMaazUpaMRCQfOB+4UVV9qvom8BRwUZjulwC3qupGVd0E3Apc6m6bhXN/1G2qul9VbwcEOA1AVd9T1QcAT5RwrgdeBFZ1+o11UY1NB3h8yUZsiRBjTHeQypHRWKBFVdcEtX0EhBsZTXS3hes3EVimh3/LLotwnM8QkeHAZcBNcfSdJyLVIlLt9XrjOXyXcfPzq7j+0Y94u3Z7ukMxxpiYUpmMCoDdIW2NQGGEvo0h/Qrc60ah26IdJ5zbcUdnsTqq6nxVrVLVqvLymNUsupTd+w4A8Mqq+jRHYowxsaUyGfmAopC2ImBPHH2LAJ87GmrPcQ4jImcDhar6SLxBd1fePfsBeG7FVjtVZ4zp8lKZjNYAWSIyJqhtClATpm+Nuy1cvxpgcvDsOpxJDuGOE2o2UCUiW0VkK/B14D9EZGGc76Hb8Hh9iMCmXXtZvil0IGmMMV1LypKRqvqBJ4CbRCRfRE4EzgEeCNP9fuA6ERkiIoNxJhzc625bBLQC14pIjohc47a/AiAiGSKSC2Q7TyU3aKbdjTjXrqa6j6eAPwLfTOibTbPGpgM0+Jq58uTRZGYIz63Ymu6QjDEmqlRP7b4a6AvUAw8DV6lqjYicJCLB13DuAZ4GlgMrgGfcNlS1GWfa98XALpzJCHPcdoCTgb3As0Cl+/uL7r57VHVr4OFu86vqjiS+55SrbXA+ymOGFzNjdCnPLd9ip+qMMV1aSpeQcL/054RpfwNnYkLguQI3uI9wx1kKHBNh2yKcqd7xxHNpPP26G4/XD8Co8nzOmDSQHz65glVb93DkoNBLbcYY0zVYOaAeyOP1kZUhVJbk8YWJA8nMEJ7+aHO6wzLGmIgsGfVAtV4flaV5ZGdmUFaQw8wjylj44Wba2uxUnTGma7Jk1AN5vH5Glx8868l5Rw9h0669vL+uR10aM8b0IJaMepiW1jbWbfczqjz/YNvnJ1SQ1yeTJ5duSmNkxhgTmSWjHmbjzr0caFVGlx0aGeX1yeKMSQN5ZvkW9h1oTWN0xhgTniWjHsbjTusePSD/sPZzpw1hz74WKw9kjOmSLBn1MLX17rTuoJERwIzRZVQU5fDEBxvTEZYxxkRlyaiH8TT4KMnvQ3F+n8PaMzOE844eyiur6tnauC9N0RljTHiWjHqY2no/o8ryw2674NhK2hQeeX9DiqMyxpjoLBn1MJ4G32Ez6YJVluZx0pgyHnm/jla758gY04VYMupBAgVSg+8xCjX3uEo2N+7jtTU2kcEY03VYMupBAgVSR0VJRp+bUEFZQQ4PvVuXqrCMMSYmS0Y9SKBA6ugIp+kAsjMz+FqVM5Fh8669qQrNGGOismTUg9S6BVKHleRF7Td3eiUA9729LvlBGWNMHCwZ9SAer4/hboHUaIYW53HGpIE8/G4d/v0tKYrOGGMis2TUg9R6/VGvFwW7fOYodu9r4bEldhOsMSb9LBn1EC2tbawPKZAazTHDi5k6rD9/WfypLS1hjEk7S0Y9xMECqXGOjAAunzmSddubeNnq1Rlj0iylyUhESkTkSRHxi8h6EZkboZ+IyC0ist193CIiErR9qogsEZEm9+fUoG2nisirItIoIutCjjtARB4Wkc3u9sUiMj1pbziFar1ugdQ4R0YAZ04ayOB+ucx/vTZZYRljTFxSPTK6E2gGKoALgbtEZGKYfvOAOcAUYDJwNnAlgIj0ARYCDwLFwH3AQrcdwA8sAL4X5rgFwPvAMUCJu+8zIhL/cKKLCkzrDi2QGk1WZgbzTh7F++t28o5ne7JCM8aYmFKWjEQkHzgfuFFVfar6JvAUcFGY7pcAt6rqRlXdBNwKXOpumwVkAbep6n5VvR0Q4DQAVX1PVR8APKEHVVWPqv6fqm5R1VZVnQ/0AcYl8r2mQ603fIHUWL5xXCVlBTnc8craJEVmjDGxpXJkNBZoUdU1QW0fAeFGRhPdbeH6TQSWqWrwVfdlEY4TlXt6rw8Q9ptYROaJSLWIVHu93vYePqU83sgFUqPJzc5k3skjeXNtAx/U7UxCZMYYE1sqk1EBsDukrREojNC3MaRfgXvdKHRbtONEJCJFwAPAT1U19HgAqOp8Va1S1ary8vL2HD7lar2+dk1eCHbh9OEU52Xz+5c/SXBUxhgTn1QmIx9QFNJWBOyJo28R4HNHQ+05Tlgi0hd4GnhHVX8V735dVWPTAbb7m+Oe1h0qPyeLy2eO5NXVXpZvDJuXjTEmqVKZjNYAWSIyJqhtClATpm+Nuy1cvxpgcvDsOpxJDuGO8xkikgP8A9iIOymiuwsUSO3oyAjg4hkjKM7L5tcvrEpUWMYYE7eUJSNV9QNPADeJSL6InAicg3OqLNT9wHUiMkREBgPXA/e62xYBrcC1IpIjIte47a8AiEiGiOQC2c5TyQ3MtBORbOAxYC9wiaq2JeGtplxtfaBad8dGRgBFudn8+6lH8MYnDby1tiFRoRljTFxSPbX7aqAvUA88DFylqjUicpKI+IL63YNzGm05sAJ4xm1DVZtxpn1fDOwCLgPmuO0AJ+Mkm2eBSvf3F91tM4CzgNOBXSLicx8nJen9poSnwU92ZuwCqbH82/HDGdwvl1ueX8Xh80OMMSa5slL5Yqq6AyeRhLa/gTMxIfBcgRvcR7jjLMW5VyjctkU4U73DbXst0rburLbeR2VJ7AKpseRmZ/Ldz4/le48t47kVW/niUYMSFKExxkRn5YB6AE9D/AVSYznv6KGMrSjgf19YzYHWHnEW0xjTDVgy6uYCBVI7M3khWGaG8IMzx+Np8HPfW+sSckxjjInFklE3t8EtkNqZyQuhTh03gFnjyvndvz7Bu2d/wo5rjDGRWDLq5jzezk/rDiUi/OisCexraeXXz9tUb2NM8lky6uY6Uq07HqPKC7hs5kgeXbKRDzfsSuixjTEmlCWjbs7j9VOS34f+ee0rkBqP/3faGAYU5vCjhStotQX4jDFJZMmom3Nq0iV2VBRQkJPFD790JMs2NtpkBmNMUlky6uacat3JW47py1MGc+q4cn7zwmo27GhK2usYY3o3S0bd2K6mZrb7mxk9IDkjI3AmM/z83KPIEPjhP1ZYZQZjTFJYMurGajuwumtHDOnflxvOGM/ra7w8uXRTUl/LGNM7WTLqxg5O6x6Q/FXTLzp+OMcML+anT69ka+O+pL+eMaZ3sWTUjdV63QKpxX2T/loZGcJvvjKZ5pY2vvfYR7TZ7DpjTAJZMurGPF6nQGpWJwukxmtUeQE3njWBNz5p4F6bXWeMSSBLRt2YpyFxNenidcFxw/jckQO4+flVrN4a9+K6xhgTlSWjbipQIDVR1brjJSLcfP5kinKz+M7flrLvQGtKX98Y0zNZMuqmAgVSk3XDazRlBTn85itTWLV1Dz95Kq7V3o0xJipLRt3UoaXGUzsyCjh1/AD+/dTR/O39DTxavSEtMRhjeg5LRt2UpyE5BVLb47ufG8sJo0r5n3+s4OMtu9MWhzGm+4s7GYnImSLyTxFZKSLD3LYrRGR2O45RIiJPiohfRNaLyNwI/UREbhGR7e7jFhGRoO1TRWSJiDS5P6cGbTtVRF4VkUYRWRfm2CPc7U0iskpEPhdv/F1Jbb2f0iQVSI1XVmYGt18wjX59s7nqwSXs3ncgbbEYY7q3uJKRiFwI/B34BBgJZLubMoEb2vF6dwLNQAVwIXCXiEwM028eMAeYAkwGzgaudGPpAywEHgSKgfuAhW47gB9YAHwvQgwPA0uBUuCHwGMiUt6O99AleBp8CV1Qr6PKC3O488Kj2bhzL9c8tJQWW6rcGNMB8Y6MbgC+parfBVqC2t8Bpobf5XAikg+cD9yoqj5VfRN4CrgoTPdLgFtVdaOqbgJuBS51t80CsoDbVHW/qt4OCHAagKq+p6oPAJ4wMYwFjgZ+rKp7VfVxYLkbV7fi8aZ+Wnckx44o4edzJvH6Gi+/ePbjdIdjjOmG4k1GY4C3w7T7gKI4jzEWaFHVNUFtHwHhRkYT3W3h+k0ElunhFTuXRThOuON6VDX4BplIMSAi80SkWkSqvV5vHIdPjUCB1K4wMgr4xnGVXD5zJH9ZvI6/vrs+3eEYY7qZeJPRZpxkEupkoDbOYxQAoVe5G4HCCH0bQ/oVuNeNQrdFO06s40bdV1Xnq2qVqlaVl3edM3mBAqldZWQU8N9fPJJTx5Xz44U1LF7bkO5wjDHdSLzJaD5wu4ic6D4fJiKXAL8G7orzGOFGUUVAuNv4Q/sWAT53NNSe43Qmhi4rsNR4uqZ1R5KZIdx+wTRGlxdw5QNLWLEpNO8bY0x4cSUjVf018ATwEpAPvArcDdytqnfG+VprgCwRGRPUNgUId9dkjbstXL8aYHLw7DqcSQ7x3H1ZA4wSkeCRUKQYuixPCguktldhbjb3XnYsRblZXPqX91m/3Z/ukIwx3UDcU7tV9YdAGXAccDxQrqo3tmN/P05Cu0lE8t1R1jnAA2G63w9cJyJDRGQwcD1wr7ttEdAKXCsiOSJyjdv+CoCIZIhILs6MPxGR3MBMO/d61YfAj932c3ES2ePxvo+uwOP1Mbw0P2UFUttrUL++3H/5dFrb2rjoz+9Rv8eWnDDGRBfv1O4FIlKoqk2qWu3OWPO5SWVBO17vaqAvUI8zxfoqVa0RkZNExBfU7x7gaZyZbiuAZ9w2VLUZZ9r3xcAu4DJgjtsOznWsvcCzQKX7+4tBx/4GUAXsBG4GvqKqXWd2QhxqvT5GlXWdyQvhHDGggAWXHot3z34uXfA+jXvtHiRjTGQSzzLSItIKDFLV+pD2MmCrqmYlKb4uo6qqSqurq9MdBgda25jwo+e54qRRfP+M8ekOJ6bX1ni54r73mTCoiAeumE5RbnbsnYwxPYaILFHVqlj9oo6M3IoJpTj38RS7zwOPcuAsYFtiQjbx2LCjiQOt2uVHRgGnjC3nrguPYeWW3Vz85/fYY1UajDFhxDpN14BzSk2BlYA36LEV+BPwh2QGaA7ncad1d7WZdNF8bkIFd849mhWbGrlkgSUkY8xnxUpGpwKzcUZGX8GpchB4zAQqVfUXSY3QHCYwrTudBVI74vSJA7lj7tEs2+gkpMYmS0jGmEOiJiNVfU1VF+HUo1voPg883lbVzSmJ0hzk8aa/QGpHnTFpIHfMncbyTY18ff7bNsvOGHNQvPcZrVfVNhEZLCLHi8jJwY9kB2kO8TT4ulzlhfY4Y9IgFlx6LHU7mvjq3W+zYUdTukMyxnQB8U7tHiwii4CNwGKce31eDXqYFKn1+rtUTbqOOGlMOQ9eMZ1dTQc4/663WL21WxXAMMYkQbx3Td6Gc6PpBKAJOAn4KvAxcEZyQjOhdvqb2eFv7tYjo4CjK4v5+5UnAPCVu9+yWnbG9HLxJqNTgO+r6iqcmXVeVX0C+D7ws2QFZw4XWN21u4+MAsYNLOSJq2cwqF8ulyx4j7+9V5fukIwxaRJvMuqLM80bYAcwwP19JU45HZMCtd1wWncsQ4vzeOyqGcw4oowfPLGcXz33MW1tsW/ENsb0LPEmo1VA4Hb/D4Fvi8hw4N+BTckIzHxWVy6Q2hlFudksuKSKfzu+knte83DVX5fg298Se0djTI8RbzL6HTDQ/f0m4HSclVSvBv47CXGZMGq7eIHUzsjKzOBn50zixrMm8NLKbcy5c/HBe6qMMT1fvFO7/6qq97q/fwCMAI7Fuen10aRFZw7j8fq63c2u7SEiXD5zJA9ePp2d/mbOuWMxz6/Yku6wjDEp0KE/sd3q3R8AfhH5QYJjMmEcaG1j/famHnW9KJIZR5Tx9P+byegBBXz7wQ+4+blVtLS2pTssY0wSxUxGIlImIl8SkdNFJNNtyxaR/wDWAf+Z5BgNToHUlrbuUyC1swb378vfrzyeudMrufu1Wi744zts2rU33WEZY5IkVtXuGcAnOGsLPQcsFpHxwDLgGpxp3ZXJDtIcmkk3ekDPHxkF5GRl8stzj+K3X5/Cys27OfO213l2uZ22M6YnijUy+hnwAs707dtwVnn9J/ArYIyq3qGqVs8lBTyBAqllvScZBZw7bSjPfuckRpYXcPVfP+AHjy+jqdlm2xnTk8RKRlOAn6nqCuBGnBte/0tV79d4VuUzCePx+ikr6EO/vN65ON3w0nwe+/YJXD1rNI9Ub+Cs299kyfod6Q7LGJMgsZJRCc7aRbgjoCZgabKDMp/lLDXe+0ZFwbIzM7jhjPH89Yrp7G9p4yt3v80vnlnJvgOt6Q7NGNNJ8cymKw5a8VWBopAVX0uSHKMBPA1+Rg/oHZMXYpkxuowXvnsyF06v5I9vfMoXf/cG1etslGRMdxZPMgqs8FoPFADvc2i11wb3Z1zc5PWkiPhFZL2IzI3QT0TkFhHZ7j5uEREJ2j5VRJaISJP7c2o79j1NRD4Qkd0i4hGRefHGny6BAqm9fWQUrCAni5/POYqHrphOc2sbX73nbX7yVI2tImtMN5UVY/upCX69O4FmoAKYCjwjIh+pak1Iv3nAHJxrVgq8BHwK3C0ifYCFOBMq/gBcCSwUkTGq2hxj32zgSeAGYD5QBbwqIu+q6kcJfq8J09MKpCbSjCPKeOE/TuaW51dx39vreG7FFm48awJfOmoQQX+DGGO6uHhWeo35iOeFRCQfOB+4UVV9qvom8BRwUZjulwC3qupGVd0E3Apc6m6bhZNEb1PV/ap6O86y6KfFsW8JUAQ8oI73cZbBmBDPe0iXg9O6e8ENrx2Rn5PFTedM4smrT6S8MIdrHlrKxQve49MGf7pDM8bEKZVFzsYCLaq6JqjtI2BimL4T3W3h+k0EloXM5lsWsj3svqq6DXgY+KaIZIrICcBw4M1wAYvIPBGpFpFqrzfus5EJV+v1kZ0pDO1hBVITbeqw/iz895n89MsT+bBuF1/47ev834urbRq4Md1AKpNRAbA7pK0RKIzQtzGkX4F77Sd0W+hxou0LTjL6EbAfeAP4oapuCBewqs5X1SpVrSovL4/23pLK4/UzoocWSE20zAzhkhkjePk/T+HMowZy+ytrmfWbRTxavcGWpjCmC0vlt5sP5xRZsCIg3JrToX2LAJ87Gop1nIj7utUj/gZcDPTBGTHdICJfav/bSZ1ar8+uF7XTgMJcfveNaTx+1QkM7t+X7z22jLPveJO3am1FWWO6olQmozVAloiMCWqbAoROXsBtmxKhXw0wWQ6/Oj05ZHukfScBa1T1BVVtU9XVwDPAmR14PylxoLWNul5SIDUZjhlewpNXz+D2C6axq+kAc//4LlfcV80n28L9DWSMSZeUJSNV9QNPADeJSL6InAicAzwQpvv9wHUiMkREBgPXA/e62xYBrcC1IpIjIte47a/Ese9SYIw7vVtEZDRwFs41py6pzi2QapMXOk5E+PKUwbx8/SnccMY43vFs5wu3vc51j3xI3XarZmVMVxBrajcAIrIgwiYF9gFrgUdUdXOMQ10NLMC5Z2k7cJWq1ojIScBzqhr4xr0HGAUsd5//yW1DVZtFZI7bdjPObLg57rTuWPvWishlwO04Excagb+6fbokz8Glxu00XWflZmdy9awj+MaxTiXw+95ax1MfbearVcO4dvYRDOpnE0SMSReJp8SciDwNnAS0ASvc5kk4U6qX4Fx7KQBOUtUPkxNqelVVVWl1dXXKX/ee12r51XOr+OhHp/faunTJsm33Pu58dS0Pv1eHiHDh9ErmnTzKkpIxCSQiS1S1Kla/eE/TLcZZQmKoqp6sqicDQ4FngRdxRhnP4NzTYxKo1uvr1QVSk6miKJebzpnEK9fP4pwpg7n/7fWc/OtX+cHjy1hn9ygZk1Lxjoy2AKep6sch7ROAl1V1kIhMA/6lqqXJCTW90jUy+spdb5GRIfz9yhNS/tq9zYYdTcx/3cMj1RtoaW3jS5MHc/Ws0Rw5KHTypjEmXokeGRUAg8K0D3S3gXMPUVzXoEz8ar0+Rtv1opQYVpLHz+ZM4s3vn8q3Th7FKx9v48zfvcHl977P27XbsVVTjEmeeJPRk8CfReSrIjLCfXwV+DPODDlwFt5bE/EIpt12+JvZ2XTACqSm2IDCXP7rzCN56wezue7zY/mgbicX/PEdvnj7mzxavcGWrDAmCeJNRt/GWfH1QaDWfTwIPI8zQw6cWW3fSnSAvdnB1V1t6Yi06JeXzbWzx/D2f83m5vOOorWtje89toyZt7zCb19aQ/2efekO0ZgeI67Tau7Cet8WkeuB0W5zrXvvUKBPj5xFl04Hp3XbyCitcrMz+cZxlXz92GEsXrudBYs/5Xcvf8Jdi2o5a/IgLjy+kqMri61KuDGd0K5rPG7y6bI3iPY0tQ0++mRmWIHULkJEmDmmjJljyvB4fdz71joeX7KRJ5ZuYlxFIXOnVzJn2hD69bWZj8a0V7yz6XKB7wCzgQGEnN5T1clJia4LScdsuivuq2b9dj8vXXdKSl/XxM+/v4WnPtrMQ+/WsXxTI7nZGZw9eTBzp1cydVh/Gy2ZXi/e2XTxjoz+AJwLPAq8hVN5wSSZp8HH2AHhipqbriI/J4sLjqvkguMqWb6xkYfeq2Phh5t4dMlGxlUUct7RQ5gzbQgVRbnpDtWYLi3ekdEO4Guq+q/kh9Q1pXpkdKC1jSNvfJ55J4/ihjPGp+x1Tef59rew8MNNPLZkI0vrdpEhMHNMOecfPYTTJwykb5/MdIdoTMokemTUBIRd88ckR6BAqlXr7n4KcrK4cPpwLpw+HI/Xx5NLN/HEB5v4zt8+pCAniy8dNYjzjh7CsSNKyMiw03jGQPzJ6Nc4lbC/rXbnX0p4Di41btO6u7NR5QVcf/o4vvu5sbz76Q4e/2Aj/1y2mUeqNzCwKJcvHjWIL00exNGVdn3J9G7xJqPP4xRKPUNEVgIHgjeq6pcTHVhvV+veY2Qjo54hI0M4YXQpJ4wu5aZzJvLSym38c9kWHnxnPQsWf8qQ/n354lEDOWvyYCYP7WeJyfQ68SajBpwqDCZFPF4fZQU5Nk24B8rrk8U5U4dwztQh7N53gH+5ienet9bxxzc+ZVhJX7501GC+MLGCKUP726k80yvEe9PrN5MdiDlcrddvaxj1AkW52Zx39FDOO3oojU0HeGHlVp5ZtoU/veHh7tdqGVCYw+cmVPD5CRXMGF1KTpZNfjA9kxU27aI8Xh9nTBqY7jBMCvXLy+ZrVcP4WtUwdjU18+rqel5auY1/LN3EQ+/Wkd8nk1njBvD5CRWcOm6ALStiepSIyUhElgGnqOpOEVlOlHuLesNNr6kUKJBqS433Xv3z+nDutKGcO20o+w608nbtdl5cuY2XVm7jmeVbyMoQqkYUM2vcAGaNK2dcRaFdZzLdWrSR0ePAfvf3x1IQi3F5Dk5esNN0xqmNd+r4AZw6fgC/mDOJDzfu4qWV23h1VT03P7eKm59bxcCiXE4ZW86sceWcOKaMolwbNZnuJWIyUtWfhvu9M0SkBGfZidNxJkX8l6o+FKafADcDV7hNfwJ+EJhWLiJT3eMciVMt/PJAodY49s0EfgpcBhQCa4FTVXVXIt5jIhya1m0jI3O4jAzh6Mpijq4s5vtnjGdr4z5eW1PPotVenl2+hUeqN5CZIRxTWcwp48o5ZWw5EwYV2SQI0+Wl+prRnUAzUAFMBZ4RkY9UtSak3zxgDjAF5/TgS8CnwN0i0gdYCNyGU6boSmChiIxR1eZo+7rH/ikwAzgBqAMmAl1qLYBab6BAal66QzFd3MB+uXz92Eq+fmwlB1rbWFq3i0Wr63ltjZffvLCa37ywmv552Rw/spQTjyjlhNFljC7Pt1N6psuJtxxQCfALIhdKjbkus4jkAzuBSaq6xm17ANikqj8I6fsWcK+qznefXw58S1WPF5HTgb8AQ4NGO3XAPFV9Psa+xTiVJKaoam3MNx4kleWArrivmrodfl78rhVINR1Xv3sfi2sbeGvtdt6q3c6mXXsBqCjKYcboMmaMLmXGEWUM6W9V4U3yJLoc0J+BacB8YDMdK5Q6FmgJJCLXR0C4b9yJ7rbgfhODti0LqQSxzG1/Psa+RwEtwFdE5Ls4S6X/TlXvDBewiMzDGWlRWVkZ6/0ljMfrY2yFFUg1nTOgKPfgJAhVpW5HE4vXbuet2gZeX+PlyaWbABhRmsf0kaUcO7KE40aUMKykr42cTMrFm4xmA59X1Xc78VoFOF/+wRpxrtuE69sY0q/AvR4Uui30ONH2HQr0w0mMI4ExwMsiskZVXwoNwh1dzQdnZBTrDSbCgdY26nY02bRuk1AiwvDSfIaX5jN3eiWqyuptew6Omp6v2coj1U75yYqiHKpGOInp2BEljBtYSKZdczJJFm8yqgd8nXwtHxB6Oq8I2BNH3yLAp6oqIrGOE23fvW7bTaq6F1gmIn8DvohzbSntAgVSbfKCSSYRYfzAIsYPLOKymSNpa1M+qffx3rodvP/pDt5ft4Nnlm0BoDA3i6rhxRw7soSq4WL88WAAABp9SURBVCUcNaSfVR43CRdvMvohcJOIXKKqHU1Ka4Asd6LBJ27bFCB08gJu2xTgvTD9aoDrRUSCTtVNxpkcEWvfwCq1waOcLlX4tbbepnWb1MvIEMYNLGTcwEIuOn44ABt3NvH+uh289+lO3l+3g1dXrwYgM0M4clAhU4f1Z9qwYqZW9mdkab7N2DOdEm8y+h9gBFAvIuv5bKHUmDe9qqpfRJ7ASWpX4MymOwdnZluo+3GqhD+LkyyuB37vblsEtALXisjdwLfc9ldi7auqtSLyBvBDEbkWGAV8A7ggVvyp4mlwpnVbgVSTbkOL8xhanMe504YCzs3YS+t2srRuFx9u2MU/lm7mwXfqAOjXN5spw/ozbVh/plX2Z+qw/vTP65PO8E03E28yStRNr1cDC3BO+20HrlLVGhE5CXhOVQPfwPfgJIrl7vM/uW2oarOIzHHbbsa5z2iOO6076r6uC3AmZGx347hRVV9O0PvrtNp6K5BquqaS/D7MPrKC2UdWANDaptR6fXxYt4ulG5wk9ftXPqHNPdcwsiyfSUP6cdSQIiYN6cfEwf3s37WJKObUbhHJxpnWfaeqrk9JVF1QqqZ2n3/XW2RmCH+/8oSkv5Yxiebb38Kyjc7I6cO6XazY1MjmxkO38Q0vzWPS4H5ukurHpCFFNoLq4RI2tVtVD4jIVTg3mJokq/X6OHPSoHSHYUyHFORkufcwlR1s2+7bz4rNu1mxqZEVmxr5aOMunlm+5eD2ocV9mTS4H0cN7ceEwUUcObCIiqIcm17ey8R7mu5F4DScU2wmSXb4m9nVdMBWdzU9SmlBDqeMdUoTBexqambFpt0s39TIis1Oknq+ZuvB7f3zshk/sJDxA4s4cpDzc2xFoc3i68HiTUYvA78UkcnAEsAfvFFVn0h0YL1RoECqTes2PV3/vD7MHFPGzDGHRlCNew+wastuVm3dw6qtu/l4yx4eeX8Dew+0AiACI0vzGT+okCMHFjF+UBHjBxYytNhu0u0J4k1Gd7g/rw2zTQH7cyUBaq1at+nF+vXNZvqoUqaPKj3Y1tbmVI4IJKfVW/ewcvNunl1+aBSV3yeTIyoKGTOggCMGFDBmQAFjBjhJyqabdx/xrvSaEbuX6SyP128FUo0JkpEhjCjLZ0RZPmcEXUv1729hzbY9rNrqJKhP6vfwxideHluy8WCf3OwMRpe7yami0Pm9ooDhJXlkZdpXWldjK712IbVeHyPK8qz0ijEx5OdkMa2ymGmVxYe1N+49wNp6H2vr9/DJNh+f1Pt4f91O/vHh5oN9+mRmMLIsn9ED8hlZls/IsgLneXm+zexLo7iTkVvx+kygEjjsv5iq3pTguHolj9dvBVKN6YR+fbM5Zngxxww/PEn59rdQW+8kp0/q97B2m4+Pt+zhhZpttLYdur2lOC+bkWX5jCp3EtSosnxGluczojSf3Gy7GpFMcSUjETkeeAZn5ddyYBMwyH2+DrBk1EmBAqlnHmUFUo1JtIKcLKYM68+UYf0Paz/Q2saGHU182uDH4/XjafDzaYPvM6f8RGBwv76MKndGU8NL86ksyWN4aR6VJXmWqBIg3pHRb4C/At/Bqbx9Gs6MuodxqhmYTlq/3SmQOqrMZtIZkyrZmRmMKi9gVHkBs488fJtvfwvrGtwE5XWS1KcNfp78YBN79rcc1reiKIfKkjwqS/IZXnooSVWW5FGS38dm+8Uh3mQ0GWdpbxWRViBHVT0i8n3gIZxEZTrh4LTuAZaMjOkKCnKymDTEqRYRTFXZ2XSA9dv91O1oom57E+vdn4vXNvD4B/s+c5zgUVRlaR7DivMYUtyXIf372qjKFW8yag76fRswHKcmnA8YnOigeqNab6BAqk3rNqYrExFK8vtQkt/nMxMoAPYdaGXDjibWu0nK+d3P6m17ePnjeppb2w7rX1aQw9Divgwp7svQ4r4M7d+XoUHJKj+nd8wzi/ddfgAci7MMxCLg5yJSAfwbh5ZlMJ3g8ToFUotyrZCkMd1ZbnYmYyoKGRNmMlJrm7J19z427dzLxp1NbNq5l0279rJx515Wbt7NSzXbPpOsivOyneTUv29Q0nKeD+nfl6K+WT3iNGB71jMKfLL/g7NMw+9xktM3kxBXr1Pr9VkZIGN6uMwMOZhEjhtZ8pntbW1Kg28/Gw4mqSY3ce1lrdfHojX17DtweLLK65PJwH65DOqXy6B+fRncL5eB/foyqP+htqLcrp+w4r3ptTrody/OFG+TQJ4GvxVINaaXy8gQBhTlMqAo9zPT08G5XrXD38xGN1lt3rWXLY372Nq4j82Ne1m8toFtu/fRFrIYQ16fzIOJaVC/XAb1d38G2vrnUpiT3oTVrpORIlIFjAb+6S6Wlw/sV9WWGLuaKKxAqjEmHiJCaUEOpQU5n5mmHtDS2kb9nv1sadzHlsa9TqLatY+tu/eyedc+3vikgfo94RPWwKJcBhTlMLAol4p+uVQU5jKwXy6zjxxATlZyJ1rEe59RBbAQOA6nFt0YwAP8H7APZ8q36aBaK5BqjEmQrMwMBvfvy+D+fYHPjq7Aub/Ku2c/WxqdkdWWXfvYutt5bGvcx5K6nWxr3H/w+tXqn5+R/Ljj7PdbnFl0pUBdUPujHFoO3HSQVes2xqRS9mEJK7zAFPb6PfuSPiqC+JPRbGC2qu4MOadYi1MeyHRCrddPn6wMhhRH/odhjDGpFDyFPRXiLV3bl8PvNQooxzlNFxcRKRGRJ0XELyLrRWRuhH4iIreIyHb3cYsEZUERmSoiS0Skyf05Nd59g/pdLCIqIlfEG3+yeLw+RpRagVRjTO8VbzJ6Hbg06LmKSCbwfZyF9+J1J05SqwAuBO4SkYlh+s0D5gBTcKo/nA1cCSAifXCuXz2Ic0L0PmCh2x513wC36Ot/AzXtiD1pPF6/naIzxvRq8SajG4BvichLQA5wK7ASOBH4r3gO4M68Ox+4UVV9qvom8BRwUZjulwC3qupGVd3kvt6l7rZZOKcXb1PV/ap6OyA49fJi7RvwK+B2oCGe2JOpuaWN9TuarPKCMaZXiysZqepK4CjgLeBFIBdn8sI0Va2N87XGAi2quiao7SMg3MhoorstXL+JwDJVDZ6YuCxke6R9EZHjgCrg7lgBi8g8EakWkWqv1xure4fU7WiitU1tZGSM6dXivs9IVbcCPw5uE5HhIvJ3Vf1aHIcowKn4HayRQ5UdQvs2hvQrcK/9hG4LPU60fTOAPwDXqGpbrBu8VHU+MB+gqqpKo3buoENLjVsyMsb0Xp1de7c/zqm3ePiAopC2ImBPHH2LAJ87Gop1nGj7Xo0zqnonzpiTzmMFUo0xptPJqD3WAFkiMiaobQrhJxHUuNvC9asBJofMkJscsj3SvrOBc0Vkq4hsBWYAt4rIHR14PwlR6/VRXmgFUo0xvVvKkpGq+oEngJtEJF9ETgTOAR4I0/1+4DoRGSIig4HrgXvdbYuAVuBaEckRkWvc9lfi2PdS4EhgqvuoBn6KUwg2LTxeH6PKbFRkjOndUjkyAuc0WV+gHmeV2KtUtUZEThIRX1C/e4CngeXACpwlz+8BUNVmnKnbFwO7gMuAOW57rH13qerWwANnmvluVQ29BpUSqkqt128L6hljer2oExhE5KkY+4deu4lKVXfgJJLQ9jdwJh4EnivOdPIbIhxnKXBMhG1R9w3pOyueuJNlh7+Zxr0HbGRkjOn1Ys2m2x7H9k8TFEuv42lwJi/YyMgY09tFTUaqagvnJVFtvVsgtcySkTGmd0v1NSMTxNNgBVKNMQYsGaWVx+tjZGm+FUg1xvR6lozSqNbrt5tdjTEGS0Zp09zSRt2OJqtJZ4wxWDJKm7odflrb1EZGxhiDJaO0qT1Yk85GRsYYY8koTQ5V67aRkTHGWDJKE4/XbwVSjTHGZckoTTxeH6NtVGSMMYAlo7QIFEi160XGGOOwZJQGgQKpNq3bGGMclozSoNZWdzXGmMNYMkoDj9cKpBpjTDBLRmlgBVKNMeZwlozSoLbeCqQaY0wwS0Zp4GnwM3qAXS8yxpiAlCYjESkRkSdFxC8i60VkboR+IiK3iMh293GLiEjQ9qkiskREmtyfU+PZV0TGishCEfGKyA4ReUFExiX/nR8SKJA6yq4XGWPMQakeGd0JNAMVwIXAXSIyMUy/ecAcYAowGTgbuBJARPoAC4EHgWLgPmCh2x51X6A/8BQwzo3hPfdYKWMFUo0x5rNSloxEJB84H7hRVX2q+iZOYrgoTPdLgFtVdaOqbgJuBS51t83CWS79NlXdr6q3AwKcFmtfVX1PVf+sqjtU9QDwW2CciJQm/h2Ht7bemdZt9xgZY8whqRwZjQVaVHVNUNtHQLiR0UR3W7h+E4FlqqpB25eFbI+0b6iTga2quj2ud5AAngYrkGqMMaFSmYwKgN0hbY1AYYS+jSH9CtxrP6HbQo8Tbd+DRGQozmnD6yIFLCLzRKRaRKq9Xm+kbu3i8foZUJhDoRVINcaYg1KZjHxAUUhbEbAnjr5FgM8dDcU6TrR9ARCRcuBF4A+q+nCkgFV1vqpWqWpVeXl5xDfWHrVen42KjDEmRCqT0RogS0TGBLVNAWrC9K1xt4XrVwNMDhnpTA7ZHmlfRKQYJxE9paq/6MD76DBVxeP12/UiY4wJkbJkpKp+4AngJhHJF5ETgXOAB8J0vx+4TkSGiMhg4HrgXnfbIqAVuFZEckTkGrf9lVj7ikgR8AKwWFV/kOC3GNN2t0CqVes2xpjDpXpq99VAX6AeeBi4SlVrROQkEfEF9bsHeBpYDqwAnnHbUNVmnKnbFwO7gMuAOW571H2Bc4FjgW+KiC/oUZmsNxzMYwVSjTEmrKxUvpiq7sBJJKHtb+BMPAg8V+AG9xHuOEuBYyJsi7ivqt6Hc19SWgQKpB5hIyNjjDmMlQNKoVqvjz5ZGQzubwVSjTEmmCWjFPJ4/YwqswKpxhgTypJRCtm0bmOMCc+SUYrsb2llw869ViDVGGPCsGSUInXbm2htU1s6whhjwrBklCK1gWndNjIyxpjPsGSUIlYg1RhjIrNklCK19VYg1RhjIrFklCKeBp/VpDPGmAgsGaWAqlJbb9O6jTEmEktGKbDd38zufS1WINUYYyKwZJQCtfXO5IXRNjIyxpiwLBmlgKfBmdZt14yMMSY8S0Yp4PH6yLECqcYYE5EloxSo9foZaQVSjTEmIktGKeDx2rRuY4yJxpJRku1vaaVuR5NN6zbGmCgsGSVZ3fYm2tTKABljTDSWjJIsUCDVTtMZY0xkKU1GIlIiIk+KiF9E1ovI3Aj9RERuEZHt7uMWEZGg7VNFZImINLk/pyZi32So9Tr3GI0ss5GRMcZEkuqR0Z1AM1ABXAjcJSITw/SbB8wBpgCTgbOBKwFEpA+wEHgQKAbuAxa67Z3dN+E8Xj8VRVYg1RhjoklZMhKRfOB84EZV9anqm8BTwEVhul8C3KqqG1V1E3ArcKm7bRaQBdymqvtV9XZAgNMSsG/C1Xp9toaRMcbEkMqR0VigRVXXBLV9BIQbGU10t4XrNxFYpqoatH1ZyPaO7nsYEZknItUiUu31eiO+sWiOGV7M7CMHdGhfY4zpLbJS+FoFwO6QtkagMELfxpB+Be61n9BtocfpzL6HUdX5wHyAqqoqDdcnlhvPmtCR3YwxpldJ5cjIBxSFtBUBe+LoWwT43BFNrON0Zl9jjDFpkMpktAbIEpExQW1TgJowfWvcbeH61QCTg2fI4UxUqEnAvsYYY9IgZclIVf3AE8BNIpIvIicC5wAPhOl+P3CdiAwRkcHA9cC97rZFQCtwrYjkiMg1bvsrCdjXGGNMGqR6avfVQF+gHngYuEpVa0TkJBHxBfW7B3gaWA6sAJ5x21DVZpyp2xcDu4DLgDlue2f3NcYYkwZy+MQyE0lVVZVWV1enOwxjjOlWRGSJqlbF6mflgIwxxqSdJSNjjDFpZ8nIGGNM2tk1oziJiBdY38Hdy4CGBIaTKBZX+1hc7WNxtU9PjWu4qpbH6mTJKAVEpDqeC3ipZnG1j8XVPhZX+/T2uOw0nTHGmLSzZGSMMSbtLBmlxvx0BxCBxdU+Flf7WFzt06vjsmtGxhhj0s5GRsYYY9LOkpExxpi0s2RkjDEm7SwZJZGIlIjIkyLiF5H1IjI3ia+1SET2iYjPfawO2jbXfX2/iPxDRErijTHavhHiuMZdqn2/iNwbsm22iKwSkSYReVVEhgdtyxGRBSKyW0S2ish1ido3WlwiMkJENOhz84nIjamIy93+Z/fz3SMiH4rImen+vKLFlc7Py+3zoIhscfusEZEr0v15RYsr3Z9XUN8x4nw/PBjUlpTvhVj7RqSq9kjSA2eZjEdwljufibPE+cQkvdYi4Iow7RNxVrI92Y3jIeBv8cQYa98IcZyHs0zHXcC9Qe1l7rG/CuQCvwHeCdr+K+ANoBg4EtgKnNHZfeOIawSgQFaE95O0uIB84CduDBnAWe7nPSKdn1eMuNL2eQX9m8xxfx/v9jkmnZ9XjLjS+nkF9X3R7ftgsr8Xou0b9bsjGV+M9jj4P3QzMDao7QHg5iS93iLCJ6NfAg8FPR/txlUYK8Zo+8YRz885/Et/HvBWyOezFxjvPt8MnB60/WeBf+Cd2TeOuGJ9WaQkrqA+y4Dzu8rnFSauLvN5AeOALcDXutLnFRJX2j8v4BvA33H+wAgko6R8L8TaN9rDTtMlz1igRVXXBLV9hPNXRbL8SkQaRGSxiMxy2ya6rwuAqtbi/mOJI8Zo+7ZX6LH8QC0wUUSKgUHB22PE0Z5947VeRDaKyF9EpAwg1XGJSAXOZ1vTyWMnM66AtH1eIvIHEWkCVuF86T/byWMnM66AtHxeIlIE3ASEnsJL1vdCh7/3LBklTwGwO6StEeevh2T4PjAKGIJzk9rTIjLajaMxQhyxYoy2b3vFioOQ7fHGEWvfWBqAY4HhOKdVCoG/Br1uSuISkWz3de9T1VWdPHYy40r756WqV7vtJwFPAPs7eexkxpXuz+tnwJ9VdWNIe7K+Fzr8vWfJKHl8QFFIWxHOudaEU9V3VXWPqu5X1fuAxcAXY8QRK8ZEvodYcRCyPd44Yu0blar6VLVaVVtUdRtwDXC6iBSmKi4RycA5ldHsvn5nj520uLrC5+XG0aqqbwJDgas6eeykxZXOz0tEpgKfA34bJtRkfS90+DvDklHyrAGyRGRMUNsUDj/VkUwKiPt6UwKNIjIKyHHjixVjtH3bK/RY+TjnmmtUdSfOaY0pQf2jxdGefdsrUJIkIxVxiYgAfwYqgPNV9UACjp3MuEKl9PMKIytwjE4cO5lxhUrl5zUL55pVnYhsBf4TOF9EPghz7ER9L3T8ey/WRSV7dPwB/A1nZkk+cCJJmk0H9Ae+gDPjJgu4EPDjnL+diDNsPsmN40EOn/kSMcZY+0aIJcuN41c4f1UHYip3j32+23YLh88Muhl4DWdm0Hic/9ECs4o6vG8ccU3HueCcAZTizAJ6NYVx3Q28AxSEtKf784oUV9o+L2AAzsX4AiAT59+8H/hyOj+vGHGl8/PKAwYGPf4XeMw9btK+F6LtG/W7I51f1j39AZQA/3D/YdYBc5P0OuXA+zhD4V04XyKfD9o+1319P7AQKIk3xmj7RojlJzh//QU/fuJu+xzOxd29OLP/RgTtlwMscP+RbwOuCzluh/eNFhdwAfCp+/62APcDA1MRF851BAX24ZzeCDwuTOfnFS2uNH9e5ThfvrvcPsuBbyXo2EmJK52fV4T/Bx5M9vdCrH0jPaxQqjHGmLSza0bGGGPSzpKRMcaYtLNkZIwxJu0sGRljjEk7S0bGGGPSzpKRMcaYtLNkZEwXIiL3isg/0x2HMalmyciYDnAXTIv2uLeDh/4O8G8JiO8KEVnqLubWKCLLROTnQdt/IiIrOvs6xiRKVroDMKabGhT0+1nAH0Pa9gZ3FpFsjVzf7SBVDa2G3G4ichlwO/Bd4GUgG5gEnNDZYxuTLDYyMqYDVHVr4IFTBoag57nALhG5QEReEZG9wJUiUioiD7vr2uwVkRoR+WbwcUNP04mznPwfROSX7lpV9SLyv25F7Ui+DDyhqveo6lpV/VhVH1XV69xjXgr8GGddnMBI7lJ3Wz8Rme++zh4ReU1EqoLiudQdbZ0tzvLa+8RZDntUIj5X03tZMjImeX4F/AGYgFOrKxf4AGckNRH4HXCPiMyOcZwLgRZgBs4SBP8BfD1K/63AcVESxCPArcBqnNHcIOARt1L3MzhrYp0FTANeB14RkeBRXw5OMvsmzmgrE3jC3d+YDrHTdMYkz+9V9bGQtt8E/T5fRE7DKab5cpTjrFTVH7m/rxGRbwGzcSojh/NTnLL9tSKyFngXeBF4WFUPqOpeEfHhrMi5NbCTG8tUoFxVA6cZbxSRs4GLgF+7bVnAd1R1sbvfRYDHjelfUd6HMRHZyMiY5KkOfiIimSLyQ3cywXY3IZwHVMY4zrKQ55txli0IS1W3qOoJwFHAbTjrWt0DvCcieVFe5xicZQe87qk4nxvjJA5fm6cNeC/o9da7MU2I8T6MichGRsYkjz/k+X8C1+PMmFuOsyzDL4mSWFyhEx+UOP6QVNUVwArgThGZCbwBfA24N8IuGThLEZwUZlvoUtJW7t8klCUjY1JnJvC0qj4AB1dTHYs7ASLJVro/C9yfzTjXeoJ9gLOya5uqeqIcKwM4DngLQEQqgcHAxwmL1vQ6dprOmNRZA8wWkZkiMh64AxiZ6BcRkbtE5EYROVFEhovI8TiLujXhXDsCWAcMF5GjRaRMRHJwrvcsBhaKyJkiMlJEThCRn4pI8GipBbjN3TYVuA9nWWm7XmQ6zJKRManzc5xrLc/hzFLzA39Nwuu8hLPc9d9xEuCTbvvnVXWN+/vjwLM4Eye8wAXqrLT5ReAVnPumVrvHGIdzTShgP/ALnAT3Ls73yHlqK3WaTrCVXo0xcXPvR7pDVQti9TWmPWxkZIwxJu0sGRljjEk7O01njDEm7WxkZIwxJu0sGRljjEk7S0bGGGPSzpKRMcaYtLNkZIwxJu3+Pwy1VmUNoEo7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "class CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n",
    "    def __init__(self, d_model, warmup_steps=4000):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.d_model = d_model\n",
    "        self.d_model = tf.cast(self.d_model, tf.float32)\n",
    "        \n",
    "        self.warmup_steps = warmup_steps\n",
    "    \n",
    "    def __call__(self, step):\n",
    "        step = tf.cast(step, dtype=tf.float32)\n",
    "        arg1 = tf.math.rsqrt(step)\n",
    "        arg2 = step * (self.warmup_steps ** -1.5)\n",
    "        \n",
    "        return tf.math.rsqrt(self.d_model) * tf.math.minimum(arg1, arg2)\n",
    "\n",
    "\n",
    "\n",
    "learning_rate = CustomSchedule(d_model)\n",
    "\n",
    "# for demonstration\n",
    "plt.plot(learning_rate(tf.range(40000, dtype=tf.float32)))\n",
    "plt.ylabel('Learning Rate')\n",
    "plt.xlabel('Train Step')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "executionInfo": {
     "elapsed": 14,
     "status": "ok",
     "timestamp": 1671396948470,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "TY4BPgG7z7sn"
   },
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam(learning_rate, beta_1=0.9, beta_2=0.98,\n",
    "                                     epsilon=1e-9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "executionInfo": {
     "elapsed": 12,
     "status": "ok",
     "timestamp": 1671396948470,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "5yyFN8wSNunJ"
   },
   "outputs": [],
   "source": [
    "# adding padding mask in the loss and accuracy.\n",
    "\n",
    "def masked_loss(label, pred):\n",
    "    mask = label != 0\n",
    "    loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True, reduction='none')\n",
    "    loss = loss_object(label, pred)\n",
    "    \n",
    "    mask = tf.cast(mask, dtype=loss.dtype)\n",
    "    loss *= mask\n",
    "    \n",
    "    loss = tf.reduce_sum(loss)/tf.reduce_sum(mask)\n",
    "    return loss\n",
    "\n",
    "\n",
    "def masked_accuracy(label, pred):\n",
    "    pred = tf.argmax(pred, axis=2)\n",
    "    label = tf.cast(label, pred.dtype)\n",
    "    match = label == pred\n",
    "    \n",
    "    mask = label != 0\n",
    "    \n",
    "    match = match & mask\n",
    "    \n",
    "    match = tf.cast(match, dtype=tf.float32)\n",
    "    mask = tf.cast(mask, dtype=tf.float32)\n",
    "    return tf.reduce_sum(match)/tf.reduce_sum(mask)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 4042487,
     "status": "ok",
     "timestamp": 1671400990948,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "krk5juRnNupb",
    "outputId": "f238c119-eeba-4458-8a2f-8ee6de16e798"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "810/810 [==============================] - 262s 285ms/step - loss: 6.6096 - masked_accuracy: 0.1413 - val_loss: 5.0489 - val_masked_accuracy: 0.2469\n",
      "Epoch 2/20\n",
      "810/810 [==============================] - 197s 242ms/step - loss: 4.5759 - masked_accuracy: 0.2968 - val_loss: 4.1175 - val_masked_accuracy: 0.3484\n",
      "Epoch 3/20\n",
      "810/810 [==============================] - 199s 245ms/step - loss: 3.8229 - masked_accuracy: 0.3802 - val_loss: 3.3999 - val_masked_accuracy: 0.4394\n",
      "Epoch 4/20\n",
      "810/810 [==============================] - 194s 239ms/step - loss: 3.2881 - masked_accuracy: 0.4383 - val_loss: 3.0520 - val_masked_accuracy: 0.4797\n",
      "Epoch 5/20\n",
      "810/810 [==============================] - 193s 238ms/step - loss: 2.8940 - masked_accuracy: 0.4828 - val_loss: 2.7508 - val_masked_accuracy: 0.5196\n",
      "Epoch 6/20\n",
      "810/810 [==============================] - 190s 234ms/step - loss: 2.5764 - masked_accuracy: 0.5216 - val_loss: 2.4920 - val_masked_accuracy: 0.5512\n",
      "Epoch 7/20\n",
      "810/810 [==============================] - 192s 236ms/step - loss: 2.3080 - masked_accuracy: 0.5566 - val_loss: 2.3569 - val_masked_accuracy: 0.5706\n",
      "Epoch 8/20\n",
      "810/810 [==============================] - 192s 237ms/step - loss: 2.1151 - masked_accuracy: 0.5825 - val_loss: 2.2532 - val_masked_accuracy: 0.5858\n",
      "Epoch 9/20\n",
      "810/810 [==============================] - 191s 235ms/step - loss: 1.9710 - masked_accuracy: 0.6026 - val_loss: 2.1846 - val_masked_accuracy: 0.5933\n",
      "Epoch 10/20\n",
      "810/810 [==============================] - 190s 234ms/step - loss: 1.8518 - masked_accuracy: 0.6199 - val_loss: 2.1497 - val_masked_accuracy: 0.6033\n",
      "Epoch 11/20\n",
      "810/810 [==============================] - 191s 235ms/step - loss: 1.7516 - masked_accuracy: 0.6339 - val_loss: 2.1274 - val_masked_accuracy: 0.6062\n",
      "Epoch 12/20\n",
      "810/810 [==============================] - 191s 235ms/step - loss: 1.6697 - masked_accuracy: 0.6462 - val_loss: 2.0947 - val_masked_accuracy: 0.6144\n",
      "Epoch 13/20\n",
      "810/810 [==============================] - 189s 234ms/step - loss: 1.5976 - masked_accuracy: 0.6574 - val_loss: 2.0840 - val_masked_accuracy: 0.6148\n",
      "Epoch 14/20\n",
      "810/810 [==============================] - 192s 236ms/step - loss: 1.5328 - masked_accuracy: 0.6669 - val_loss: 2.0550 - val_masked_accuracy: 0.6219\n",
      "Epoch 15/20\n",
      "810/810 [==============================] - 193s 237ms/step - loss: 1.4758 - masked_accuracy: 0.6761 - val_loss: 2.0592 - val_masked_accuracy: 0.6218\n",
      "Epoch 16/20\n",
      "810/810 [==============================] - 190s 234ms/step - loss: 1.4260 - masked_accuracy: 0.6837 - val_loss: 2.0618 - val_masked_accuracy: 0.6247\n",
      "Epoch 17/20\n",
      "810/810 [==============================] - 190s 234ms/step - loss: 1.3822 - masked_accuracy: 0.6910 - val_loss: 2.0520 - val_masked_accuracy: 0.6255\n",
      "Epoch 18/20\n",
      "810/810 [==============================] - 191s 236ms/step - loss: 1.3384 - masked_accuracy: 0.6974 - val_loss: 2.0585 - val_masked_accuracy: 0.6232\n",
      "Epoch 19/20\n",
      "810/810 [==============================] - 194s 239ms/step - loss: 1.2980 - masked_accuracy: 0.7043 - val_loss: 2.0613 - val_masked_accuracy: 0.6275\n",
      "Epoch 20/20\n",
      "810/810 [==============================] - 190s 234ms/step - loss: 1.2625 - masked_accuracy: 0.7100 - val_loss: 2.0891 - val_masked_accuracy: 0.6299\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fcb479b8bb0>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# training\n",
    "\n",
    "transformer.compile(\n",
    "    loss=masked_loss,\n",
    "    optimizer=optimizer,\n",
    "    metrics=[masked_accuracy])\n",
    "\n",
    "transformer.fit(train_batches,\n",
    "                epochs=20,\n",
    "                validation_data=val_batches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NGtYBiVfO9ew"
   },
   "source": [
    "- two points about inference:\n",
    "  - we may use `tf.Module` that is in the higher level of Keras models or layers to denfine inference object (not only for a more organized coding but also it has a memory and can be saved), and\n",
    "  - use `training=False` in the Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "executionInfo": {
     "elapsed": 471,
     "status": "ok",
     "timestamp": 1671401028364,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "bjSnZVTINur7"
   },
   "outputs": [],
   "source": [
    "# run inference\n",
    "\n",
    "class Translator(tf.Module):\n",
    "    def __init__(self, tokenizers, transformer):\n",
    "        self.tokenizers = tokenizers\n",
    "        self.transformer = transformer\n",
    "    \n",
    "    def __call__(self, sentence, max_length=MAX_TOKENS):\n",
    "        # The input sentence is Portuguese, hence adding the `[START]` and `[END]` tokens.\n",
    "        assert isinstance(sentence, tf.Tensor)\n",
    "        if len(sentence.shape) == 0:\n",
    "            sentence = sentence[tf.newaxis]\n",
    "        \n",
    "        sentence = self.tokenizers.pt.tokenize(sentence).to_tensor()\n",
    "        \n",
    "        encoder_input = sentence\n",
    "        \n",
    "        # As the output language is English, initialize the output with the\n",
    "        # English `[START]` token.\n",
    "\n",
    "        # The following has only IDs of [START] and [END]\n",
    "        start_end = self.tokenizers.en.tokenize([''])[0]\n",
    "        start = start_end[0][tf.newaxis]\n",
    "        end = start_end[1][tf.newaxis]\n",
    "        \n",
    "        # `tf.TensorArray` is required here (instead of a Python list), so that \n",
    "        # the dynamic-loop can be traced by `tf.function`.\n",
    "        output_array = tf.TensorArray(dtype=tf.int64, size=0, dynamic_size=True)\n",
    "        output_array = output_array.write(0, start)\n",
    "        \n",
    "        # input to the loop should be tf.range(max_length) not range(max_length)\n",
    "        # to keep this loop dynamic.\n",
    "        for i in tf.range(max_length):\n",
    "            output = tf.transpose(output_array.stack())\n",
    "            predictions = self.transformer(\n",
    "                [encoder_input, output], training=False\n",
    "                )\n",
    "            \n",
    "            # Select the last token from the `seq_len` dimension.\n",
    "            # Below shape `(batch_size, 1, vocab_size)`.\n",
    "            predictions = predictions[:, -1:, :]\n",
    "            \n",
    "            predicted_id = tf.argmax(predictions, axis=-1)\n",
    "            \n",
    "            # Concatenate the `predicted_id` to the output which is given to the\n",
    "            # decoder as its input.\n",
    "            output_array = output_array.write(i+1, predicted_id[0])\n",
    "            \n",
    "            if predicted_id == end:\n",
    "                break\n",
    "        \n",
    "        output = tf.transpose(output_array.stack())\n",
    "        # The output shape is `(1, tokens)`.\n",
    "        text = tokenizers.en.detokenize(output)[0]  # Shape: `()`.\n",
    "        \n",
    "        tokens = tokenizers.en.lookup(output)[0]\n",
    "        \n",
    "        # `tf.function` prevents us from using the attention_weights that were\n",
    "        # calculated on the last iteration of the loop.\n",
    "        # So, recalculate them outside the loop.\n",
    "        self.transformer([encoder_input, output[:,:-1]], training=False)\n",
    "        attention_weights = self.transformer.decoder.last_attn_scores\n",
    "        \n",
    "        return text, tokens, attention_weights\n",
    "\n",
    "\n",
    "translator = Translator(tokenizers, transformer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 2558,
     "status": "ok",
     "timestamp": 1671401033873,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "SjONfnAYDQL3",
    "outputId": "ecf05c39-a77a-4270-d7bf-be2d3efd7bbb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:         : este é o primeiro livro que eu fiz.\n",
      "Prediction     : this is the first book i did .\n",
      "Ground truth   : this is the first book i've ever done.\n"
     ]
    }
   ],
   "source": [
    "# e.g., trying a translation\n",
    "\n",
    "def print_translation(sentence, tokens, ground_truth):\n",
    "    print(f'{\"Input:\":15s}: {sentence}')\n",
    "    print(f'{\"Prediction\":15s}: {tokens.numpy().decode(\"utf-8\")}')\n",
    "    print(f'{\"Ground truth\":15s}: {ground_truth}')\n",
    "\n",
    "\n",
    "sentence = 'este é o primeiro livro que eu fiz.'\n",
    "ground_truth = \"this is the first book i've ever done.\"\n",
    "\n",
    "translated_text, translated_tokens, attention_weights = translator(\n",
    "    tf.constant(sentence))\n",
    "print_translation(sentence, translated_text, ground_truth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 309
    },
    "executionInfo": {
     "elapsed": 601,
     "status": "ok",
     "timestamp": 1671401040467,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "RmwFtyiuDVQ0",
    "outputId": "1c425619-36f6-49f7-a5e4-206e021dea03"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUIAAAEkCAYAAABaADjbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAbWUlEQVR4nO3debRdZZ3m8e+TBIgBkQ4zCKEYnYclIthSRLEUZbAWhRK0UEFNOZR0KTbSIgiFYHWLOICl0ioKlqBCOSFWWSgqWkzRJdpoiSKEKQwBIgmBkJCn/9j7wsnlzvvsc27O+3zWuivn7L3v731zh999934n2SYiomQz+l2BiIh+SyKMiOIlEUZE8ZIII6J4SYQRUbwkwogoXhJhRBQviTAiipdEGBHFSyKMiOIlEUZPSTpK0o8k/b7+96h+1yliVr8rEOWQdALwRuBjwGJgHnCcpO1sn9bXykXRlEUXolck3QTMt72449g84Ke25/WvZlG6gW8RSvrHCV662vaprVYmNgbuGXbsXuBJfahLxGMGvkUo6RHgXyZw6WG2n9x2fUom6TzgycDxwC1Ut8anASttH9nPukXZSkiEyyeS4CTdb/u/9aJOpZK0KXA2cDjV3chq4OvAMbaX9bNuUbYSEuGGth/p1nUxNZJmAh8CTgceAbYAltpe29eKRVBAIgSQ9D3bB/a7HqWTtBTYKskvpptSxhHu2+8KBADnAW/vdyUihiulRfiA7U37XY/SSfoZ8CLgduBW4LEfPtt/2a96RZSSCB8BLhzrGttv7FF1iiXpTaOds/3lXtYlotPAjyOsGbix35UoXZJdTFeltAhza9wnko60fX79+ujRrrP9xd7VKmJdpSTCCY0ljO6TdKntV9evLx/lMtt+WQ+rFbGOUhLhZ2y/Y5RzzwZOsv3aHlcrIiZA0s4TvHSt7ZunVEYhiXBjqmldzwP+AJxMNaD3Y8BfAV+2/a6+VbAgkjYHXg1sY/ujkrYDZti+rc9Vi2lK0lqq5/wa59KHbG88lTJKGUd4FnAw8Fvg5cDFwE+A64GdkgR7Q9J+wO+BNwAn1Yd3Az7Tt0rF+uBB2zNtzxjrg2rG0pSU0mt8APA823dLOotqwv9+tq/oc71K8wngcNs/lHR/fexqYK8+1immv/8xweveM9UCSrk1XqfXOL3I/dG5sIWk+2zPlTQDuMf25n2uXhSslBbhLEkvpeMZw/D3tn/Uj4oV5reSXmn73zuOvRz4Tb8qFOsXSa+mmjI7F7iPalHf7zeOW0iL8GY6pnONwLYn2jMVUyRpb+AS4HvA66jmHh8MvMb2tf2sW0xvkjYELgX2ARYBdwDbAy8ArgJe1WT1qCISYUwfkran6iyZRzXf+CvpMY7xSHo/sAA4xPatHcd3AL4FfMP2P005fsmJsH4+9SrgTbZf1+/6RMTIJF0LHGv7pyOc2w840/YLphq/lGeE65D0XOBNwOuBOVS3aN2KvQOwve2ruhVzUEh6CnAM8Hxgk85ztl/Rl0rF+mI3YLTHJ9cAuzYJXkwilLQV8LdUCfAZwE+pNhN69lRHow+LvyNwAdWgbQObSDoMOMD2W5vGHxDfAGYC3wQeaquQ+nuxPXC77VvaKid6SrZH/Jmx/ZA03ljrsRWRCCV9j2oGya+BLwMX2r5D0hJgZZeK+RxVJ8C+VDuzAfwH1eyVqOwNbNHWlgiStqVabm0fqu/B5pKuAhbYvqONMqNnNpR0FKPPLtmgSfAiEiGwH/AA8H3g0pZ+KfYCDrS9VpIBbP+5vh2Mys+Ap1H9QWrDZ4DrgFfbfrCeWnk68FngkJbKjN64GhhrzdCrmwQvJRFuDfwN1W3xByRdR7XF5waMPaxmMu6iek5xw9ABSc+gmsUSlTcDl0q6murr9RjbE91/eiwvAba1vbqO+aCk46hWxG5E0kZU0wKPADa3/RRJrwB2t3120/gxNtvz24xfxFxj2w/aPs/2/sDOwL8CC6kGZZ5fD9Js6gzgkrr5PkvSEcDXgP/dhdiD4jRgB6o/TLt1fDR60N3hfqrnv532ALqxVejHgWdRDf0Z+uN5PTDiqkaxfili+Iykf7b9zhGO70PVSjmsG1O8JL0G+DuqMXK3AJ+z/a2mcQeFpOVULaglLcV/G9Wt8BeAxVTfh6OAE22f0zD2EmDXupV5n+259fFltjdrWPUYR8cKNKOx7Snf4ZaSCMecWyxpI9urGpbxIttPeE4haS/b1zSJPSjqRxL7217aYhkvoxoWtR3V7IMLbP+wC3EXA8+pn/sOzZPeErjK9i5N48fYJI32NT4UeD+wxPazpxw/ibDdMjpbD6WT9D6qH9yzeOIzwkZzvesN5G8AntH0j9oo8c+guoV/D/AL4JlUq+n80fYJ3S4vxibplcCpwGZU64te4AbJrJREuIpxntXZPmms82PEnkHVpb8M2JR1u/d3AX5ue6upxB40km4a5VRX5npLugF4oe0/N401QuwNqX6G3kY1CH8l8H+B97c1HCieSNK+VI8/dqRKhOfafrRp3FJ6jUX1kL4Na3j82cWaYefWUnUQBGD7L1ou4hPA1yWdDtzGuvsm/6lJ4DrZvQd4T31LvLRJCyQmR9KewIeB51AlwnO6+QeolBZha7fGkuZRJdqfAJ2blJtqnb3WZlDEuuoH6iOx7ZkNY4/aYm2aZGN89ff2XqoJESNOgpjqXR2U1SJshe3F9ct56xQoPYmqRVg0Sb+z/fT69a2M0vNne8emZdXLtbfljzxx34yh/0ujJDukza+PpDcCP7R9+7DjR9i+oEnsHjmP6muzef3RVaW0CB/bUrLFMs4Avm77GkkHAhdRfeMOt/3dNsueziS9xPbP6tf7jXad7Z90oay/Bi6xPfwRRddJ2gb4EHCF7a92Kebwr8+2VMvUX2j7kw1jPwosAV5r+8qO41mtHcD2wH9QrXSyccd7UT30/iTVPNRulLEEmFO/vppqJsvLgd/0+/8/HT6oWk1fBjZqsYxfUa1a/HngpT34P20ELG65jG2AX3UhznKqvXvuAt7SebzfPxsTrP9rh73fY9j7f2gSv4iZJVQT8Q/teH8G8E9UY80+JenYLpQxx/bKervKnW1fbPsyht0yl8pVz94raPFxge3nUU2zuxP4vKTbJH1M0pTXqRvHHlQ9yG1aBXSjk8m2/41qUZBjJZ1VDzlaX24JvzDs/ZXD3jeaolnKrfES4Om2l9XDIJZSLQ9/uaS9gPNsP61hGddS9VruSvXX6vWStgCut7110//DIKjn/W4GfMj1fOCWy9ub6hdkfzfvLLmCdZPGHKqxhKfaPr1J7I4yhv8yz6HaA/rXthc0jL3c9pPr10+hmms/B9jbdtvJvLHO+tfvH9sIbKTzk1VKZ8kc20PzTfcE1ti+HMDVM71tu1DGu6gS4SPAW+pjrwR+0IXYg+LdVLd675V0D493Pthd6CwZUi+Ou4Bqhsk84NwuhP38sPcPAtfZ/kMXYg/pHOJlqt7RM4HzuxD78scCV7NjDqYahtL2kKZuGd5iG+/9pJTSIvwdVafFryWdDDzf9mvqc5sBN7gLg57r1UgWAFvaPrge+7Spu7RDnqS/quNv1VL83ahWV9measWWC23fMPZnTSp+250l76RKfs+l2ujnAqpl1xqPN5N0KhP4ZXODIRx1OUM/Q1vZPqjb3+P1laQHgKdQ/eEUcA9V7/HQ+/vdoNOnlBbhGcAPJP0nVSut83nhK+nC+niS3k3Vw/d5qo4SqFZh/hTw4i7HP6yF+AdT3S5dQrVgwR7AtZKOtP2dpvFrVwIfpEq2Q3OBL6R7g84Polog95u2V3Qp5pBdqb6v11J9fXakWoPyYuDh+ppmrRLp74F/oEs/Q5JOsH1a/XrUZ2hNk3ePbMK6ExbU8V6kRTgxdWtkT+Bq18M56uOHUP01uaJh/BupnkXdPPT8on4Yfbe7s7JN2/F/Axwz9MigPjYfONv2s5rGr+N9gSrBnsbjq8N8APiD7aO7UUZbJF1ItVPaxR3HDqXqzTyiS2V09XssaY3rFVkk/QvVY5snsH1Uo4r3QD1xYUx+fEzv5OOXkgjbJuluqkVBH+1YnWQ2cJPtxs8gexD/fqpb+jUdx2ZRTSXryjJTku4Fdul4XoukuVQLF0xpYQpJ59heWL8edRMu22OtbjyRcv4MzHXHvNb663Ov7a6sQt7t73HnGMGMFxzbwA+fqZ/tTOS6UxoW9VPg+GHHjqHjIfU0j/8rYPgwovfWx7vlTp443ORJVGMwp6pzIYcbx/ho6o9UHWKd3tGl2EO6/T2+sR4+dDTVYsFHSTp6+EejGvdAL36HB75FqGox0Ocw/jS7X3R2x0+hnG2B7wJbUHU2/IlqEOtBtu+catwexn9aHX9jqo3Xd6DqtTzY9u+axq/LOJ6qM+MsqkURdqBKLl+lY6vGyXQMqFp/cFxNOxskPZ9q971ZVB1J21M9ozrU9i+bxO4oo6vfY0m7A8dRPYJ4KTDS4x/bntDXsAlJx3uKG7D34ne4hEQ4tLLteF/Eh5uOp5Ik4IVUP3i3AtfY7toA4h7En0W109xQR8bV3Rzvp9GX4epkT2JJrjZijlHWBjz+9VkCXNnt8ZBtfY8l/dDVVhV9oQbTXHvxOzzwiTAiYjwD/4wwImI8RSZCSQvX9zISf7Dj96KMxH9ckYmQaivP9b2MxB/s+L0oI/FrpSbCiIjHDERnyYbayLPZeMLXr2YVG7DRhK/XRhtOuk6PPLqSDWdOrANry90emHT8B+5bw6ZzJz5D8u7/N3tS8Sf7NZqsxO9/GaXFX879S21vOdK5gZhrPJuNedGMl7cWf9ZT211S8O++1f4CNZ/efY92C1DLNxdrG29UNtjU2m4Uj1vPG02X+aJRp+Dl1jgiipdEGBHFSyKMiOIlEUZE8ZIII6J4SYQRUbyeJEJJO0lyvbrJSOc/IGn45jgRET3R2jhCSTcDb6339h1Tt7ZDjIiYitwaR0TxWkmEks6n2uXru5JWAK+rT71B0i2Slko6oeP6kyV9pX49W9JXJN0raZmkayVlg/SIaE0ridD2kcAtVMu8bwJ8vT71EqpdzPYHTpL09BE+/U1U+5fuQLVv6duptjRch6SFkhZJWrSaVS38LyKiFL2+NT7F9kO2rwOuo9qIe7jVVAlwV9uP2v6F7SesSmD7HNt72t6z7cnvETHYep0IOzegWUm1afNw5wP/Dlwo6Q5J/6feKyIiohVtJsIpLVVhe7XtU2w/A3gxcBDQaE/aiIixtJkI7wImvXOYpJdKerakmcADVLfKXdupLSJiuDYT4UeAD0paBhw2ic/bBriIKgn+DvgJ1e1yREQrWhtQbfvbwLc7Dp0x7Pz8jtcnd7y+ALigrXpFRAyXAdURUbwkwogoXhJhRBQviTAiijcQu9gBre6wteamUTe/6opDNl7ZanyAT7e9A5mzy1xfrec7zPVbWoQRUbwkwogoXhJhRBQviTAiipdEGBHFSyKMiOIlEUZE8ZIII6J40yYRSrpe0vx+1yMiyjNtZpbYfma/6xARZZo2LcKIiH6ZNolQ0s2SXi5pr3qbzgck3SXpzH7XLSIG27RJhB0+CXzS9qbALjy+J/I6sq9xRHTLdEyEq4FdJW1he4Xtq0a6KPsaR0S3TMdE+BZgd+C/JF0r6aB+VygiBtu06TUeYvsPwBGSZgCHAhdJ2tz2g32uWkQMqGnXIpT0t5K2tL0WWFYfzr7GEdGaadciBA4AzpQ0B1gMLLD9UJ/rFBEDbNokQts71S8v62c9IqI80+7WOCKi15III6J4SYQRUbwkwogo3rTpLJnOZm23bavxD9hxZqvxAbRBy3/zZqjV8Npww1bjr12xotX4d79jn1bjL9+5/RFmu7xvxEleAyEtwogoXhJhRBQviTAiipdEGBHFSyKMiOIlEUZE8ZIII6J4SYQRUbyeJkJJO0mypAzkjohpo/VEOLQ7XdvlRERMVW6NI6J4rSZCSecDOwLflbQCeF196g2SbpG0VNIJHdfPkHS8pBsl3Svp65LmtlnHiIhWE6HtI4FbgINtb8LjexS/BNgD2B84SdLT6+PvBv4a2A/YDrgf+PRIsbOvcUR0S79ujU+x/ZDt64DrgOfWx98OnGD7NturgJOBw0bqXMm+xhHRLf3qvb2z4/VKYJP69Tzgm5I61xR6FNgauL1HdYuIwvQiEXoS194KHG37521VJiJiuF7cGt8F7DzBaz8LnCZpHoCkLSW9prWaRUTQm0T4EeCDkpYBh41z7SeB7wA/kLQcuAp4Ucv1i4jCtX5rbPvbwLc7Dp0x7Pz8jtdrgTPrj4iInsiA6ogoXhJhRBQviTAiipdEGBHFSyKMiOJlXcAJWHPHknYLUPt/j2Zu/pRW49/0rt1ajb921mTG5U/eTie2u3n5Nl/6Vavxt3rooVbjD7q0CCOieEmEEVG8JMKIKF4SYUQUL4kwIoqXRBgRxUsijIjidT0RStpD0q8kLZe0VtKJ3S4jIqKb2mgRHgdcbvvJtmfYPnUqQbIfckT0ShuJcB5w/XgXjbQhU0REP3Q1EUr6EfBS4GxJKyR9VdKH63PzJd0m6f2S7gTOlbSFpEskLZN0n6Qr6r2N19kPWdJx3axnRESnrrbKbL9M0o+Br9j+vKQvDbtkG2AuVatxBnAScBuwZX1+7yqMj5S0L/BW25eNVJakhcBCgNnM6eZ/IyIK0+te47XAh2yvsv0QsBrYFphne7XtK2xPaHZ99jWOiG7pdSK8x/bDHe8/CvyRarOmP0k6vsf1iYjoeSJcp7Vne7ntY23vDBwCvFfS/iNdGxHRlr4OqJZ0kKRdJQn4M/Ao1e0zTG4/5IiIKev3zJLdgMuAFcCVwD/bvrw+99h+yJLe168KRsTg6/pYvmH7FL+54/WPgacOu/bjwMdHiTN8P+SIiFb0u0UYEdF3SYQRUbwkwogoXhJhRBQviTAiipcVYCZiYrP+Glg7/iUNPXrPPa3G3/GUpa3Gv/CWn7caf8GJ/73V+F69ptX47f+MDra0CCOieEmEEVG8JMKIKF4SYUQUL4kwIoqXRBgRxUsijIjiTTkRtrHdpqQfS3prN2NGRIwnLcKIKF4SYUQUr2kifKGk30q6X9K5kmYDSHqbpD/WexV/R9J2Q58g6cWSrpX05/rfF48UWNK2kn4t6X82rGNExJiaJsI3AK8EdgF2p1pa/2VUy+y/jmqrzsXAhQCS5gLfAz4FbA6cCXxP0uadQSX9BfAT4GzbH21Yx4iIMTVNhGfbvtX2fcBpwBFUyfGLtn9pexXwv4B9JO0EHAj8wfb5ttfYvgD4L+DgjpjPAC6n2v/4nNEKlrRQ0iJJi1azquF/IyJK1jQR3trxejGwXf2xeOig7RXAvcD2w891fN72He/fANwOXDRWwdngPSK6pWki3KHj9Y7AHfXHvKGDkjamug2+ffi5js+7veP9ycBS4KuSZjasX0TEuJomwndJemr97O8E4GvABcBRkp4naSPgdOBq2zcDlwK7S3q9pFmSDqe6Fb6kI+Zq4LXAxsB5ktKzHRGtappkvgr8APgTcCPwYduXAScCFwNLqDpSFgDYvhc4CDiW6nb5OOAg2+us6mn7EeBQYGvgi0mGEdGmKa9QbXun+uVHRjj3WeCzo3zez4AXjHJufsfrh4GuzlyJiBhJWloRUbwkwogoXhJhRBQviTAiipdEGBHFy77G08Eg7Enb8v9hwQ4jrs3RRe3W/7QbftZq/JP+8tBW4wOsufW21svol7QII6J4SYQRUbwkwogoXhJhRBQviTAiipdEGBHFSyKMiOIlEUZE8aZ9IpR0vaT5/a5HRAyuaT+zxPYz+12HiBhs075FGBHRtmmfCCXdLCkrVUdEa6b9rfFoJC0EFgLMZk6faxMR67Np3yIcTfY1johuWW8TYUREtyQRRkTxkggjonhJhBFRvGnfa9yxkXxERCvSIoyI4iURRkTxkggjonhJhBFRvGnfWRIxCE7c+6BW4591zYWtxgd457yXtF5Gv6RFGBHFSyKMiOIlEUZE8ZIII6J4SYQRUbwkwogoXhJhRBQviTAiiteXRCjpS5I+LGlfSb8f77pe1i0iytPXmSW2rwD26GcdIiJyaxwRxetJIpT0fEm/lLRc0teA2fXx+ZJuG++6iIg2tZ4IJW0IfAs4H5gLfAP4m6le13H9QkmLJC1azao2qh4RhehFi3BvYAPgE7ZX274IuLbBdUD2NY6I7ulFItwOuN22O44tbnBdRERX9SIRLgG2l6SOYzs2uC4ioqt6kQivBNYAx0jaQNKhwF4NrouI6KrWE6HtR4BDgTcD9wGHA/861esiIrqtJwOqbS8Cnj/K6adO8LqIiFZkQHVEFC+JMCKKl0QYEcVLIoyI4iURRkTxssH7RMyY2Wp4zWw3PgBe23J4j39RAzPnbtZqfM1q91dh7dZzW42/4DdHtxofYIs5t41/UQNrV65sNf5Y0iKMiOIlEUZE8ZIII6J4SYQRUbwkwogoXhJhRBQviTAiipdEGBHFSyKMiOIlEUZE8ZIII6J46+1cY0kLgYUAs5nT59pExPpsvW0RZl/jiOiW9TYRRkR0SxJhRBQviTAiijftE6Gk70v6QL/rERGDa9r3Gtt+Vb/rEBGDbdq3CCMi2pZEGBHFSyKMiOIlEUZE8ZIII6J4077XeMKk1kLP2nrL1mIDPLr0vlbjA3jNmnYLaPHrD/Do0ntbjT9jTrvz1fXww63GX/mfz2w1PsCSt27Ravztzv9dq/EZ49csLcKIKF4SYUQUL4kwIoqXRBgRxUsijIjiJRFGRPGSCCOieEmEEVG8cROhJEt6UNJpvahQR7lvkbSiLn/XXpYdEWWZaIvwubZPAJC0U52cVgz7OLw+/6X6/F5DnyxpV0nueP9jSQ9LWi7pAUm/kHS8pMd2YbL9BdubdOn/GRExqia3xpvZ3qTj42sd5+4DPjzO5/+97ScD2wLHAguAS6WW52pFRAzT1jPCLwPPkbTfeBfaftD2j4FDgH2AAydSgKSFkhZJWrSaVY0qGxFlaysRrgROByb8XNH2LcAiYN8JXp99jSOiK5okwqWSlnV8PH3Y+c8BO0qazJ4jdwBzG9QpImLSmiTCLWxv1vGxzho6tlcBp9YfE7U9Yy6WExHRfW2PIzwX2Aw4dLwLJe0AvAC4ouU6RUSso9VEaHsN8CHg/aNdI2lO3anybeAa4NI26xQRMVyTRLhs2DjC945y3QXAkhGOny1pOXAX8AngYuAA22sb1CkiYtImslT/KuAXkj5l+0TbNwOjjvWz/eZh79cCzxp2bP54hUo6Cvh4XX6SY0S0ZtxEaHt2LyoyQrnnUj1jjIhoVRZdiIjiJRFGRPGSCCOieLI9/lXTnKR7gMWT+JQtgKUtVadXZST+YMfvRRmlxZ9ne8RNygciEU6WpEW291yfy0j8wY7fizIS/3G5NY6I4iURRkTxSk2E5wxAGYk/2PF7UUbi14p8RhgR0anUFmFExGOSCCOieEmEEVG8JMKIKF4SYUQU7/8DVvspmYYZRRkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_attention_head(in_tokens, translated_tokens, attention):\n",
    "    # The model didn't generate `<START>` in the output. Skip it.\n",
    "    translated_tokens = translated_tokens[1:]\n",
    "    \n",
    "    ax = plt.gca()\n",
    "    ax.matshow(attention)\n",
    "    ax.set_xticks(range(len(in_tokens)))\n",
    "    ax.set_yticks(range(len(translated_tokens)))\n",
    "    \n",
    "    labels = [label.decode('utf-8') for label in in_tokens.numpy()]\n",
    "    ax.set_xticklabels(labels, rotation=90)\n",
    "    \n",
    "    labels = [label.decode('utf-8') for label in translated_tokens.numpy()]\n",
    "    ax.set_yticklabels(labels)\n",
    "\n",
    "\n",
    "head = 0\n",
    "# Shape: `(batch=1, num_heads, seq_len_q, seq_len_k)`.\n",
    "attention_heads = tf.squeeze(attention_weights, 0)\n",
    "attention = attention_heads[head]\n",
    "\n",
    "in_tokens = tf.convert_to_tensor([sentence])\n",
    "in_tokens = tokenizers.pt.tokenize(in_tokens).to_tensor()\n",
    "in_tokens = tokenizers.pt.lookup(in_tokens)[0]\n",
    "\n",
    "plot_attention_head(in_tokens, translated_tokens, attention)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 528
    },
    "executionInfo": {
     "elapsed": 597,
     "status": "ok",
     "timestamp": 1671401041560,
     "user": {
      "displayName": "Shayan ss",
      "userId": "05409118761901866233"
     },
     "user_tz": -60
    },
    "id": "ocfU4DoEES7e",
    "outputId": "c40133d3-eb9e-4720-849f-db1d57dc5721"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHgAAAIxCAYAAAA2dQ/dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZxkdXn3/c81Kw7D4gw7whAWUdxvR8TcIqi4gNsdV4wRxSiJMTExequPKK5o8rhHEpXHRAUV3KJGxWgUFU3CZiIadxCGVWVmWIZttr6eP6pGi6Z7urrqd07VOfV5v171mupTp666qrr626evOedUZCaSJEmSJElqrgWjbkCSJEmSJEnDccAjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIZzwCNJkiRJktRwDng0kIg4MSLOjYifdf89cdQ9SWovM0dS3cwdSXUyc1TColE3oOaJiJOBE4B3AmuAVcArI2KfzDx1pM1Jah0zR1LdzB1JdTJzVEpk5qh7UMNExOXA0Zm5pmfZKuC8zFw1us4ktZGZI6lu5o6kOpk5KqW1e/BExJv6XHVzZr650mbaZ0fg+mnL1gF3G0Ev0tgwdypj5kgzMHMqZe5I05g5lTJzVERr9+CJiE3Ax/tY9emZuVPV/bRJRJwB7AS8GriSzi6EpwK3ZeZzR9mbNErmTjXMHGlmZk51zB3prsyc6pg5KqXNA54N/QRLRNyQmXevo6e2iIidgdOAZ9HZC2wz8CngpZl54yh7k0bJ3KmGmSPNzMypjrkj3ZWZUx0zR6W0ecCzJDM3lVpPHRGxEHg98FZgE7AbsDYzp0bamDQGzJ3yzBxpdmZONcwdaWZmTjXMHJXU2gEPQER8OTOfMOo+2iYi1gJ7GDrSXZk75Zk50uzMnGqYO9LMzJxqmDkqZcGoG6jYkaNuoKXOAP501E1IY8rcKc/MkWZn5lTD3JFmZuZUw8xREW3fg+fmzNx51H20TUR8F3gocA1wFfDbN1FmPmJUfUnjwNwpz8yRZmfmVMPckWZm5lTDzFEpbR/wbALO3t46mXlCTe20RkQ8b7bbMvOjdfYijRtzpzwzR5qdmVMNc0eamZlTDTNHpSwadQMVS+CyUTfRNoaMtF3mTmFmjrRdZk4FzB1pVmZOBcwcldL2PXjchbCQiHhuZp7Zvf6C2dbLzH+qrytp/Jg7ZZg5Un/MnHLMHWluZk45Zo6q0PYBz4bM3GnUfbRBRJyTmcd1r39zltUyMx9VY1vS2DF3yjBzpP6YOeWYO9LczJxyzBxVoe0Dnvdn5otnue1+wCmZ+Yya25IaJyIO7HPVqcy8ospexp25Iw3PzOmfmSMNz8zpn5kjlVFV7rR9wLMj8GrggcAvgDcAuwHvBB4DfDQzXzKyBhssIlYCxwF7ZebbI2IfYEFmXj3i1lSBiJiic8x1zLHq7Zm5Yw0tjS1zpxpmzmQxc/pn5lTH3JkcZk7/zJzqmDmTparcWTBUV+PvfcCTgB8DxwCfBb4N/Ag4wPAZTEQcBfwMeA5wSnfxIcD7R9aUqnZrZi7MzAXbuwCbRt3oGDB3CjNzJpKZ0z8zpwLmzsQxc/pn5lTAzJlIleRO2z9F6/HAAzPzNxHxPuBK4KjM/M6I+2q69wDPysxvRMQN3WUXAIePsCdV6y/7XO9llXbRDOZOeWbO5DFz+mfmVMPcmSxmTv/MnGqYOZOnktxp+yFadzrLu2d9LyMibsjMu3evr8/MFRGxALg+M1eOuD1ppMyd8swcaXZmTjXMHWlmZk41zByV0vY9eBZFxCPpOa5t+teZee4oGmu4H0fE4zLzqz3LjgF+OKqGVJ+IOA44ElgBrAfOy8yvjLarsWLulGfmTDAzZ05mTjXMnQll5szJzKmGmTPBSuZO2/fguYLOiYtmk5nZ79mr1RURRwBfAr4MPBM4g86xuE/JzItG2ZuqExFLgHOAhwEXA9cC+wIPBs4Hjs3MiT823dwpz8yZTGZOf8ycapg7k8fM6Y+ZUw0zZzJVkTutHvCoOhGxL52TgK0CrgI+5hne2y0iXgUcDzw5M6/qWb4f8Hng05n5N6PqT+1m5kweM0ejZu5MFjNHo2bmTJ4qcmciBzzd4xmPBZ6Xmc8cdT9SE0TERcDLM/O8GW47CnhXZj64/s6awdyR5sfMGY6ZI82PmTMcM0eavypyp+3n4LmTiHgA8DzgD4FldHZ9G6befsC+mXl+gfYaIyJ2AV4KPAhY3ntbZj52JE2pDocAs+0ieiFwcI29NEbJ3DFzzJwJY+YMwG2dMsydiWTmDMDMKcPMmVjFc6f1A56I2AP4IzrBcxhwHrAjcL/MvGLAmvsDZwEPpHMM6vKIeDrw+Mx8YYm+x9yngYXA54DbSxbuvrb7Atdk5pUla2tokZkzfr8z8/aImOmmiVQ6d8wcM2dCmTl9clunEubO5DFz+mTmVMLMmUzFc6fVA56I+DLwGOAHwEeBszPz2oi4DrhtiNIfpHMCrCOBdd1l/wa8c4iaTXIEsFvJE81FxN7A2XROMLUOWBkR5wPHZ+a1pR5HQ1kSESfS8ykJ0yyus5lxVVHumDlmziQyc/rgtk5lzJ3JY+b0wcypjJkzmYrnTqsHPMBRwM3AV4BzCr6RDweekJlTEZEAmXlTd9e6SfBd4F50gr2U9wOXAMdl5q0RsSPwVuADwJMLPo4GdwFwwhy3q5rcMXPMnElk5vTHbZ1qmDuTx8zpj5lTDTNnMhXPnbYPePYEnkZn98HXRMQlwMfpTMKGObv0r+kcD/fzbQsi4jBgUnZ5ez5wTkRcQOe1+K3MfNOANR8O7J2Zm7t1bo2IVwLXDNpkRCwFTgGeDazMzF0i4rHAPTPztEHrTqrMPHrUPTREFblj5pg5E8fM6ZvbOtV4PmOeO2ZOWWZO38ycajyfMc8cMHdKqyJ3FpQuOE4y89bMPCMzHw0cCPwzcBKwAjgzIo4bsPQ7gC91d6daFBHPBj4J/G2JvhvgVGA/OgF/SM9lmJPP3UDnGN5ehwI3DlHz3cB96Xzc4LZfOD8CXjxETWm7KsodM8fMkWbktk5lmpA7Zo5qZ+ZUpgmZA+bO2Gv1x6RHxD9k5p/NsPxhdKakT8/MlQPWfgrwJ8AqOpPlD2bm54dotzEiYgOdKe11BWu+iM4ug/8IrKHzup4IvC4zTx+w5nXAwd1p9frMXNFdfmNm7lqo9YkREVNs/39mMjPbvlfgnKrKHTPHzJk0Zk5/3NapRhNyx8wpy8zpj5lTjSZkTremuVNQFbnT9gHPzZm583ZuX5qZGweo+9DMvMvxcBFxeGZeON96TdPdFfPRmbm2cN1H0fmIxX2Aa4GzMvMbQ9RbA9y/e/zu+sxcERG7A+dn5kFlup4cETHba/ZU4FXAdZl5vxpbGktV5I6ZY+ZMIjOnP27rVKMJuWPmlGXm9MfMqUYTMqdbz9wpqIrcmegBT+m6vVPMNouIV9B5072Pux4jeu4A9RbSOd72sEF+IWyn7jvo7Nb4MuB7wH2A9wCXZubJpR5nUkXE44A3A7sCb6DzC6O9gdKnKnLHzDFzZObMxm2dajQhd8ycapk5MzNzqtGEzOnWNXcqVCJ32j7g2cgcx21m5inzqLeAzkeY3QjszJ0/zuwg4N8zc48BWm2UiLh8lpsyMw8csObPgYdk5k2Dd3aXmkvofP9fBCyj89GN/x/wqiz4EYSTJiKOpLO75/50AujDmbl1tF2Nj5K5Y+Z0mDmTzczZPrd1qtGE3DFzqmHmbJ+ZU40mZE63prlTgZK50/bjSIPOyapK2cLvjpHbMu22KTonx2q9zPy9Csq+B/hURLwVuJqeYxEz85eDFOyGzMuAl3V3HVzr/7wMLiJWA28B7k8ngE43yGdUMnfMHMycSWXm9M1tnQo0IXfMnLLMnL6ZORVoQuZ072fuFFRF7rR9D56iuxBGxCo6ofZt4BE9NyVwfWbeXuqxJk33BFMzycxcOGDNWafdg4baJOt+j9YBH6Uzrb+L+fyPTVuVzB0zpzpmzvgzc/rjtk5zlM4dM6csM6c/Zk5zuK0z/qrInUnYg6eYzFzTvbrqTg8ScTc6E+bWioifZOa9u9evYpazfWfm/oPUz8wFQ7Q3m0vp9Nn7PtjW96ChVvy5R8QJwDcy85ppy5+dmWcNUrMiZ9B57iu7F82sWO6YOWaOmWPm9MFtnUIamDtmTllmTn/MnEIamDlg7pRWPHfavgfPOZl5XAV13wF8KjMvjIgnAJ+h8415VmZ+sfTjjYOIeHhmfrd7/ajZ1svMbw9Y//8AX8rM6btmFhMRewGvB76TmZ8YsMb057438JfA2Zn53gFrbgWuA56Rmf/Zs7ySk9ipWlXkjplj5vQwc3QnbuuU0/TcMXNUBzOnnKZnTvcxzJ1xk5mtvQDLgR17vg46J4R6L3D8EHWvA5Z1r18APA04BvjhqJ9zDa/pQjq7kC0tXPf7wHrgQ8AjK+x/KbCmcM29gO8Pcf8NwOPpnDH/j3uXj/r7Pa3PZ0z7+tBpX//VqHsch0sVuWPmmDnTapo5aeb0vA5u65R/TRubO2bOUM/TzOnvdTJzyr+mjc2c7uOYO4P3WTx3Rv6kKn7BvgQ8t+frd9I5xu3TwG+Alw9Y96buvyvpHBu6bfnNo37ONb2u1wGLK6h7GJ2TTF1G50Rg7wQeXPgx7t/7PStU8+7b3hMD3v/m7r/3BH5M5+MRF47b+2l6P8D67d0+qZcqcsfMMXOm1TRzZrh9Ui9u61T2ujYyd8ycoZ6nmdPf62TmVPO6NjJzuo9h7gzZZ8/XQ+dO2w/Rug64d2be2P1It7XAUzLzmxFxOHBGZt5rgLoX0Tkr+cF0pmx/GBG7AT/KzD1LPodxFBGvBHYFXp+Zmyt6jCOANwGPzsFPAvYd7nw85zLgPsCbM/OtA9Z807RFy4DjgB9k5vED1tyQmTt1r+8CfLxb94jMXDZIzSr09tn9+obMvPtst0+qKnLHzDFzptU0c2a4fVK5rVONJuSOmVOWmdMfM6caTcicbg1zp6AqcqftJ1lelpk3dq+vBrZk5jcBsnN8594D1n0JnQDaBPxxd9njgK8N02yD/AWd3eb+OiKu53cn2soc8ERY20TEfsDxwB/SOdnah4co96FpX98KXJKZvxiiZu/HQiads52/CzhziJrf/G3BzJsi4kl0Piavio9LHMb0afBcX0+qKnLHzDFzwMwxc2bmtk41mpA7Zk5ZZk5/zJxqNCFzwNwprXjutH0Pnp/QOTHXDyLiDcCDMvMp3dt2BX6emXsMWPuxdH5Qds/MJ0XnM+x3zsxzh+j3Md2aexSseQjwbGBf4Bo6J6v6+aD1ujWPmu22HPwkYH9GJ3QeAJwDnAWck5mbBmqyU/PN9PFDkfP86Lme7/0emfnEEt+nJoiIm4Fd6PyyCeB6OrvRbvv6hmz7Scv6UFXuVJE53bpjnztmjpmDmTMrt3Umd1vHzCnLzOmPmTO5mdOtae4UVEXutH0PnncAX4uI/6AzAX5qz22PA34wSNGI+As6Z/b+EJ0TgAHcDvwd8PsFaj69UM0n0dkd7UvAGuBQ4KKIeG5m/ssgNbv+E3gtnWDbB7gWOBs4dYiaTwQ+CHwuM28Zok6vg+l8fy6i8/z3Bw4HPgvc0V1nXhPOiPhz4K8Y8nsfESdn5qnd69N3S/yt+YZjxZYDvWfhj56vA/9na5viuVNF5sxQd5xzx8wxc8DMmY3bOpO7rWPmlGXm9MfMmdzMAXOntOK50+o9eOC309DVwAXZ/Ri67vIn05mIfWeAmpfROXbxim3HyUXEQuA3mTnQ59dXVPOHwEu37TbZXXY0cFpm3neQmt0a/0gnzE6l84O9CngN8IvMfMGgdUuLiLOBT2fmZ3uWPZXO2cqfPWDNIt+niNiSmYu61z9OZ3fUu8jMEwfpswoRsWqudTJzTR29jLvSuVNFPlRVt4rcMXPMnNmYOR1u60zmto6ZU5aZ0z8zZzIzB8yd0qrIndYPeKoQEb8B9s7MrRGxPjNXRMQOwOWZOdBxpxXVvIHOLo5bepYtAtZm5q6D1OzWWAcclL87/paIWAFcmpkr5lHn9Mw8qXv9jNnWy8wTBuzzJmBFZm7tWbYIWJeZuwxYs8j3KSJu3ra7Xe91aSZV5ENVdavIHTPHzFH93NYZ/9wxc9QmZs74Z063prkz5haMuoGqROf4wH7We+MA5c8DXj1t2UvpOZnTmNT8PvDyacv+urt8GL+icxbyXnej8/F+83F5z/XLtnMZ1KV0TtjW68VD1iz1fbosIt4ZES8AFkXEiRHxgumXIfosquKfp9ao8HWqIh+qqltF7pg5d2bm/G49M6e/9dzWmb8m5I6ZU4iZ0x8zB5jszAFzp5iqfp5auwdPRGwA7k/n2LXt+V72fBRZn7X3Br4I7Ebn5Fq/BDYAT8zMXw3QblU179WtuSNwFZ0zlN8GPCkzfzJIzW7dV9M5Ydf7gKu7dV8CfILO8ZgA5BwnxYqIR/XzeHPV2U79BwGfo3OuqWvovK5bgKdm5n8NWLPI9yki7gm8ks7ul48EZtqVNTOzr9doPiLi1Zn5N/O8T2U/T21S1etURT5UVbeK3DFzzJztMHPc1pnYbR0zZ9bHNnMqYuZMduZ065s7Mz/22OROmwc8U/zu4+W2547MnD4t7ad+AA+h8wa6CrgwM6fm3Wj1NRcBR/C7k3VdkJmbh6x5+dxrkZl5YB115niMxfzu+V8H/GeB51/0+xQR38jMRw/T0zwf75zMPG6e96n056ktqnydqsiHquqWzh0zx8zZDjPHbZ2J3tYxc2Z8PDOnImbOb2tObOZ0H8PcuevjjU3utHbAI0mSJEmSNClaew4eSZIkSZKkSTFxA56IOMma412zqrrWHP+abdSU196fZWtOYs028mfZmpNYs6q65k5/mvLaW7O8pvQ6STUnbsADVPHmtmYz6lpz/Gu2UVNee3+WrTmJNdvIn2VrTmLNquqaO/1pymtvzWbUteYQJnHAI0mSJEmS1CqtOMnykliaO7BjX+tuZiOLWTrnerF0Sd+Pv2nrbSxZOPeJrXc/5Oa+a968fgs7r1jU17q/+Z8d+lqv3+c+H1XUrKquNUdXcwM3rM3M3Ys2MGL95s58Xqd73v+2vta7ft1Wdl+5sK91f/HD/rJxExtZMo/3SL+/O0b93rPmZNa8g1vZlBvn+lSKRlkSO+TdFiyfc71NeQdLor/tAhb2lyMAm6ZuZ8mCu8294lT/H3qyKW9nSfRRE6DPzJnP89+6c3+PvXnjLSxeOvdrD7B5l/763LrhVhbu1F8+73DVHX2tN5/nnn1+n0b9szzquv3WbGfmlP/7iuj/Jdqcd7C4j/fzPe93a98157P99PMf9PehRU35GWnjz92k15zt76v+Jghjbgd25KELjilac9E9VhWtB/Ann/9a8ZoAf3/PQ8sXjQp27praWr5mU8zjF9q8NGRA+/X8zJpR91DaDuzIQ6Pspy9+9avfL1oP4NgDjyheE2Dqjv7+4JBG4YL8xqhbKO5uC5ZzxLInFq25YNdditYDyFv7/2NrXnU3byle85ZH3bd4zasfP9SnL8/oXq/4afGaUxs2FK85ydqYOVVs58Ti/v8DvV//+tULi9cEeNw+DyxftKq/B0pryN8Xk262v688REuSJEmSJKnhHPBIkiRJkiQ1nAMeSZIkSZKkhnPAI0mSJEmS1HC1DHgi4oCIyIiY8aTOEfGaiPhQHb1Iaj8zR1KdzBxJdTJzJM2msk/RiogrgBdm5tfnWjcz31pVH5Img5kjqU5mjqQ6mTmS+uEhWpIkSZIkSQ1XyYAnIs4E9ge+GBG3AM/s3vSciLgyItZGxMk9678hIj7Wvb5DRHwsItZFxI0RcVFE7FlFn5LawcyRVCczR1KdzBxJ/apkwJOZzwWuBJ6UmcuBT3VvejhwKPBo4JSIuPcMd38esAuwH7AS+FPg9ukrRcRJEXFxRFy8mY0VPAtJTVFH5oC5I6ljFJmzKe8o/CwkNYXbOZL6VfchWm/MzNsz8xLgEuABM6yzmU74HJyZWzPze5l58/SVMvP0zFydmasXs7TitiU1VLHMAXNH0pwqy5wlsUOFbUtqKLdzJN1J3QOeX/Vcvw1YPsM6ZwJfBc6OiGsj4v+NiMW1dCepbcwcSXUycyTVycyRdCdVDnhyoDtlbs7MN2bmYcDvA08ETijamaQ2MnMk1cnMkVQnM0fSnKoc8PwaOHC+d4qIR0bE/SJiIXAznd0Kp0o3J6l1zBxJdTJzJNXJzJE0pyoHPG8DXhsRNwJPn8f99gI+QyeAfgJ8m86uhZK0PWaOpDqZOZLqZOZImtOiqgpn5heAL/Qsese024/uuf6GnutnAWdV1ZekdjJzJNXJzJFUJzNHUj/qPsmyJEmSJEmSCnPAI0mSJEmS1HAOeCRJkiRJkhqusnPw1C4H+uTAWW25fE3RegBP3vG24jUB/r7wcwcgt5avOcmq+B5ppGLJYhbts1/Rmk948F5F6wHc8PQDitcEWPkfvype89rj9ilec5/PXVG85pbrfl28ZiyI4jVzqqLcyQo+fMWMnNuSxcQB9yhacusO5TcD191/VfGaALt98pLiNZf/4qbiNQ+6YVnxmrFkcfGaRPnMqezneMHC8jWn3M7tS+H3SSwunzmH/cOfFa8JcMDuvyheM5aXz4cbH7J38ZrLP31B8ZqxZEnxmgtWlf2duM3WS68oX7SSbaeZF7sHjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIYbmwFPRPwoIo4edR+SJoOZI6lOZo6kupk70uRZNOoGtsnM+4y6B0mTw8yRVCczR1LdzB1p8ozNHjySJEmSJEkazNgMeCLiiog4JiIOj4iLI+LmiPh1RLxr1L1Jah8zR1KdzBxJdTN3pMkzNgOeHu8F3puZOwMHAZ+aaaWIOKkbVBdvZmOtDUpqlb4yB+6cO5u23l5bg5JaZcDMua22BiW1jn9fSRNiHAc8m4GDI2K3zLwlM8+faaXMPD0zV2fm6sUsrblFSS3SV+bAnXNnycK71diipBYZMHOW1diipJbx7ytpQozjgOePgXsCP42IiyLiiaNuSFKrmTmS6mTmSKqbuSNNiLH5FK1tMvMXwLMjYgHwVOAzEbEyM28dcWuSWsjMkVQnM0dS3cwdaXKM3R48EfFHEbF7Zk4BN3YXT42yJ0ntZeZIqpOZI6lu5o40OcZuDx7g8cC7ImIZsAY4PjM9m6mkqpg5kupk5kiqm7kjTYixGfBk5gHdq18fZR+SJoOZI6lOZo6kupk70uQZu0O0JEmSJEmSND8OeCRJkiRJkhrOAY8kSZIkSVLDjc05eMbNon32Ll7z8fsvLF4TIBZXMKdbEMVLxpIlxWtO3XJL8Zq/efHDitfccGA1H1Rw0CvOr6Su+rBwIVN336loybjm+qL1AFb86y+K1wT4zUdWFq+55x9cXLzmli2bi9es4vfDlmuvK16TzPI1NTJ5x0a2/uhnZYsuKL9dsvKS8tsPAFNbt5avufJuxWsuef2viteME5YVr8m69eVrViX9sKdRiEULWXj3FUVrTt14U9F6APv/bfltB4DYc/fyRTdvKV7y2mPKZ+O9zy37fQeI5TsWr8nGTeVrAkyVf03r5B48kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsPVOuCJiAMiIiNiUZ2PK2kymTmS6mbuSKqTmSOpV+UDnoi4IiKOqfpxJAnMHEn1M3ck1cnMkTQbD9GSJEmSJElquEoHPBFxJrA/8MWIuAV4Zvem50TElRGxNiJO7ll/QUS8OiIui4h1EfGpiFhRZY+S2sPMkVQ3c0dSncwcSdtT6YAnM58LXAk8KTOXA5/q3vRw4FDg0cApEXHv7vK/AP4PcBSwD3AD8Pcz1Y6IkyLi4oi4eDMbK3wWkpqiysyBO+fOpi23VfQsJDWJ2zqS6lRX5myauqPCZyGpKqM6ROuNmXl7Zl4CXAI8oLv8T4GTM/PqzNwIvAF4+kwnDcvM0zNzdWauXszS2hqX1EhDZw7cOXeWLFpWS+OSGsttHUl1Kpo5SxbsUFvjksoZ1dnWf9Vz/TZgeff6KuBzETHVc/tWYE/gmpp6k9Q+Zo6kupk7kupk5kiqZcCT81j3KuAFmfnvVTUjqfXMHEl1M3ck1cnMkTSjOg7R+jVwYJ/rfgA4NSJWAUTE7hHxlMo6k9RGZo6kupk7kupk5kiaUR0DnrcBr42IG4Gnz7Hue4F/Ab4WERuA84GHVtyfpHYxcyTVzdyRVCczR9KMKj9EKzO/AHyhZ9E7pt1+dM/1KeBd3YskzZuZI6lu5o6kOpk5kmYzqk/RkiRJkiRJUiEOeCRJkiRJkhrOAY8kSZIkSVLD1fEx6Y205drryheNauZpC1fuUrzm5S85pHjNqUXz+UTH/hzwuvOL19zrI98vXnOP228vXlOjlXdsJH9yWdGaUxs3Fq1XpZV/sKF4zU9e/u3iNZ+5/8OL19xyzbXFa0p9iShbb2pr2XpAThUvWZl/O+vDxWs+7h4PLl6ziu9To2T57UfNLbdsZevadaNuY2S2/vr64jVXfGvH4jVXvbV4SfKWW4vX3LpuffGampl78EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWq44gOeiDg0Ir4fERsiYioiXlf6MSRpGzNHUp3MHEl1M3ck9auKPXheCXwzM3fKzAWZ+eZBikTEFRFxTOHeJLWPmSOpTmaOpLqZO5L6UsWAZxXwo7lWiohFFTy2pMlj5kiqk5kjqW7mjqS+FB3wRMS5wCOB0yLiloj4RES8pXvb0RFxdUS8KiJ+BXw4InaLiC9FxI0RsT4ivhMRCyLiTGB/4IvdOq8s2aekdjBzJNXJzJFUN3NH0nwUnfJm5qMi4lvAxzLzQxHxkWmr7AWsoDOFXgCcAkcToB4AACAASURBVFwN7N69/YhOmXxuRBwJvDAzvz7TY0XEScBJADuwrOTTkNQQdWYOmDvSpDNzJNXNv68kzUfdn6I1Bbw+Mzdm5u3AZmBvYFVmbs7M72Rm9lMoM0/PzNWZuXoxS6vsWVJzFcscmJY7sUNVPUtqruoyx20dSTPz7ytJv1X3gOf6zLyj5+u3A5cCX4uIX0bEq2vuR1K7mTmS6mTmSKqbuSPpt+oe8NxpepyZGzLz5Zl5IPBk4K8j4tEzrStJAzBzJNXJzJFUN3NH0m/VPeC5k4h4YkQcHBEB3ARspbObIcCvgQNH1pyk1jFzJNXJzJFUN3NHmmwjHfAAhwBfB24B/hP4h8z8Zve2twGv7Z4B/hWjalBSq5g5kupk5kiqm7kjTbCin6IFkJlH91x/fs/1bwH3mLbuu4F3z1LnC8AXSvcnqV3MHEl1MnMk1c3ckdSvUe/BI0mSJEmSpCE54JEkSZIkSWo4BzySJEmSJEkNV/wcPK2RVXyK4NTcqwxg6/XXF6+5/xvXFq959pX/Xrzm8a/738Vr5uYtxWtW837SSGWSGzeOuouRyS2bi9d85v4PL17zq1d/r3jNx+37oOI1zQi1xoKF1dSd2lq85LGHHlm85hsv/Vb5mg84unjNqVtuKV6zshyLKF/TzG2PKt4fQG7eVLzmDY8p/6f3n/73Z4rX/Mh9DyleM5YsKV5z4W4ri9cE2HLNtcVrxqIKxi6zbIq7B48kSZIkSVLDOeCRJEmSJElqOAc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGG3jAExFXRMQxJZuJiG9FxAtL1pTUHuaOpDqZOZLqZOZIGpZ78EiSJEmSJDWcAx5JkiRJkqSGG3bA85CI+HFE3BARH46IHQAi4kURcWlErI+If4mIfbbdISJ+PyIuioibuv/+/kyFI2LviPhBRPzfIXuU1C7mjqQ6mTmS6mTmSBrYsAOe5wCPAw4C7gm8NiIeBbwNeCawN7AGOBsgIlYAXwb+DlgJvAv4ckSs7C0aEb8HfBs4LTPfPtMDR8RJEXFxRFy8mY1DPg1JDWLuSKqTmSOpTmaOpIENO+A5LTOvysz1wKnAs+mE0j9l5n9l5kbg/wEeFhEHAE8AfpGZZ2bmlsw8C/gp8KSemocB3wRen5mnz/bAmXl6Zq7OzNWLWTrk05DUIOaOpDqZOZLqZOZIGtiwA56req6vAfbpXtZsW5iZtwDrgH2n39Zzv317vn4OcA3wmSF7k9RO5o6kOpk5kupk5kga2LADnv16ru8PXNu9rNq2MCJ2pLO74DXTb+u53zU9X78BWAt8IiIWDtmfpPYxdyTVycyRVCczR9LAhh3wvCQi7tE99vNk4JPAWcCJEfHAiFgKvBW4IDOvAM4B7hkRfxgRiyLiWXR2GfxST83NwDOAHYEzIsJP+pLUy9yRVCczR1KdzBxJAxv2h/sTwNeAXwKXAW/JzK8DrwM+C1xH5wRhxwNk5jrgicDL6exW+ErgiZm5trdoZm4CngrsCfyTISSph7kjqU5mjqQ6mTmSBrZo0Dtm5gHdq2+b4bYPAB+Y5X7fBR48y21H91y/Azhm0P4ktY+5I6lOZo6kOpk5kobl5FaSJEmSJKnhHPBIkiRJkiQ1nAMeSZIkSZKkhhv4HDwaQOaoO+hfBb0ev9/vF68J5fs89effLV7zlEc8tXhNgC1XXV1JXc0tFixgwfKditacuuWWovUAFixfXrwmwNSGDeWL5tbiJY898IjiNV/6i+8Xr3naA2Y8dcJQpm69tXhNjVgTtiNyatQd9K2KHHvjA44uXvOnb7938Zr3+GoUr7nzhVcVrwkwtW598ZoLdltZtF78anHReq0V5d93i/bco3hNgK1r1xWvOXX77cVrfvQZjytec8GO1xWv+ZO3HVq85u99tvx2I8Dia66tpG5d3INHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIZzwCNJkiRJktRwDngkSZIkSZIazgGPJEmSJElSw439gCcifhQRR4+6D0mTwcyRVDdzR1KdzBypvRaNuoG5ZOZ9Rt2DpMlh5kiqm7kjqU5mjtReY78HjyRJkiRJkrZv7Ac8EXFFRBwz6j4kTQYzR1LdzB1JdTJzpPYa+0O0ZhMRJwEnAezAshF3I2kS3Cl3YscRdyOp7dzWkVQnM0dqvrHfg2c2mXl6Zq7OzNWLWTrqdiRNgN7cWRI7jLodSS3nto6kOpk5UvM1dsAjSZIkSZKkDgc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDXc2H+KVmYeMOoeJE0OM0dS3cwdSXUyc6T2cg8eSZIkSZKkhnPAI0mSJEmS1HAOeCRJkiRJkhpu7M/BI9XtdUc8sXjN9114dvGaAH+26uGV1NXccmqKqQ0bRt3GnJrQY5Wm7rijeM3T7v+/itfccOx9i9fc52WXFq8JcNPD11VSVy2ROeoO+hdRvOT1x5f/WV71xS3Fay678PLiNdcfc1DxmgArzl9cvOaWNVcXrZdbNxet11oV5MOWX/26eM0myR+X/10/tXVr8Zr3etkPitf89KXfKl4T4Gn7Pax4zdxSPsdn4x48kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcA54JEmSJEmSGm4kA56I+EhEvCUijoyIn821Xp29SWonc0dSncwcSXUycyQBLBrlg2fmd4BDR9mDpMli7kiqk5kjqU5mjjTZPERLkiRJkiSp4WoZ8ETEgyLivyJiQ0R8Etihu/zoiLh6rvUkab7MHUl1MnMk1cnMkTSTygc8EbEE+DxwJrAC+DTwtEHX61n/pIi4OCIu3szGKlqX1FDmjqQ6mTmS6mTmSJpNHXvwHAEsBt6TmZsz8zPARUOsB0Bmnp6ZqzNz9WKWVtK4pMYydyTVycyRVCczR9KM6hjw7ANck5nZs2zNEOtJ0lzMHUl1MnMk1cnMkTSjOgY81wH7RkT0LNt/iPUkaS7mjqQ6mTmS6mTmSJpRHQOe/wS2AC+NiMUR8VTg8CHWk6S5mDuS6mTmSKqTmSNpRpUPeDJzE/BU4PnAeuBZwD8Pup4kzcXckVQnM0dSncwcSbNZVMeDZObFwINmufkefa4nSX0zdyTVycyRVCczR9JM6jhES5IkSZIkSRVywCNJkiRJktRwDngkSZIkSZIazgGPJEmSJElSw9VykuVGWrCweMlYWL4mADlVQcksXnPhil2L14xF5d/CU3uuKF7z+B++oHhNgN2WXV285tRttxWvqRaKKF8zy+dOFX3e9qj7Fq+56PbyOf4/Xzm0eE2AAw79TfGaW39+WdmCFbyV1J9YvKSSurl5UyV1S1vxo4b8Dt2ypXjJu//X2uI1Aa59wr7Fa+597tKi9eKysvVaq4pth6hof4WprdXULWzhPfYpXnNq3Q3Fa3LQfsVLPuq11Zw7fMX/url4zZvuuVPxmnziMzMudg8eSZIkSZKkhnPAI0mSJEmS1HAOeCRJkiRJkhrOAY8kSZIkSVLDOeCRJEmSJElqOAc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGWzTqBgYVEScBJwHswLIRdyNpEpg7kupk5kiqk5kjNV9j9+DJzNMzc3Vmrl7M0lG3I2kCmDuS6mTmSKqTmSM1X2MHPJIkSZIkSepwwCNJkiRJktRwDngkSZIkSZIabuwHPBHxlYh4zaj7kDQZzBxJdTJzJNXN3JHaa+w/RSszjx11D5Imh5kjqU5mjqS6mTtSe439HjySJEmSJEnaPgc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDXc2J9kuW8RRcst2nP3ovUAtq5dX7wmQG7ZUr5o4dcTYOvadcVrLli2rHjNuOOO4jVv+4/7FK8JcN0Ldytec58zf1K8JtW89dungp87MsvXrLJuaRX0ucM53ytek5wqXnLVuUuL1wSYquB3zsev/G7Reo897pai9dS/3Lxp1C30r4J8iP+4pHjNKmytouhNN1dRlb3WXF285mcuPa9ovUcc64ZOXyrZdij/+7NJtlxx5ahb6M8l5f++WPGTJcVrVuXfvvDt4jV3+cTMy92DR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsPNOeCJiIyIWyPi1Doa6nncP46IW7qPf3Cdjy1ptMwdSXUycyTVycyRVJV+9+B5QGaeDBARB3RD4ZZpl2d1b/9I9/bDt905Ig6OiOz5+lsRcUdEbIiImyPiexHx6ohYum2dzPzHzFxe6HlKah5zR1KdzBxJdTJzJBU3zCFau2bm8p7LJ3tuWw+8ZY77/3lm7gTsDbwcOB44JyJiiJ4ktZu5I6lOZo6kOpk5koZS1Tl4PgrcPyKOmmvFzLw1M78FPBl4GPCEfh4gIk6KiIsj4uLNbByqWUmtYO5IqpOZI6lOZo6kOVU14LkNeCvQ93GlmXklcDFwZJ/rn56ZqzNz9WKWzn0HSW1n7kiqk5kjqU5mjqQ5DTPgWRsRN/Zc7j3t9g8C+0fEsfOoeS2wYoieJLWbuSOpTmaOpDqZOZKGMsyAZ7fM3LXn8pPeGzNzI/Dm7qVf+9I5vlSSZmLuSKqTmSOpTmaOpKFUdYjWNh8GdgWeOteKEbEf8GDgOxX3JKndzB1JdTJzJNXJzJE0q0oHPJm5BXg98KrZ1omIZd2ThX0BuBA4p8qeJLWbuSOpTmaOpDqZOZK2Z5gBz40RcUvP5a9nWe8s4LoZlp8WERuAXwPvAT4LPD4zp4boSVK7mTuS6mTmSKqTmSNpKIv6WGcj8L2I+LvMfF1mXgHEbCtn5vOnfT0F3HfasqPnetCIOBF4d/fxDSVpspg7kupk5kiqk5kjqRJzDngyc4c6GpnhcT9M5xhTSRPG3JFUJzNHUp3MHElVqfoky5IkSZIkSaqYAx5JkiRJkqSGi8wcdQ9Di4jrgTV9rr4bsLZwC9Ysrym9WrM/qzJz98KPP1LzyJ1Rv/ajrFlVXWtacy5mTnmj/p5a05qjqNtvzUnOHGjO+8Sa5TWl1zbWnDF3WjHgmY+IuDgzV1tzfGtWVdea41+zjZry2vuzbM1JrNlG/ixbcxJrVlXX3OlPU157a07uz90k1fQQLUmSJEmSpIZzwCNJkiRJktRwkzjgOd2aY1+zqrrWHP+abdSU196fZWtOYs028mfZmpNYs6q65k5/mvLaW7MZda05hIk7B4/aIyJuAf48Mz8y6l4ktZ+ZI6lOZo6kOpk57TCJe/CogIj4SER8aYblqyMiI+KA+ru6Sy/3iYjPRMQvuz29YdQ9SRpMQzLnRRHxnYi4ISJujIhvRsTDR92XpPlrSOY8IyIu7ubNrRHx/Yh43qj7kjR/TcicXhHx7G5fd+lZo+WAR222DLgCeC1w+WhbkTQBjgY+CTwKeCjwM+CrEXHIKJuS1FrrgLcARwD3Bz4M/GNEHDfSriS1WkQcCLwd+M6oe9FdOeBR5SLisIj4ckRsiIjfRMRZEbFXz+0PiYivRcTaiLg5Ir4bEQ+bVuPgiPhWRNwRET+LiCfO9biZeVFmviIzPwHcVsFTkzSGRpg5z8nM0zLzvzPzZ8CLgQ3A44s/SUljY4SZc25mfj4zf5qZl2Xme4EfAEeWf5aSxsWoMqd7v8XAWcDJwC+LPjEV4YBHlYqIvYHzgP8BDgeOAZYDX4iIbe+/nYAz6WyQHA58HzgnIlZ2aywAPkfn/fow4AXAG4CltT0RSY0wZpmzBNgBuGHwZyRpnI1L5kTHo4FDu/1IaqExyJxTgSsy86OFnpIKWzTqBtRoj4/Oybh6TR8avhi4JDNftW1BRJwArAdWAxdm5rm9d4iIvwCeBhwLfIxOcB0G/F5mXtld569wt0Bp0jQtc94C3AL8yzzvJ2k8jH3mRMQuwDV0/jDbCrwkM7/S9zOUNE7GOnMi4rHAM4EHzvN5qUYOeDSM84CTpi27L52J8DYPBh4xQ1gBHARcGBF7AG8GHgnsCSwE7gbs313v3sA12wKo6wJgauhnIKlJGpM5EfGXwJ8Ax2Tmzf3eT9JYaULmbKDzx9Zy4NHAuyLiisz8Rh/3lTRexjZzImJ34CPAszPzxn6fkOrngEfDuC0zL+1dEBG7TltnAfBl4BUz3P/X3X8/Sid8XkbnpMgbgW/QObxBkrZpROZ0/xfszcCxmXlhiZqSRmLsMyczp4BtPX4/Iu4NvKZbX1KzjHPm3AfYG/hGRPT2QkRsAe7TPf+gRswBj6r2X3R25VuTmZtnWefhwEsz88sAEbEnnQDZ5ifAvhGxX2Ze1V12OJ5DStJdjTRzIuKvgTcCT8jM7w74HCQ1x7ht5yzAcxRKbTaqzLkIuN+0ZW8B7g68BD+xeGz4B7Kq9vfALsAnI+KhEXFgRBwTEadHxE7ddX4O/FH3jPAPAc4GNvXU+DrwU+CMiHhg9yzw7wa2bO+BI2JJd/0H0jnR6V7drw8u/BwljY9RZs7/Bf4G+GPg5xGxV/eyS9mnKGmMjDJzTu4+1oERce+IeDnwXDrn2JDUTiPJnMy8NTP/p/cC3Ahs6H69abb7ql4OeFSpzLwW+N90jun8V+BHdIJpY/cCnTO3Lwe+RyeA/onO7oTbakwBf0Dn/XoBcAadifG2+89mH+C/u5eD6JwP47+BDw39xCSNpRFnzkuAxcAnget6Lu8d+olJGksjzpzlwPu7j/nvdE6iekJmfmD4ZyZpHI04c9QAkZmj7kGSJEmSJElDcA8eSZIkSZKkhnPAI0mSJEmS1HAOeCRJkiRJkhrOAY8kSZIkSVLDOeCRJEmSJElqOAc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIZzwCNJkiRJktRwDngkSZIkSZIazgGPJEmSJElSwzngkSRJkiRJajgHPJIkSZIkSQ3ngEcDiYgTI+LciPhZ998TR92TpPYycyTVzdyRVCczRyUsGnUDap6IOBk4AXgnsAZYBbwyIvbJzFNH2pyk1jFzJNXN3JFUJzNHpURmjroHNUxEXA4cnZlrepatAs7LzFWj60xSG5k5kupm7kiqk5mjUlq7B09EvKnPVTdn5psrbaZ9dgSun7ZsHXC3EfQijQ1zpzJmjjQDM6dS5o40jZlTKTNHRbR2D56I2AR8vI9Vn56ZO1XdT5tExBnATsCrgSvp7EJ4KnBbZj53lL1Jo2TuVMPMkWZm5lTH3JHuysypjpmjUto84NnQT7BExA2Zefc6emqLiNgZOA14Fp29wDYDnwJempk3jrI3aZTMnWqYOdLMzJzqmDvSXZk51TFzVEqbBzxLMnNTqfXUERELgdcDbwU2AbsBazNzaqSNSWPA3CnPzJFmZ+ZUw9yRZmbmVMPMUUmtHfAARMSXM/MJo+6jbSJiLbCHoSPdlblTnpkjzc7MqYa5I83MzKmGmaNSFoy6gYodOeoGWuoM4E9H3YQ0psyd8swcaXZmTjXMHWlmZk41zBwV0fY9eG7OzJ1H3UfbRMR3gYcC1wBXAb99E2XmI0bVlzQOzJ3yzBxpdmZONcwdaWZmTjXMHJXS9gHPJuDs7a2TmSfU1E5rRMTzZrstMz9aZy/SuDF3yjNzpNmZOdUwd6SZmTnVMHNUyqJRN1CxBC4bdRNtY8hI22XuFGbmSNtl5lTA3JFmZeZUwMxRKW3fg8ddCAuJiOdm5pnd6y+Ybb3M/Kf6upLGj7lThpkj9cfMKcfckeZm5pRj5qgKbR/wbMjMnUbdRxtExDmZeVz3+jdnWS0z81E1tiWNHXOnDDNH6o+ZU465I83NzCnHzFEV2j7geX9mvniW2+4HnJKZz6i5LalxIuLAPledyswrquxl3Jk70vDMnP6ZOdLwzJz+mTlSGVXlTtsHPDsCrwYeCPwCeAOwG/BO4DHARzPzJSNrsMEiYiVwHLBXZr49IvYBFmTm1SNuTRWIiCk6x1zHHKvenpk71tDS2DJ3qmHmTBYzp39mTnXMnclh5vTPzKmOmTNZqsqdBUN1Nf7eBzwJ+DFwDPBZ4NvAj4ADDJ/BRMRRwM+A5wCndBcfArx/ZE2pardm5sLMXLC9C7Bp1I2OAXOnMDNnIpk5/TNzKmDuTBwzp39mTgXMnIlUSe60/VO0Hg88MDN/ExHvA64EjsrM74y4r6Z7D/CszPxGRNzQXXYBcPgIe1K1/rLP9V5WaRfNYO6UZ+ZMHjOnf2ZONcydyWLm9M/MqYaZM3kqyZ22H6J1p7O8e9b3MiLihsy8e/f6+sxcERELgOszc+WI25NGytwpz8yRZmfmVMPckWZm5lTDzFEpbd+DZ1FEPJKe49qmf52Z546isYb7cUQ8LjO/2rPsGOCHo2pI9YmI44AjgRXAeuC8zPzKaLsaK+ZOeWbOBDNz5mTmVMPcmVBmzpzMnGqYOROsZO60fQ+eK+icuGg2mZn9nr1aXRFxBPAl4MvAM4Ez6ByL+5TMvGiUvak6EbEEOAd4GHAxcC2wL/Bg4Hzg2Myc+GPTzZ3yzJzJZOb0x8yphrkzecyc/pg51TBzJlMVudPqAY+qExH70jkJ2CrgKuBjnuG93SLiVcDxwJMz86qe5fsBnwc+nZl/M6r+1G5mzuQxczRq5s5kMXM0ambO5KkidyZywNM9nvFY4HmZ+cxR9yM1QURcBLw8M8+b4bajgHdl5oPr76wZzB1pfsyc4Zg50vyYOcMxc6T5qyJ32n4OnjuJiAcAzwP+EFhGZ9e3YertB+ybmecXaK8xImIX4KXAg4Dlvbdl5mNH0pTqcAgw2y6iFwIH19hLY5TMHTPHzJkwZs4A3NYpw9yZSGbOAMycMsyciVU8d1o/4ImIPYA/ohM8hwHnATsC98vMKwasuT9wFvBAOsegLo+IpwOPz8wXluh7zH0aWAh8Dri9ZOHua7svcE1mXlmytoYWmTnj9zszb4+ImW6aSKVzx8wxcyaUmdMnt3UqYe5MHjOnT2ZOJcycyVQ8d1o94ImILwOPAX4AfBQ4OzOvjYjrgNuGKP1BOifAOhJY1132b8A7h6jZJEcAu5U80VxE7A2cTecEU+uAlRFxPnB8Zl5b6nE0lCURcSI9n5IwzeI6mxlXFeWOmWPmTCIzpw9u61TG3Jk8Zk4fzJzKmDmTqXjutHrAAxwF3Ax8BTin4Bv5cOAJmTkVEQmQmTd1d62bBN8F7kUn2Et5P3AJcFxm3hoROwJvBT4APLng42hwFwAnzHG7qskdM8fMmURmTn/c1qmGuTN5zJz+mDnVMHMmU/HcafuAZ0/gaXR2H3xNRFwCfJzOJGyYs0v/ms7xcD/ftiAiDgMmZZe35wPnRMQFdF6L38rMNw1Y8+HA3pm5uVvn1oh4JXDNoE1GxFLgFODZwMrM3CUiHgvcMzNPG7TupMrMo0fdQ0NUkTtmjpkzccycvrmtU43nM+a5Y+aUZeb0zcypxvMZ88wBc6e0KnJnQemC4yQzb83MMzLz0cCBwD8DJwErgDMj4rgBS78D+FJ3d6pFEfFs4JPA35bouwFOBfajE/CH9FyGOfncDXSO4e11KHDjEDXfDdyXzscNbvuF8yPgxUPUlLarotwxc8wcaUZu61SmCblj5qh2Zk5lmpA5YO6MvVZ/THpE/ENm/tkMyx9GZ0r69MxcOWDtpwB/AqyiM1n+YGZ+foh2GyMiNtCZ0l5XsOaL6Owy+I/AGjqv64nA6zLz9AFrXgcc3J1Wr8/MFd3lN2bmroVanxgRMcX2/2cmM7PtewXOqarcMXPMnElj5vTHbZ1qNCF3zJyyzJz+mDnVaELmdGuaOwVVkTttH/DcnJk7b+f2pZm5cYC6D83MuxwPFxGHZ+aF863XNN1dMR+dmWsL130UnY9Y3Ae4FjgrM78xRL01wP27x++uz8wVEbE7cH5mHlSm68kREbO9Zk8FXgVcl5n3q7GlsVRF7pg5Zs4kMnP647ZONZqQO2ZOWWZOf8ycajQhc7r1zJ2CqsidiR7wlK7bO8Vss4h4BZ033fu46zGi5w5QbyGd420PG+QXwnbqvoPObo0vA74H3Ad4D3BpZp5c6nEmVUQ8DngzsCvwBjq/MNobKH2qInfMHDNHZs5s3NapRhNyx8yplpkzMzOnGk3InG5dc6dCJXKn7QOejcxx3GZmnjKPegvofITZjcDO3PnjzA4C/j0z9xig1UaJiMtnuSkz88ABa/4ceEhm3jR4Z3epuYTO9/9FwP/f3p2HSVbX9x5/f3tWh0FxAJF9FUSNSxwRjUZUElBBDVHconGdG5O4J+oVFVDU3LhFo7lm3EUFiV4vgphw3dEgMBjRIC6AbMMmDMsMMD0z3d/7R9VI0XZPV3f9zqk6Ve/X8/Qz1VWnP/Wrqu7PVH/71KlltN668ePAm7LgWxCOmoh4PK3dPfeiVUCfzsyJ/q5qcJTsHTunxc4ZbXbOtvlcpxpN6B07pxp2zrbZOdVoQue0M+2dCpTsnWF/HWnQOlhVKVu4+zVyW6ZcNknr4FhDLzP3rSD2n4DTIuLdwDV0vBYxMy+fT2C7ZF4HvK696+BN/uVl/iJiJXAS8FBaBbTaIp9Wyd6xc7BzRpWd0zWf61SgCb1j55Rl53TNzqlAEzqn/XX2TkFV9M6w78FTdBfCiNibVql9D/jjjosS+G1m3lXqukZN+wBT08nMXDDPzBmn3fMttVHWfoxuBj5La1r/e+byF5thVbJ37Jzq2DmDz87pjs91mqN079g5Zdk53bFzmsPnOoOvit4ZhT14isnMK9sn977HlUTci9aEeWhFxCWZeXD79NXMcLTvzNxrPvmZOdbD8mZyXemreAAAIABJREFUKa11dn4fbF33fEut+G2PiBcB38rMtVPOf15mnjKfzIp8jtZt37H9oekV6x07x86xc+ycLvhcp5AG9o6dU5ad0x07p5AGdg7YO6UV751h34PnrMx8agW57wNOy8zzI+JpwJdpPTDPycwzSl/fIIiIx2XmD9qnnzDTdpn5vXnmPxM4MzOn7ppZTETcHzgeOCczvzjPjKm3fVfgNcCpmfmheWZOANcBz87MczvOr+QgdqpWFb1j59g5Hewc3YPPdcppeu/YOaqDnVNO0zunfR32zqDJzKH9AJYD23V8HrQOCPUh4Lk95F4HLGufPg/4c+Bw4Gf9vs013KcLaO1CtqRw7k+AdcAngCdWuP4lwJWFM+8P/KSHr18PHEnriPkv6zy/34/3lHU+e8rnB035/LX9XuMgfFTRO3aOnTMl085JO6fjfvC5Tvn7tLG9Y+f0dDvtnO7uJzun/H3a2M5pX4+9M/91Fu+dvt+oiu+wM4EXdnz+flqvcfs34EbgDfPMva397460Xhu69fzb+32ba7pfrwMWVZD7IFoHmbqM1oHA3g88svB1PLTzMSuUed+t3xPz/Prb2/8eCPyc1tsjLhi076ep6wHWbevyUf2oonfsHDtnSqadM83lo/rhc53K7tdG9o6d09PttHO6u5/snGru10Z2Tvs67J0e19nxec+9M+wv0boOODgzb22/pdtNwDMy8zsRcQjwucx84DxyL6B1VPIDaE3Znh8ROwEXZ+YuJW/DIIqINwI7AMdn5uaKruNQ4B3Ak3P+BwE7h3u+nnMZ8GDgnZn57nlmvmPKWcuApwI/zcznzjNzfWZu3z59H+AL7dxDM3PZfDKr0LnO9ue3ZOZ9Z7p8VFXRO3aOnTMl086Z5vJR5XOdajShd+ycsuyc7tg51WhC57Qz7J2CquidYT/I8rLMvLV9eiWwJTO/A5Ct13fuOs/cv6FVQJuAl7XPOwI4u5fFNsiraO029/qI+C13H2grc54HwtoqIvYEngs8n9bB1j7dQ9wnpnx+B3BRZv66h8zOt4VMWkc7/wBwcg+Z3/ldYOZtEXE0rbfJq+LtEnsxdRo82+ejqoresXPsHLBz7Jzp+VynGk3oHTunLDunO3ZONZrQOWDvlFa8d4Z9D55LaB2Y66cRcQLwiMx8RvuyHYBfZeb95pn9p7R+UHbOzKOj9R72987Mb/ew3j9pZ96vYOYDgOcBuwNraR2s6lfzzWtnPmGmy3L+BwH7a1ql8zDgLOAU4KzM3DSvRbYy30kXPxQ5x7ee63js75eZR5V4nJogIm4H7kPrP5sAfktrN9qtn9+Sw37Qsi5U1TtVdE47d+B7x86xc7BzZuRzndF9rmPnlGXndMfOGd3OaWfaOwVV0TvDvgfP+4CzI+I/aU2Aj+m47Ajgp/MJjYhX0Tqy9ydoHQAM4C7gw8BjC2Q+q1Dm0bR2RzsTuBI4CLggIl6YmV+bT2bbucBbaRXbbsC1wKnAu3rIPAr4V+Crmbmhh5xOB9B6fC6gdfv3Ag4BvgJsbG8zpwlnRPwt8Fp6fOwj4rjMfFf79NTdEn9nruVYseVA51H4o+PzwL9sbVW8d6ronGlyB7l37Bw7B+ycmfhcZ3Sf69g5Zdk53bFzRrdzwN4prXjvDPUePPC7aehK4Lxsvw1d+/yn05qInTOPzMtovXbxiq2vk4uIBcCNmTmv96+vKPNnwKu37jbZPu8w4COZ+ZD5ZLYzPkmrzN5F6wd7b+AtwK8z86XzzS0tIk4F/i0zv9Jx3jG0jlb+vHlmFnmcImJLZi5sn/4Crd1Rf09mvmQ+66xCROw92zaZeWUdaxl0pXunin6oKreK3rFz7JyZ2DktPtcZzec6dk5Zdk737JzR7Bywd0qroneGfsBThYi4Edg1MyciYl1mroiIpcBvMnNerzutKPMWWrs4buk4byFwU2buMJ/MdsbNwP559+tviYgVwKWZuWIOOaszc1X79Odm2i4zXzTPdd4GrMjMiY7zFgI3Z+Z95plZ5HGKiNu37m7XeVqaThX9UFVuFb1j59g5qp/PdQa/d+wcDRM7Z/A7p51p7wy4sX4voCrRen1gN9udOI/47wNvnnLeq+k4mNOAZP4EeMOU817fPr8X19M6Cnmne9F6e7+5+E3H6cu28TFfl9I6YFunV/aYWepxuiwi3h8RLwUWRsRLIuKlUz96WGdRFf88DY0K76cq+qGq3Cp6x865Jzvn7u3snO6287nO3DWhd+ycQuyc7tg5wGh3Dtg7xVT18zS0e/BExHrgobReu7YtF2bHW5F1mb0rcAawE62Da10OrAeOyszr57HcqjIf2M7cDria1hHK7wSOzsxL5pPZzn0zrQN2/TNwTTv3b4Av0no9JgA5y0GxIuJJ3VzfbDnbyH8E8FVax5paS+t+3QIck5k/nmdmkccpIg4E3khr98snAtPtypqZ2dV9NBcR8ebM/Ic5fk1lP0/DpKr7qYp+qCq3it6xc+ycbbBzfK4zss917JwZr9vOqYidM9qd0863d6a/7oHpnWEe8Exy99vLbcvGzJw6Le0mP4BH0foGuho4PzMn57zQ6jMXAody98G6zsvMzT1m/mb2rcjM3K+OnFmuYxF33/7rgHML3P6ij1NEfCszn9zLmuZ4fWdl5lPn+DWV/jwNiyrvpyr6oarc0r1j59g522Dn+FxnpJ/r2DnTXp+dUxE753eZI9s57euwd37/+gamd4Z2wCNJkiRJkjQqhvYYPJIkSZIkSaNi5AY8EbHKzMHOrCrXzMHPHEZNue/9WTZzFDOHkT/LZo5iZlW59k53mnLfm1leU9Y6SpkjN+ABqvjmNrMZuWYOfuYwasp978+ymaOYOYz8WTZzFDOryrV3utOU+97MZuSa2YNRHPBIkiRJkiQNlaE4yPLiWJpLY7uutt2cG1kUS2fdbv8/WN/19d908yQ77Tj7rOyyny3vOnNzjrMolnS3cZcP4WbGWUSXmV2qIrOqXDP7l7meW27KzJ2LLqDPFseSXMrsvTOX+2mPP9jQ1Xa3rJvkviu6m89f02XvDMLPcixc2NV2mybvYvHYvbraNrds6Wq7fv+MmFk2cyN3sCnHZ3tXikbpZ+dA973TbedA/79PYtGirrabU+ds7u6NZPp924cts6rcbjNHuXOg+/tpnzn8fnXzukl27KJzrvjZ9l1nzul7JLp7OLv93bIV2V3mptzI4i4zc7K7N5oaxp+7Uc+c6fer7p5ND7ilsR2HLjqyaOaXvvG9onkAz9nn8cUzoftfYKR++WZ++cp+r6G0pWzHo6Psuy/+rzPOK5oH8Kb9Di2eCUAFfxxYsNP9imdO3HBj8UwNvvPyW/1eQnGN6Zx9H108syoLd9mteOaWtdcWz9TgG9rOGTu8aObHzzqnaB7AK/au5verWLy4fGaXf8iai8k77iieqWaY6fcrX6IlSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWq4WgY8EbFPRGRETHtkqYh4S0R8oo61SBp+do6kOtk5kupk50iaSWXvohURVwAvz8xvzrZtZr67qnVIGg12jqQ62TmS6mTnSOqGL9GSJEmSJElquEoGPBFxMrAXcEZEbACObV/0goi4KiJuiojjOrY/ISI+3z69NCI+HxE3R8StEXFBROxSxTolDQc7R1Kd7BxJdbJzJHWrkgFPZr4QuAo4OjOXA6e1L3occBDwZODtEXHwNF/+l8B9gD2BHYG/Au6aulFErIqINRGxZnNurOBWSGqKOjoHpvQO44VvhaSmsHMk1cnOkdStul+idWJm3pWZFwEXAQ+bZpvNtMrngMycyMwLM/P2qRtl5urMXJmZKxfF0oqXLamhinUOTOkdllS4bEkNZedIqpOdI+ke6h7wXN9x+k5g+TTbnAz8B3BqRFwbEf8YEYtqWZ2kYWPnSKqTnSOpTnaOpHuocsCT8/qizM2ZeWJmPgh4LHAU8KKiK5M0jOwcSXWycyTVyc6RNKsqBzw3APvN9Ysi4okR8QcRsQC4ndZuhZOlFydp6Ng5kupk50iqk50jaVZVDnjeA7w1Im4FnjWHr7s/8GVaBXQJ8D1auxZK0rbYOZLqZOdIqpOdI2lWC6sKzszTgdM7znrflMsP6zh9QsfpU4BTqlqXpOFk50iqk50jqU52jqRu1H2QZUmSJEmSJBXmgEeSJEmSJKnhHPBIkiRJkiQ1XGXH4KndWBSNO3aPxxTNA/ja2nOLZwI8ffdHVZIraRZRtneOe+Kzi+YB3PoXuxfPBFjxlYuKZ17yrr2KZx74it8WzyTn9U619Sv8/Vmpptyn/daAzolF1xfPBCDLv+nPz0/crXjmzufsXTzzvp+t5vljY1TRZXbO7CKIxYuLRr7y0XM5NnR3xo/cs3gmwFVHLiie+cB3X148c2xionjm5MaNxTOrsHDPPSrJnbj+xuKZuXlT8cyZuAePJEmSJElSwzngkSRJkiRJajgHPJIkSZIkSQ3ngEeSJEmSJKnhHPBIkiRJkiQ1nAMeSZIkSZKkhhuYAU9EXBwRh/V7HZJGg50jqU52jqS62TvS6FnY7wVslZkP7vcaJI0OO0dSnewcSXWzd6TRMzB78EiSJEmSJGl+BmbAExFXRMThEXFIRKyJiNsj4oaI+EC/1yZp+Ng5kupk50iqm70jjZ6BGfB0+BDwocy8N7A/cNp0G0XEqnZRrdmcG2tdoKSh0lXnwJTeYby2BUoaKnaOpLr5+5U0IgZxwLMZOCAidsrMDZn5o+k2yszVmbkyM1cuiqU1L1HSEOmqc2BK77CkxiVKGiJ2jqS6+fuVNCIGccDzMuBA4BcRcUFEHNXvBUkaanaOpDrZOZLqZu9II2Jg3kVrq8z8NfC8iBgDjgG+HBE7ZuYdfV6apCFk50iqk50jqW72jjQ6Bm4Pnoj4i4jYOTMngVvbZ0/2c02ShpedI6lOdo6kutk70ugYuD14gCOBD0TEMuBK4LmZeVef1yRpeNk5kupk50iqm70jjYiBGfBk5j7tk9/s5zokjQY7R1Kd7BxJdbN3pNEzcC/RkiRJkiRJ0tw44JEkSZIkSWo4BzySJEmSJEkNNzDH4OlFjAVjS5YUzcwFC4rmAfzZwU8ungnwkAs3FM+8+LGLimcyMVE8MhY241t4ctPmSnKjgu/T3LypeObQirIz8lxU/vt5h8//qHgmAMuWFY981h9eWDzzZ4sXF8+MiOKZk+PjxTPJLJ8JUMHtV5cKP6YT11xbNA8gt2wpngnAWPn/745/3NeKZ572D39UPLP8s6eKVPAYVSYbc6/2Tya5qexzwslbbp19ozla8u83FM8E2GPJo4pnXvqa/Ytn7vaD8p275KwLimeObb998cwtV19TPHMYuAePJEmSJElSwzngkSRJkiRJajgHPJIkSZIkSQ3ngEeSJEmSJKnhHPBIkiRJkiQ1nAMeSZIkSZKkhqt1wBMR+0RERkQz3ttaUqPZOZLqZu9IqpOdI6lT5QOeiLgiIg6v+nokCewcSfWzdyTVyc6RNBNfoiVJkiRJktRwlQ54IuJkYC/gjIjYABzbvugFEXFVRNwUEcd1bD8WEW+OiMsi4uaIOC0iVlS5RknDw86RVDd7R1Kd7BxJ21LpgCczXwhcBRydmcuB09oXPQ44CHgy8PaIOLh9/quAZwJPAHYDbgE+Ol12RKyKiDURsWbT5MYKb4Wkpqiyc+CevbOZ8YpuhaQmqeu5jp0jCewcSdvWr5donZiZd2XmRcBFwMPa5/8VcFxmXpOZ48AJwLOmO2hYZq7OzJWZuXLx2NLaFi6pkXruHLhn7yxiSS0Ll9RYRZ/r2DmSZmHnSKJfR1u/vuP0ncDy9um9ga9GxGTH5RPALsDamtYmafjYOZLqZu9IqpOdI6mWAU/OYdurgZdm5g+rWoykoWfnSKqbvSOpTnaOpGnV8RKtG4D9utz2Y8C7ImJvgIjYOSKeUdnKJA0jO0dS3ewdSXWycyRNq44Bz3uAt0bErcCzZtn2Q8DXgLMjYj3wI+DRFa9P0nCxcyTVzd6RVCc7R9K0Kn+JVmaeDpzecdb7plx+WMfpSeAD7Q9JmjM7R1Ld7B1JdbJzJM2kX++iJUmSJEmSpEIc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcJUfZLkOOTHJxO2393sZffPfjyyf+R/Xnlc884jdH1E8M7dsKZ7ZJDk50e8ljLbC9//Ery8vmlelyTvuKJ753vv/pHjmEeMPL56ZxRMbJkf+Hhgajfo/tIL/71587xuLZ55y6W+KZzaGz0mGT+G+H9t5p6J5AJPXrC2eCbDdty4pnvm8E9cVz1zz4T2KZ04sLD8imNywoXimpucePJIkSZIkSQ3ngEeSJEmSJKnhHPBIkiRJkiQ1nAMeSZIkSZKkhnPAI0mSJEmS1HAOeCRJkiRJkhqu+IAnIg6KiJ9ExPqImIyIt5W+Dknays6RVCc7R1Ld7B1J3apiD543At/JzO0zcywz3zmfkIi4IiIOL7w2ScPHzpFUJztHUt3sHUldqWLAszdw8WwbRcTCCq5b0uixcyTVyc6RVDd7R1JXig54IuLbwBOBj0TEhoj4YkSc1L7ssIi4JiLeFBHXA5+OiJ0i4syIuDUi1kXEORExFhEnA3sBZ7Rz3lhynZKGg50jqU52jqS62TuS5qLolDcznxQR3wU+n5mfiIjPTNnk/sAKWlPoMeDtwDXAzu3LD23F5Asj4vHAyzPzm9NdV0SsAlYBLGVZyZshqSHq7Bywd6RRZ+dIqpu/X0mai7rfRWsSOD4zxzPzLmAzsCuwd2ZuzsxzMjO7CcrM1Zm5MjNXLmJJlWuW1FzFOgfsHUmzsnMk1c3fryT9Tt0Dnt9m5saOz98LXAqcHRGXR8Sba16PpOFm50iqk50jqW72jqTfqXvAc4/pcWauz8w3ZOZ+wNOB10fEk6fbVpLmwc6RVCc7R1Ld7B1Jv1P3gOceIuKoiDggIgK4DZigtZshwA3Afn1bnKShY+dIqpOdI6lu9o402vo64AEeAHwT2ACcC/xLZn6nfdl7gLe2jwD/d/1aoKShYudIqpOdI6lu9o40woq+ixZAZh7WcfrFHae/C+wxZdsPAh+cIed04PTS65M0XOwcSXWycyTVzd6R1K1+78EjSZIkSZKkHjngkSRJkiRJajgHPJIkSZIkSQ1X/Bg8mlksrObuzsny73h45N6HFM/85JXfLp75sr0fXzyTqGDuOTlRPlMaMk996JNn32iO3nvF14tn/v0+hxbPJKJ8ZvpuuNK2HLH7I4pnfm3t+cUzn75nBZ3j8xL1wZarr+n3Ero2uX598cwfPWxR8cyz1n6jeOZT93hk8cxKfr9Ke2w67sEjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4eY94ImIKyLi8JKLiYjvRsTLS2ZKGh72jqQ62TmS6mTnSOqVe/BIkiRJkiQ1nAMeSZIkSZKkhut1wPOoiPh5RNwSEZ+OiKUAEfGKiLg0ItZFxNciYretXxARj42ICyLitva/j50uOCJ2jYifRsTf97hGScPF3pFUJztHUp3sHEnz1uuA5wXAEcD+wIHAWyPiScB7gGOBXYErgVMBImIF8HXgw8COwAeAr0fEjp2hEbEv8D3gI5n53umuOCJWRcSaiFizmfEeb4akBrF3JNXJzpFUJztH0rz1OuD5SGZenZnrgHcBz6NVSp/KzB9n5jjwP4HHRMQ+wNOAX2fmyZm5JTNPAX4BHN2R+SDgO8Dxmbl6pivOzNWZuTIzVy5iSY83Q1KD2DuS6mTnSKqTnSNp3nod8FzdcfpKYLf2x5Vbz8zMDcDNwO5TL+v4ut07Pn8BsBb4co9rkzSc7B1JdbJzJNXJzpE0b70OePbsOL0XcG37Y++tZ0bEdrR2F1w79bKOr1vb8fkJwE3AFyNiQY/rkzR87B1JdbJzJNXJzpE0b70OeP4mIvZov/bzOOBLwCnASyLi4RGxBHg3cF5mXgGcBRwYEc+PiIUR8Rxauwye2ZG5GXg2sB3wuYjwnb4kdbJ3JNXJzpFUJztH0rz1+sP9ReBs4HLgMuCkzPwm8DbgK8B1tA4Q9lyAzLwZOAp4A63dCt8IHJWZN3WGZuYm4BhgF+BTlpCkDvaOpDrZOZLqZOdImreF8/3CzNynffI901z2MeBjM3zdD4BHznDZYR2nNwKHz3d9koaPvSOpTnaOpDrZOZJ65eRWkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIab9zF4Bs5Y4Xf8y8myeQALqnlXwrF7LS6eOblhQ/HMVUe+tHjmTV8rHsnOJywqnpkXXlw8UwMgonBeBTP3KrqsIpPry/fOsee/onjmfgdvLJ45uax8j49dcV3xTIDJ28s/Trl5U/FMzS6WLCmemePjxTOB8n0LLNx36jtL9+6Ii8tnLtv5zuKZsbT8Yz9x/Y3FM6tS2feptikWlv/VM7dsKZ4J1ay1is497FWvLJ55+2vL/8667Ibyz0d3OKOa368m16+vJLcu7sEjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4QZ+wBMRF0fEYf1eh6TRYOdIqpu9I6lOdo40vBb2ewGzycwH93sNkkaHnSOpbvaOpDrZOdLwGvg9eCRJkiRJkrRtAz/giYgrIuLwfq9D0miwcyTVzd6RVCc7RxpeA/8SrZlExCpgFcBSlvV5NZJGgb0jqU52jqQ62TlS8w38HjwzyczVmbkyM1cuYkm/lyNpBNg7kupk50iqk50jNV9jBzySJEmSJElqccAjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsMN/LtoZeY+/V6DpNFh50iqm70jqU52jjS83INHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIYb+GPwdG1yot8rmFWOjzcqt7SJS35dPHOXF9+3eGYs36545vN/eXXxTIDPHbRnJbnqUmbhvMHvsSrFwvL/Je3/6huLZ07eelvxzDho3+KZl350j+KZAAeceFfxzCr+f9DsmvL8ASjft8DE2uuKZy49ungkE1U8TmMLike++7Jzi2cCvPXBhxXPLP/dpG7kli39XkLXqlhrFZnLvnp++cwK+raK53jfuGpN8UyAI3Z7eCW5dXEPHkmSJEmSpIZzwCNJkiRJktRwDngkSZIkSZIazgGPJEmSJElSwzngkSRJkiRJajgHPJIkSZIkSQ3XlwFPRHwmIk6KiMdHxC9n267OtUkaTvaOpDrZOZLqZOdIAij/hvRzkJnnAAf1cw2SRou9I6lOdo6kOtk50mjzJVqSJEmSJEkNV8uAJyIeERE/joj1EfElYGn7/MMi4prZtpOkubJ3JNXJzpFUJztH0nQqH/BExGLg/wInAyuAfwP+fL7bdWy/KiLWRMSazYxXsXRJDWXvSKqTnSOpTnaOpJnUsQfPocAi4J8yc3Nmfhm4oIftAMjM1Zm5MjNXLmJJJQuX1Fj2jqQ62TmS6mTnSJpWHQOe3YC1mZkd513Zw3aSNBt7R1Kd7BxJdbJzJE2rjgHPdcDuEREd5+3Vw3aSNBt7R1Kd7BxJdbJzJE2rjgHPucAW4NURsSgijgEO6WE7SZqNvSOpTnaOpDrZOZKmVfmAJzM3AccALwbWAc8B/s98t5Ok2dg7kupk50iqk50jaSYL67iSzFwDPGKGi/focjtJ6pq9I6lOdo6kOtk5kqZTx0u0JEmSJEmSVCEHPJIkSZIkSQ3ngEeSJEmSJKnhHPBIkiRJkiQ1XC0HWa5FRNG4BdtvXzQP4I4nPLB4JsDSMy+oJLe4KD9PXPeUA4tnrjjvxuKZX3jcHxbPBNh49L7FM5eecX7xTHWpcI8BkFk+syK5ZUv5zApu/9j971c8c8uS8v8dP+AttxfPBPj5m3cunnng/yj8vd+cb/v+GltQPjMny2dCJV02tny74pmV3Kfj48UjY7tlxTNfetFfFs8E2H3PzeVDf3VZ2Tw7R30SC8p3zth97l08s4puPOS/nl08E2DZU3conrnkG2uKZ87UO+7BI0mSJEmS1HAOeCRJkiRJkhrOAY8kSZIkSVLDOeCRJEmSJElqOAc8kiRJkiRJDeeAR5IkSZIkqeEc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcAv7vYD5iohVwCqApSzr82okjQJ7R1Kd7BxJdbJzpOZr7B48mbk6M1dm5spFLOn3ciSNAHtHUp3sHEl1snOk5mvsgEeSJEmSJEktDngkSZIkSZIazgGPJEmSJElSww38gCcivhERb+n3OiSNBjtHUp3sHEl1s3ek4TXw76KVmU/p9xokjQ47R1Kd7BxJdbN3pOE18HvwSJIkSZIkadsc8EiSJEmSJDWcAx5JkiRJkqSGc8AjSZIkSZLUcAN/kOWuRdlZ1W1HPqhoHsD2X/1x8UyAzCwfGlE8csGOK4pnrvivW4pnTu6wXfHM3HF58UyApTfcVTxzbNmy4pncUT5yGN32gkcXz7zP539UPLMqUUHvTK67tXgmD3lA8cgFl60tnnnDMw8qngnwoHdcVTwzC///ELcsKJo3rNYf+6jimduf2pzOmbjltuKZC+5d/v/7yTvvLJ7J+HjxyN1ftql4JsBlryvfZft/oez/N3H5OUXzpG7lli3FM6voxrHtyv9+seOx1xbPBFj/lPK/s46ftV/xTI6c/mz34JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIZzwCNJkiRJktRwsw54IiIj4o6IeFcdC+q43pdFxIb29R9Q53VL6i97R1Kd7BxJdbJzJFWl2z14HpaZxwFExD7tUtgw5eM57cs/0778kK1fHBEHRER2fP7diNjnHGnlAAAIK0lEQVQYEesj4vaIuDAi3hwRS7Zuk5mfzMzlhW6npOaxdyTVyc6RVCc7R1JxvbxEa4fMXN7x8aWOy9YBJ83y9X+bmdsDuwJvAJ4LnBUR0cOaJA03e0dSnewcSXWycyT1pKpj8HwWeGhEPGG2DTPzjsz8LvB04DHA07q5gohYFRFrImLNZsZ7WqykoWDvSKqTnSOpTnaOpFlVNeC5E3g30PXrSjPzKmAN8Pgut1+dmSszc+Uilsz+BZKGnb0jqU52jqQ62TmSZtXLgOemiLi14+PgKZf/K7BXRDxlDpnXAit6WJOk4WbvSKqTnSOpTnaOpJ70MuDZKTN36Pi4pPPCzBwH3tn+6NbutF5fKknTsXck1cnOkVQnO0dST6p6idZWnwZ2AI6ZbcOI2BN4JHBOxWuSNNzsHUl1snMk1cnOkTSjSgc8mbkFOB5400zbRMSy9sHCTgfOB86qck2Shpu9I6lOdo6kOtk5krallwHPrRGxoePj9TNsdwpw3TTnfyQi1gM3AP8EfAU4MjMne1iTpOFm70iqk50jqU52jqSeLOxim3Hgwoj4cGa+LTOvAGKmjTPzxVM+nwQeMuW8w2a70oh4CfDB9vVbStJosXck1cnOkVQnO0dSJWYd8GTm0joWMs31fprWa0wljRh7R1Kd7BxJdbJzJFWl6oMsS5IkSZIkqWIOeCRJkiRJkhouMrPfa+hZRPwWuLLLzXcCbiq8BDPLa8pazezO3pm5c+Hr76s59E6/7/t+ZlaVa6aZs7Fzyuv3Y2qmmf3I7TZzlDsHmvN9YmZ5TVnrMGZO2ztDMeCZi4hYk5krzRzczKpyzRz8zGHUlPven2UzRzFzGPmzbOYoZlaVa+90pyn3vZmj+3M3Spm+REuSJEmSJKnhHPBIkiRJkiQ13CgOeFabOfCZVeWaOfiZw6gp970/y2aOYuYw8mfZzFHMrCrX3ulOU+57M5uRa2YPRu4YPBoeEbEB+NvM/Ey/1yJp+Nk5kupk50iqk50zHEZxDx4VEBGfiYgzpzl/ZURkROxT/6p+by0vbq9l6sfSfq9N0tw0oXMAIuLeEfHhiLg2IsYj4tKIOLbf65I0N03onIj47gzPcy7u99okzU0TOgcgIl4TEb+IiLsi4pqI+GhELO/3unS3hf1egFSxO4H9O8/IzI19WoukIRYRi4D/B6wDjgWuAfYAxvu5LklD6xhgccfnS4CfAaf1ZzmShllEPB/4R+DlwDnAfsAngaXAy/q4NHVwDx5VLiIeFBFfj4j1EXFjRJwSEffvuPxREXF2RNwUEbdHxA8i4jFTMg5o/6VqY0T8MiKO6vLqMzOv7/woeuMkDZw+ds5LgJ2BZ2TmDzLziva/FxS+iZIGSL86JzPXTXl+8zhgGfCp4jdS0sDo4/OcxwI/ysyT289xvg18Dnh02VuoXjjgUaUiYlfg+8B/A4cAhwPLgdMjYuv33/bAycDj29v8BDgrInZsZ4wBX6X1/foY4KXACbT+UjWbe0XEle1dCM+MiEeUum2SBk+fO+eZwA+Bf46I6yPi5xFxQnvPHklDaACe53R6BfDvmXl1DzdJ0gDrc+f8AHh4RBzaztkLeDpwVqGbpwJ8iZZ6cWS0DsbVaerQ8JXARZn5pq1nRMSLaL2EYSVwfnv6S8flrwL+HHgK8HlaxfUgYN/MvKq9zWtp7Rq4Lb+kVVgX0Sq61wA/jIiHZeavu76VkgbFoHfOfsCTgC8CTwP2AT5K64nX33V3EyUNkEHvnM7MA4En0Bo0S2qmge6czDy1PST6fkQErVnCycCbtvV1qpcDHvXi+8CqKec9hNZEeKtHAn88TVlB69g450fE/YB3Ak8EdgEWAPcC9mpvdzCwdmsBtZ0HTG5rcZl5LnDu1s8j4j9pTbBfBbx6m7dM0iAa6M6h9STsRuAVmTkBXNh+IvTBiPj79G0rpaYZ9M7p9ArgOuDrc/gaSYNloDsnIp4AvA346/b2BwAfAk4E3j7bjVM9HPCoF3dm5qWdZ0TEDlO2GaP1ZGO6v17f0P73s7TK53XAFbQOSPot7nngwJ5l5kRErAEeUDJXUm0GvXOuAza3hztbXULrmBg7Ab/tMV9SvQa9c7auaTHwl8DHM3NLiUxJfTHonXMScEpmfqL9+c8iYjvgExHxDvtnMDjgUdV+TOvdZK7MzM0zbPM44NWZ+XWAiNgF2LXj8kuA3SNiz47XlR/CHI8h1d6V8KG0XrIlaTj1s3N+CDw/IsYyc+tfwQ6k9W5+N839pkhqgEF4nvNMWkPkT8518ZIap5+dswyYmHLeBBBzWL8q5kGWVbWPAvcBvhQRj46I/SLi8IhYHRHbt7f5FfAX7SPCPwo4FdjUkfFN4BfA5yLi4e2jwH8Q2OaUOCKOj4gj2tf5cFpPfB4KfKzsTZQ0QPrWOcD/BlYAH4qIgyLiCFq7Lf+LL8+ShlY/O2erVcC3MvPyIrdI0iDrZ+ecAayKiOdGxL4R8Se0Xgp2pnvvDA4HPKpUZl4L/BGt13T+O3AxrWIab39A60DIy4ELaRXQp2jtTrg1YxL4M1rfr+fReju+kzq+fiY7AKtpTanPBnYH/jgzz+/9lkkaRP3snPZfwf6U1uvjf0JrmPwp4LgSt03S4Onz8xwiYuvB3T9e4vZIGmx97pyTgPfTGur8vJ17NvDynm+Yign/qChJkiRJktRs7sEjSZIkSZLUcA54JEmSJEmSGs4BjyRJkiRJUsM54JEkSZIkSWo4BzySJEmSJEkN54BHkiRJkiSp4RzwSJIkSZIkNZwDHkmSJEmSpIb7/+OqU7hX4EgyAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1152x576 with 8 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_attention_weights(sentence, translated_tokens, attention_heads):\n",
    "    in_tokens = tf.convert_to_tensor([sentence])\n",
    "    in_tokens = tokenizers.pt.tokenize(in_tokens).to_tensor()\n",
    "    in_tokens = tokenizers.pt.lookup(in_tokens)[0]\n",
    "    \n",
    "    fig = plt.figure(figsize=(16, 8))\n",
    "    \n",
    "    for h, head in enumerate(attention_heads):\n",
    "        ax = fig.add_subplot(2, 4, h+1)\n",
    "        \n",
    "        plot_attention_head(in_tokens, translated_tokens, head)\n",
    "        \n",
    "        ax.set_xlabel(f'Head {h+1}')\n",
    "    \n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "\n",
    "plot_attention_weights(sentence, translated_tokens, attention_weights[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "De57iysIFpau"
   },
   "source": [
    "## Further reading\n",
    "- You may also see this simple <a href=\"https://keras.io/examples/timeseries/timeseries_transformer_classification\" target=\"_blank\">keras implementation</a> of transformers for time-series.\n",
    "- You may consider TensorFlow tutorial on <a href=\"https://www.tensorflow.org/tutorials/text/image_captioning\" target=\"_blank\">image captioning with visual attention</a>.\n",
    "- Beyond the above-mentioned vanilla transformer, a faster ***linear transformer*** is defined coupled with RNNs to use transformers in very long sequences [27].\n",
    "- It is also possible to use fourier features for spatial positional encoding, as expalined in [28].\n",
    "\n",
    "## References\n",
    "\n",
    "[1] C. Laurent, G. Pereyra, P. Brakel, Y. Zhang, and Y. Bengio, “Batch normalized recurrent neural networks,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Mar. 2016, vol. 2016-May, pp. 2657–2661, URL: https://doi.org/10.1109/ICASSP.2016.7472159.\n",
    "\n",
    "[2] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” Jul. 2016, URL:\n",
    "https://doi.org/10.48550/arXiv.1607.06450.\n",
    "\n",
    "[3] A. van den Oord et al., “WaveNet: A Generative Model for Raw Audio,” Sep. 2016, URL:\n",
    "https://doi.org/10.48550/arXiv.1609.03499.\n",
    "\n",
    "[4] user3352632 (https://stats.stackexchange.com/users/221270/user3352632), Using 'causal' padding ensures conv layers does not peek into the future when making predictions, URL (version: 2021-11-12): https://stats.stackexchange.com/q/552039.\n",
    "\n",
    "[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016, vol. 2016-Decem, pp. 770–778, URL: https://doi.org/10.1109/CVPR.2016.90\n",
    "\n",
    "[6] A. Karpathy, “The Unreasonable Effectiveness of Recurrent Neural Networks.” URL:  https://karpathy.github.io/2015/05/21/rnn-effectiveness/ (accessed May 31, 2021).\n",
    "\n",
    "[7] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems, 2014, vol. 4, no. January, pp. 3104–3112, URL: http://arxiv.org/abs/1409.3215.\n",
    "\n",
    "[8] S. Jean, K. Cho, R. Memisevic, and Y. Bengio, “On using very large target vocabulary for neural machine translation,” in ACL-IJCNLP 2015 - 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, Proceedings of the Conference, 2015, vol. 1, pp. 1–10, URL: http://dx.doi.org/10.3115/v1/P15-1001.\n",
    "\n",
    "[9] D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2015, vol. abs/1409.0, URL: \n",
    "https://doi.org/10.48550/arXiv.1409.0473.\n",
    "\n",
    "[10] M. T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in Conference Proceedings - EMNLP 2015: Conference on Empirical Methods in Natural Language Processing, 2015, vol. abs/1508.0, pp. 1412–1421, URL: http://dx.doi.org/10.18653/v1/D15-1166.\n",
    "\n",
    "[11] Zhang, Aston, et al. \"Dive into deep learning.\" arXiv preprint arXiv:2106.11342 (2021), URL: https://doi.org/10.48550/arXiv.2106.11342.\n",
    "\n",
    "[12] Williams, R. J., & Zipser, D. (1989). A learning algorithm for continually running fully recurrent neural networks. Neural computation, 1(2), 270-280, URL: https://doi.org/10.1162/neco.1989.1.2.270.\n",
    "\n",
    "[13] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, Wojciech Zaremba, “Sequence Level Training with Recurrent Neural Networks,” ICLR (Poster) 2016, URL: https://doi.org/10.48550/arXiv.1511.06732.\n",
    "\n",
    "[14] Hyndman, Rob J., and George Athanasopoulos. Forecasting: principles and practice. OTexts, 2018, URL: https://f0nzie.github.io/hyndman-bookdown-rsuite/.\n",
    "\n",
    "[15] Kirchgässner, Gebhard, Jürgen Wolters, and Uwe Hassler. Introduction to modern time series analysis. Springer Science & Business Media, 2012, URL: https://doi.org/10.1007/978-3-540-73291-4.\n",
    "\n",
    "[16] Geron, Aurelien. \"Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems.\" (2nd. ed.), 2019, O'Reilly Media, Inc., URL: https://www.oreilly.com/library/view/hands-on-machine-learning/9781492032632/ (accessed Dec 10, 2022).\n",
    "\n",
    "[17] Geron, Aurelien. \"Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems.\" (3nd. ed.), 2022, O'Reilly Media, Inc., URL: https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/ (accessed Dec 10, 2022).\n",
    "\n",
    "[18] Martín Abadi, Ashish Agarwal, et al., TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org, URL: https://doi.org/10.5281/zenodo.4724125.\n",
    "\n",
    "[19] Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Nonautoregressive neural machine translation. International Conference on Learning Representations, URL: https://doi.org/10.48550/arXiv.1711.02281.\n",
    "\n",
    "[20] Van Den Oord, Aäron, Nal Kalchbrenner, and Koray Kavukcuoglu. \"Pixel recurrent neural networks.\" International conference on machine learning in Proceedings of Machine Learning Research, 2016, URL: https://doi.org/10.48550/arXiv.1601.06759.\n",
    "\n",
    "[21] Vaswani, Ashish, et al. \"Attention is all you need.\" Advances in neural information processing systems 30 (2017), URL: \n",
    "https://doi.org/10.48550/arXiv.1706.03762.\n",
    "\n",
    "[22] K. Xu et al., “Show, Attend and Tell: Neural Image Caption Generation with Visual Attention,” in Proceedings of the 32nd International Conference on Machine Learning, 2015, vol. 37, pp. 2048–2057, URL: https://doi.org/10.48550/arXiv.1502.03044.\n",
    "\n",
    "[23] M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier,” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1135–1144, URL: https://doi.org/10.1145/2939672.2939778.\n",
    "\n",
    "[24] M. Saeed, “A Gentle Introduction to Positional Encoding in Transformer Models, Part 1.” URL: https://machinelearningmastery.com/a-gentle-introduction-to-positional-encoding-in-transformer-models-part-1/ (accessed Dec 11, 2022).\n",
    "\n",
    "[25] Graves, A., Wayne, G., & Danihelka, I. (2014). Neural Turing Machines. ArXiv, abs/1410.5401, URL: \n",
    "https://doi.org/10.48550/arXiv.1410.5401.\n",
    "\n",
    "[26] Cheng, J., Dong, L., & Lapata, M. (2016). Long Short-Term Memory-Networks for Machine Reading. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 551-561). Association for Computational Linguistics, URL: https://doi.org/10.18653/v1/D16-1053.\n",
    "\n",
    "[27] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret Proceedings of the 37th International Conference on Machine Learning, PMLR 119:5156-5165, 2020, URL: https://doi.org/10.48550/arXiv.2006.16236.\n",
    "\n",
    "[28] Li, Yang, et al. \"Learnable fourier features for multi-dimensional spatial positional encoding.\" Advances in Neural Information Processing Systems 34 (2021): 15816-15829, URL: https://doi.org/10.48550/arXiv.2106.02795.\n",
    "\n",
    "[29] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin Proceedings of the 34th International Conference on Machine Learning, PMLR 70:1243-1252, 2017, URL: https://doi.org/10.48550/arXiv.1705.03122.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "H-lF9QlfFpau"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "_XQgdqWeFpah",
    "6qu95NNNFpai",
    "1QqrBQgqFpam",
    "4V_yjvEbFpan",
    "fwr81Fy8Fpan",
    "PlBSs_BHFpap",
    "ky4IEWyaFpaq",
    "iWRiw0ypFpaq"
   ],
   "provenance": []
  },
  "gpuClass": "standard",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "ad2bdc8ecc057115af97d19610ffacc2b4e99fae6737bb82f5d7fb13d2f2c186"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "004cfe137aed42ee91b67eccaf9a6c80": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "00cd33f6c452418db7a15306114b9f05": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "00e6339db2cd4fa4805fa52656856946": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8443ca78174a4bd3ab46d014568c0459",
      "placeholder": "​",
      "style": "IPY_MODEL_5de1f721afc345faab1488846d847b4d",
      "value": " 1066/1803 [00:00&lt;00:00, 4821.76 examples/s]"
     }
    },
    "03a511833ccb4513bface70d1e62042e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0428269d43174d6a8999e52eacf0b3df": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2cce0ab2d7d8440d878b601cba37bd3d",
       "IPY_MODEL_3f752b431ca5448c93ecd10717951d18",
       "IPY_MODEL_8b28dca3460a40e4873c3593b472290a"
      ],
      "layout": "IPY_MODEL_67b93635f5fb4fb0bb346518c1410fa4"
     }
    },
    "04c99482102e4ce799991de3308d8089": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "04f3850729e149709343df40449662ce": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "081e224382ed46de89ddbfc02442dd42": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0b32c8300f8c43b59385e833c8f1d14f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "0d408ecb99354ade8ebad91d18702705": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "0d8fcd67e58c406a9b093f681dcc7a77": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0e5ee5b41e1248e99810003836896473",
      "max": 51785,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_8acc26bb1e564c89aa05506eced82cc8",
      "value": 51785
     }
    },
    "0e5ee5b41e1248e99810003836896473": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "10957d9c84244089b7700dc7448a4265": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "11ac0a906a2f4dee908a2c5a34e71d32": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "11c5b2c9800e405f816ce77e17d2fd09": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "1335953a3e8642679e8e2b6dcf4178c0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "13662985bc6643cea402f81f32337421": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "13a4c68005e44188afe1433f2e748a76": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b5c4e89ca6a64d3f91c60a21a168fafb",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6e86db15586b412dac66232041248a21",
      "value": 25000
     }
    },
    "157002f4e0f24e479f4ba497e8ba2b76": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "16a4e508013c4b93acbb291cb77c5ce8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_9e09e6206da948788e6529bcaee79b00",
       "IPY_MODEL_37d112806dad488ca624a99a2e4e2785",
       "IPY_MODEL_41841b7a60a64791ba1e97d601fe43b3"
      ],
      "layout": "IPY_MODEL_00cd33f6c452418db7a15306114b9f05"
     }
    },
    "18153dfd563549c78770122912d7d8c3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "194ff23579c344b982a720b5ccd66ce0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1988f5bb327d404fa16bcb23add04c51": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4f24f1d28d144c1eb4ae165c12422788",
      "placeholder": "​",
      "style": "IPY_MODEL_d1775d20113b4eda97c79a787b7aefa9",
      "value": "Generating splits...: 100%"
     }
    },
    "198b8c1c8fbb4055b25ceaab843fc060": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_41a3251130634aca8d1cf5bf0ddd2a6e",
      "max": 50000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_36fb86d798cb4b1e9e87aa4615edac4a",
      "value": 50000
     }
    },
    "199f5dde766640d78f8bdb4bcb018f80": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1a30c30df15b4c5aabc9bc970a6430a5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1a82f261d2d94520b631cfe320e47992": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1cc12d9744d5424b9a23f0b9d4c5a07f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1e6dfb42441c4f17a6b64324cf62e765": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "1f88d6a0092445e8accbf3c0f6e3658f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "2484a41d0cb84776a9b41253ed4dee9c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c467d9d0438543eea173ec2a0508edd1",
      "placeholder": "​",
      "style": "IPY_MODEL_a63674b6a2094038bb50f627cfcc1df9",
      "value": "Generating train examples...: 100%"
     }
    },
    "24ba73502d364163af4eaff8f989199c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "24d27baa3de648ea8a1ab8ff3c39585c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8120c921568c4986ac25940ab7570661",
       "IPY_MODEL_13a4c68005e44188afe1433f2e748a76",
       "IPY_MODEL_548d5d72690740829094f470de2aca6e"
      ],
      "layout": "IPY_MODEL_04f3850729e149709343df40449662ce"
     }
    },
    "25ac286850d248a7bba9d2286d1b2b58": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b7e62b474add4ec997b819826deed3f1",
      "max": 1803,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_dec0aafa5c544c0fa9d9ade8a1b5ea39",
      "value": 1803
     }
    },
    "26e450f753ff4c5296abc1ab46d282bd": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_326b326fe9df4122a58b9d974a65953d",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ec56928587114c4d9a071ddeb396a6e4",
      "value": 1
     }
    },
    "26ea44ffac52464ea1f3d1d4de67da91": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "28064cce020c48f4a82dd2e4fc593495": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f9f36b064a1e4858bf4d374f33d92fb9",
      "max": 50000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_2ac2dbb64d064ff1837968220a00701e",
      "value": 50000
     }
    },
    "28d32b1732b74a859725717db2749f98": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "29d3633bb4b84b4999ad980185d019e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_bc01c4866a3e441f889ece72bf12d98a",
      "placeholder": "​",
      "style": "IPY_MODEL_c45e34d1b47942b39c2478225901dc98",
      "value": " 6145/25000 [00:00&lt;00:00, 61437.90 examples/s]"
     }
    },
    "2ac2dbb64d064ff1837968220a00701e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "2c531649f8e74016a8c8529c7cf2ba64": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "2cce0ab2d7d8440d878b601cba37bd3d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c55ea7c2c70b46ac91f06f64bb083fea",
      "placeholder": "​",
      "style": "IPY_MODEL_d28878c5599e4656af0fc7c04c6e9ab6",
      "value": "Dl Size...: 100%"
     }
    },
    "2d3d1904a204429381c78c582a7bfd0e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2da098d7744f44eb81a8eacb3f8cb711": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "315e4088282545858818d869ab84f6cf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "326b326fe9df4122a58b9d974a65953d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "32c6f8706db54aed9031efb314fa10b1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "345e089df7bb4a708ce9af7fd02f2550": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c9e9e702f51349d7bd24ca2afbb34e30",
      "placeholder": "​",
      "style": "IPY_MODEL_d89056f2ef3947ecabfdb8f414821916",
      "value": "Dl Completed...: 100%"
     }
    },
    "355605003a024848abe079a2a383ffb3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "361ffbd5f2b74f5ba58ddae9a9fcadb5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "36fb86d798cb4b1e9e87aa4615edac4a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "37d112806dad488ca624a99a2e4e2785": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_aa31e8b550d044af8ad4b8b9e3480292",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_da2b89043e2f4c24b4ff42a5376f4371",
      "value": 3
     }
    },
    "37e42ef6ce13464cb4924511a62712a6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_76ab9c4a636b460a8a757e18781ee69f",
      "placeholder": "​",
      "style": "IPY_MODEL_6cfedefadfde432286ef09fb4da7ae5d",
      "value": "Generating unsupervised examples...:  99%"
     }
    },
    "3ebd4bd4641d424f96a511c586a55330": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "3f752b431ca5448c93ecd10717951d18": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1335953a3e8642679e8e2b6dcf4178c0",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_3ebd4bd4641d424f96a511c586a55330",
      "value": 1
     }
    },
    "3fbfd0e2483d4cc69ef982f604eda87a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "402e10e3af474c8a9136edeb36614607": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "41841b7a60a64791ba1e97d601fe43b3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_4607f99cd4e64cc594db6d00fc4bee7a",
      "placeholder": "​",
      "style": "IPY_MODEL_2da098d7744f44eb81a8eacb3f8cb711",
      "value": " 3/3 [00:04&lt;00:00,  1.19s/ splits]"
     }
    },
    "41a3251130634aca8d1cf5bf0ddd2a6e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "43b2355f92684db19b8597b012a1917a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "45595c8ff42d4837836d7d7cfecd9aa3": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "45bac909d1dc4e16a297e25a0edb5a26": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "4607f99cd4e64cc594db6d00fc4bee7a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "49f9b321dbae469399eb9ade1a3cf2ec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "4a3688bcb22c47e2ac6e6660ea369f71": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4c1bcd6ede3d4aae8cd235590cc7da92": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4c3be37f55ea4b85bb2ecdcb906e9692": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "4cbf9b42f7be4b6298cee16ec037697e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_10957d9c84244089b7700dc7448a4265",
      "placeholder": "​",
      "style": "IPY_MODEL_525c7216bae048b0b5447f21e2bbb2ad",
      "value": "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-validation.tfrecord*...:   0%"
     }
    },
    "4e2c552130094e55ab157f5ff99e9572": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_315e4088282545858818d869ab84f6cf",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_6156f594196044f792d0c4b46278966c",
      "value": 25000
     }
    },
    "4f24f1d28d144c1eb4ae165c12422788": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "50ed4a177c5140dd9c958a74418e90bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "522ef828d4a744e78892b5641b99988e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "525c7216bae048b0b5447f21e2bbb2ad": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5484f083f29646739026f23f273e2dd4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1a30c30df15b4c5aabc9bc970a6430a5",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_402e10e3af474c8a9136edeb36614607",
      "value": 25000
     }
    },
    "548d5d72690740829094f470de2aca6e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d3c1d64fd84d4b15838eca22adcb59c4",
      "placeholder": "​",
      "style": "IPY_MODEL_4c1bcd6ede3d4aae8cd235590cc7da92",
      "value": " 10494/25000 [00:00&lt;00:00, 104930.09 examples/s]"
     }
    },
    "55fdbaf1e5d84ee081b01359b51fb3ed": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6df89ebc143945ac8601c26d76ec105e",
       "IPY_MODEL_198b8c1c8fbb4055b25ceaab843fc060",
       "IPY_MODEL_c7aae61c0ca9490a9c7e0a9d7e763be8"
      ],
      "layout": "IPY_MODEL_8470cc9b81394da1bda380fe6a49b528"
     }
    },
    "56a9ea20d9c04820bd2243efa1c8a0ff": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5abc8a5b299f488ab905e60be7cd0012": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d1960ffbcbc3461d8baf9076bd2c2f67",
      "max": 1193,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f51df18a6d794ec38ec7cc0fb470fea9",
      "value": 1193
     }
    },
    "5b220cd7e2b44885b384dc3eeefa51b9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_abf40f3a508c436bbf7bb0b971886aa7",
       "IPY_MODEL_5484f083f29646739026f23f273e2dd4",
       "IPY_MODEL_8300c7b4a8004ebaa21e2600f8e13dd3"
      ],
      "layout": "IPY_MODEL_cfd0a061e7ad4599a5aec67d7c658422"
     }
    },
    "5b4bd79e8c7b4b6b8e6d16732a47e050": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_88e7206a195640ff9eeeefac0cf86c2f",
      "placeholder": "​",
      "style": "IPY_MODEL_cc458249511d4b8e917bde34a9213f4c",
      "value": " 762/1193 [00:00&lt;00:00, 7614.36 examples/s]"
     }
    },
    "5de1f721afc345faab1488846d847b4d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "5eb299ccedda45678053f9fc9b6fa90f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ac82bbe16c8e4945b4985dacfdae86d7",
      "placeholder": "​",
      "style": "IPY_MODEL_9123a4b7fd18451eb5a7e16665968252",
      "value": " 49652/50000 [00:17&lt;00:00, 5417.27 examples/s]"
     }
    },
    "601bc3b00b484699ba289b354ed37a81": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6156f594196044f792d0c4b46278966c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "6562ce5c53bd4555afeceaee989c1e9d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_836e4a996cfc4486a1fb3e72dab946cf",
      "max": 1193,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_50ed4a177c5140dd9c958a74418e90bb",
      "value": 1193
     }
    },
    "662f4379bf2b473eaeb72ee31f144e5a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_522ef828d4a744e78892b5641b99988e",
      "placeholder": "​",
      "style": "IPY_MODEL_26ea44ffac52464ea1f3d1d4de67da91",
      "value": "Generating train examples...:  98%"
     }
    },
    "66edae33ebb84ab6a1ba3bb79b091542": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "67b93635f5fb4fb0bb346518c1410fa4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6a839391facd410fb9ce3fbc194efede": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_24ba73502d364163af4eaff8f989199c",
      "placeholder": "​",
      "style": "IPY_MODEL_bca7023d538c4978b401874b15717087",
      "value": "Generating validation examples...:  64%"
     }
    },
    "6b3c0db62591435c9752f9d38653e703": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6cc320d62c12425da70b687c68fbc54e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6cfedefadfde432286ef09fb4da7ae5d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6d3cef428c9840ffa9953f5e764e893f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6df89ebc143945ac8601c26d76ec105e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e5dd3a037c3b41ed849fecf223e190d5",
      "placeholder": "​",
      "style": "IPY_MODEL_6cc320d62c12425da70b687c68fbc54e",
      "value": "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-unsupervised.tfrecord*...:  58%"
     }
    },
    "6e86db15586b412dac66232041248a21": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "711083870fc9494c87f43c8b2d4ef840": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_37e42ef6ce13464cb4924511a62712a6",
       "IPY_MODEL_28064cce020c48f4a82dd2e4fc593495",
       "IPY_MODEL_5eb299ccedda45678053f9fc9b6fa90f"
      ],
      "layout": "IPY_MODEL_cdc8e700879b4573b6cffac9838bffde"
     }
    },
    "717f5951a5bb4e39babe308912dc391f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "74466042f43b4f23a78aee2a4639f808": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "75ea4dd373294c8bac2757a61dea2bc5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_04c99482102e4ce799991de3308d8089",
      "placeholder": "​",
      "style": "IPY_MODEL_973469e2dfc84062a90c87aed43310be",
      "value": " 15450/51785 [00:00&lt;00:00, 154482.47 examples/s]"
     }
    },
    "768d778dea1f47ed8a0aab960e384da4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_74466042f43b4f23a78aee2a4639f808",
      "placeholder": "​",
      "style": "IPY_MODEL_fbed69c140c044baa60bf37c2bec3857",
      "value": "Extraction completed...: 100%"
     }
    },
    "76ab9c4a636b460a8a757e18781ee69f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7a497484c0aa4019afaf46915b861447": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7ae7b2022c4946a2b36ebaf1db93b1da": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7c3bc52999644cefbd2e2a1c80acf870",
      "max": 3,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_11c5b2c9800e405f816ce77e17d2fd09",
      "value": 3
     }
    },
    "7c3bc52999644cefbd2e2a1c80acf870": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "805ff42ca705407aae5851dd0c4d311f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "8120c921568c4986ac25940ab7570661": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2d3d1904a204429381c78c582a7bfd0e",
      "placeholder": "​",
      "style": "IPY_MODEL_13662985bc6643cea402f81f32337421",
      "value": "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-test.tfrecord*...:  42%"
     }
    },
    "8300c7b4a8004ebaa21e2600f8e13dd3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6d3cef428c9840ffa9953f5e764e893f",
      "placeholder": "​",
      "style": "IPY_MODEL_e6d88215fc8c4e75af0cd4079b9044a0",
      "value": " 24852/25000 [00:13&lt;00:00, 1974.71 examples/s]"
     }
    },
    "833914562fec4da99e4a178c43b85c8e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "836e4a996cfc4486a1fb3e72dab946cf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8443ca78174a4bd3ab46d014568c0459": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8470cc9b81394da1bda380fe6a49b528": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "88e7206a195640ff9eeeefac0cf86c2f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "89f05f0c418042338acbe160f007d9f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4cbf9b42f7be4b6298cee16ec037697e",
       "IPY_MODEL_6562ce5c53bd4555afeceaee989c1e9d",
       "IPY_MODEL_da4a94ac24a14348a32ac14a18e6a9a4"
      ],
      "layout": "IPY_MODEL_f142ff7976ef4c6ebdefcfffae8ada71"
     }
    },
    "8acc26bb1e564c89aa05506eced82cc8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "8b28dca3460a40e4873c3593b472290a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9c7df8e6470148c2ad4a0ed2d50c7faa",
      "placeholder": "​",
      "style": "IPY_MODEL_9f0e39f78ea543288e7a887a25cd5996",
      "value": " 80/80 [00:09&lt;00:00, 18.22 MiB/s]"
     }
    },
    "8d49812246354321b1f65421c9baaa5c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_66edae33ebb84ab6a1ba3bb79b091542",
      "placeholder": "​",
      "style": "IPY_MODEL_e992c27b3eb8463fb03ba08cc31ad0d1",
      "value": " 50508/51785 [00:03&lt;00:00, 13473.31 examples/s]"
     }
    },
    "8e6b13758edc4bbabf8a0c18988c8c65": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_e2cb78b056de4d21b732d02c4882363e",
       "IPY_MODEL_4e2c552130094e55ab157f5ff99e9572",
       "IPY_MODEL_29d3633bb4b84b4999ad980185d019e4"
      ],
      "layout": "IPY_MODEL_1f88d6a0092445e8accbf3c0f6e3658f"
     }
    },
    "8ece172075644ab383c83565f17ed662": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8f6a9ecf93ed4406a84e4cb386aa627f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8ece172075644ab383c83565f17ed662",
      "placeholder": "​",
      "style": "IPY_MODEL_0b32c8300f8c43b59385e833c8f1d14f",
      "value": " 1/1 [00:09&lt;00:00,  9.10s/ url]"
     }
    },
    "904e62bf412c422d893f1256f494e0a0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_18153dfd563549c78770122912d7d8c3",
      "max": 1803,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_f176ee46da264916890c84776263b8d6",
      "value": 1803
     }
    },
    "9123a4b7fd18451eb5a7e16665968252": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "92f8306c13f643f2a0ea9593449741de": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1988f5bb327d404fa16bcb23add04c51",
       "IPY_MODEL_7ae7b2022c4946a2b36ebaf1db93b1da",
       "IPY_MODEL_b1e4df2309574659b8295e0fd4b6f932"
      ],
      "layout": "IPY_MODEL_9b70efb93ff149258053d0bb88e83b90"
     }
    },
    "96f2561a16454bda9411a84fd62dc9c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_49f9b321dbae469399eb9ade1a3cf2ec",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_601bc3b00b484699ba289b354ed37a81",
      "value": 1
     }
    },
    "973469e2dfc84062a90c87aed43310be": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "9b70efb93ff149258053d0bb88e83b90": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "9c7df8e6470148c2ad4a0ed2d50c7faa": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9e09e6206da948788e6529bcaee79b00": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_45595c8ff42d4837836d7d7cfecd9aa3",
      "placeholder": "​",
      "style": "IPY_MODEL_1e6dfb42441c4f17a6b64324cf62e765",
      "value": "Generating splits...: 100%"
     }
    },
    "9f0e39f78ea543288e7a887a25cd5996": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a0ad7da1aa904b3793781e82347839d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_199f5dde766640d78f8bdb4bcb018f80",
      "placeholder": "​",
      "style": "IPY_MODEL_805ff42ca705407aae5851dd0c4d311f",
      "value": "Generating test examples...:  59%"
     }
    },
    "a194453ab86d4a06ad3f1353bc7a2cec": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a1df0ae0c7c24c66b00556c56c123f28": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a23e10b75632416a831d67a22e1d18bb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0d408ecb99354ade8ebad91d18702705",
      "placeholder": "​",
      "style": "IPY_MODEL_b31dce9c8e074b99832accc016c8b0f1",
      "value": "Dl Completed...: 100%"
     }
    },
    "a63674b6a2094038bb50f627cfcc1df9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "a7591f8255d64affa8633b7265cc6685": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a7d124e655d747ba90d3974215dfcfb9",
      "placeholder": "​",
      "style": "IPY_MODEL_f385dc04949b4cd48732f35c316e76ac",
      "value": " 24946/25000 [00:08&lt;00:00, 5276.95 examples/s]"
     }
    },
    "a7d124e655d747ba90d3974215dfcfb9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a9d091eeeea545fa891cea89abd69f58": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_662f4379bf2b473eaeb72ee31f144e5a",
       "IPY_MODEL_0d8fcd67e58c406a9b093f681dcc7a77",
       "IPY_MODEL_8d49812246354321b1f65421c9baaa5c"
      ],
      "layout": "IPY_MODEL_a9ea451184f1469d9b56cbdcd51ba579"
     }
    },
    "a9ea451184f1469d9b56cbdcd51ba579": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "aa31e8b550d044af8ad4b8b9e3480292": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ab02b18ab5114634ac4333bb4fef6584": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "abf40f3a508c436bbf7bb0b971886aa7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_11ac0a906a2f4dee908a2c5a34e71d32",
      "placeholder": "​",
      "style": "IPY_MODEL_7a497484c0aa4019afaf46915b861447",
      "value": "Generating test examples...:  99%"
     }
    },
    "ac82bbe16c8e4945b4985dacfdae86d7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "acad49e817f046069d5c41f0c57392a1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "af6c0ad3bb0b45358fc284b4dc765e7a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "aff31d1ccb204103972f2ca7dd016d96": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_03a511833ccb4513bface70d1e62042e",
      "placeholder": "​",
      "style": "IPY_MODEL_833914562fec4da99e4a178c43b85c8e",
      "value": " 1/1 [00:18&lt;00:00, 12.97s/ url]"
     }
    },
    "b0671f5dfad045dd8cc22525bd8148ef": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b1e4df2309574659b8295e0fd4b6f932": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_194ff23579c344b982a720b5ccd66ce0",
      "placeholder": "​",
      "style": "IPY_MODEL_f5c625672ff347e9aa8654fcf6c4e8a7",
      "value": " 3/3 [00:55&lt;00:00, 19.05s/ splits]"
     }
    },
    "b2e300f3b3124049827dd331dc1966a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b31dce9c8e074b99832accc016c8b0f1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b5c4e89ca6a64d3f91c60a21a168fafb": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b7e62b474add4ec997b819826deed3f1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bb2ac279c8ae42b5a82bf9caf36b3818": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a0ad7da1aa904b3793781e82347839d6",
       "IPY_MODEL_25ac286850d248a7bba9d2286d1b2b58",
       "IPY_MODEL_00e6339db2cd4fa4805fa52656856946"
      ],
      "layout": "IPY_MODEL_da3501a7bae049f5908ba83c546d5c21"
     }
    },
    "bbac88fda8dd4e62a522f8d3f6049e03": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_2484a41d0cb84776a9b41253ed4dee9c",
       "IPY_MODEL_c6e19a0129984fef8099664ac826eda5",
       "IPY_MODEL_a7591f8255d64affa8633b7265cc6685"
      ],
      "layout": "IPY_MODEL_c378d6ce02d54985bbdc07926a480aea"
     }
    },
    "bc01c4866a3e441f889ece72bf12d98a": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bca7023d538c4978b401874b15717087": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "bfadf2a622a74dafa8e844383e1010e1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b0671f5dfad045dd8cc22525bd8148ef",
      "max": 51785,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_43b2355f92684db19b8597b012a1917a",
      "value": 51785
     }
    },
    "c378d6ce02d54985bbdc07926a480aea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "c4295bc4592a43c583e626adc6e5e538": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "c45e34d1b47942b39c2478225901dc98": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c467d9d0438543eea173ec2a0508edd1": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c55ea7c2c70b46ac91f06f64bb083fea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c656e3e19c88426ba2274f9ad6f7593d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_345e089df7bb4a708ce9af7fd02f2550",
       "IPY_MODEL_96f2561a16454bda9411a84fd62dc9c1",
       "IPY_MODEL_8f6a9ecf93ed4406a84e4cb386aa627f"
      ],
      "layout": "IPY_MODEL_acad49e817f046069d5c41f0c57392a1"
     }
    },
    "c6e19a0129984fef8099664ac826eda5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_004cfe137aed42ee91b67eccaf9a6c80",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_c4295bc4592a43c583e626adc6e5e538",
      "value": 25000
     }
    },
    "c7aae61c0ca9490a9c7e0a9d7e763be8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c8dc362bd3e84c0ca39bb79da762a968",
      "placeholder": "​",
      "style": "IPY_MODEL_4c3be37f55ea4b85bb2ecdcb906e9692",
      "value": " 28907/50000 [00:00&lt;00:00, 160520.33 examples/s]"
     }
    },
    "c8dc362bd3e84c0ca39bb79da762a968": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "c9e9e702f51349d7bd24ca2afbb34e30": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ca5186021ba844ce9b9b7bcdf0a3e9ec": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_157002f4e0f24e479f4ba497e8ba2b76",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_717f5951a5bb4e39babe308912dc391f",
      "value": 1
     }
    },
    "cc458249511d4b8e917bde34a9213f4c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cdc8e700879b4573b6cffac9838bffde": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "cfd0a061e7ad4599a5aec67d7c658422": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "d069f92b086441f99ed23a1aefcfbd60": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_a23e10b75632416a831d67a22e1d18bb",
       "IPY_MODEL_26e450f753ff4c5296abc1ab46d282bd",
       "IPY_MODEL_aff31d1ccb204103972f2ca7dd016d96"
      ],
      "layout": "IPY_MODEL_fc05cddd40ab4076a5e1fa79fb50468d"
     }
    },
    "d1775d20113b4eda97c79a787b7aefa9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d1960ffbcbc3461d8baf9076bd2c2f67": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d28878c5599e4656af0fc7c04c6e9ab6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "d3c1d64fd84d4b15838eca22adcb59c4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d47a155b8c9146d1a8eb80c228a8caf3": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_f52e89e54b624ed8923a63aac94b5fe4",
       "IPY_MODEL_ca5186021ba844ce9b9b7bcdf0a3e9ec",
       "IPY_MODEL_f827dcae35604af492d0984ad0ee9c97"
      ],
      "layout": "IPY_MODEL_4a3688bcb22c47e2ac6e6660ea369f71"
     }
    },
    "d4a1cbacfc3d4dc39df02c7b6da95f40": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a1df0ae0c7c24c66b00556c56c123f28",
      "placeholder": "​",
      "style": "IPY_MODEL_b2e300f3b3124049827dd331dc1966a9",
      "value": "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-train.tfrecord*...:  30%"
     }
    },
    "d66f399c9002437b9bee01932b83c82b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d7b94d523f2a4bbd91fb465178ac6e0b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d4a1cbacfc3d4dc39df02c7b6da95f40",
       "IPY_MODEL_bfadf2a622a74dafa8e844383e1010e1",
       "IPY_MODEL_75ea4dd373294c8bac2757a61dea2bc5"
      ],
      "layout": "IPY_MODEL_45bac909d1dc4e16a297e25a0edb5a26"
     }
    },
    "d89056f2ef3947ecabfdb8f414821916": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "da2b89043e2f4c24b4ff42a5376f4371": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "da3501a7bae049f5908ba83c546d5c21": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "da4a94ac24a14348a32ac14a18e6a9a4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1a82f261d2d94520b631cfe320e47992",
      "placeholder": "​",
      "style": "IPY_MODEL_1cc12d9744d5424b9a23f0b9d4c5a07f",
      "value": " 0/1193 [00:00&lt;?, ? examples/s]"
     }
    },
    "dbb6fa23119443a5bea25bc2c5e560e0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3fbfd0e2483d4cc69ef982f604eda87a",
      "placeholder": "​",
      "style": "IPY_MODEL_355605003a024848abe079a2a383ffb3",
      "value": "Shuffling ~/tensorflow_datasets/ted_hrlr_translate/pt_to_en/1.0.0.incompleteWNREJY/ted_hrlr_translate-test.tfrecord*...:   0%"
     }
    },
    "dec0aafa5c544c0fa9d9ade8a1b5ea39": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "dfaccc081ff743d7899658021fd21318": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": "20px"
     }
    },
    "e1750ad0e52540669da5affce95aca74": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dfaccc081ff743d7899658021fd21318",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_af6c0ad3bb0b45358fc284b4dc765e7a",
      "value": 1
     }
    },
    "e2cb78b056de4d21b732d02c4882363e": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_d66f399c9002437b9bee01932b83c82b",
      "placeholder": "​",
      "style": "IPY_MODEL_ab02b18ab5114634ac4333bb4fef6584",
      "value": "Shuffling ~/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteEYDRZ2/imdb_reviews-train.tfrecord*...:  25%"
     }
    },
    "e38ee97623474b60900b1110402b62ca": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e3a1ff7cbd4b470588f20c5aea5c2812": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e5dd3a037c3b41ed849fecf223e190d5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e6d88215fc8c4e75af0cd4079b9044a0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e745d8bf7da04409a025204d44961bb0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "e992c27b3eb8463fb03ba08cc31ad0d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ec56928587114c4d9a071ddeb396a6e4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f142ff7976ef4c6ebdefcfffae8ada71": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": "hidden",
      "width": null
     }
    },
    "f1430049a4934fb4b6e6bd4ffa19ff14": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_081e224382ed46de89ddbfc02442dd42",
      "placeholder": "​",
      "style": "IPY_MODEL_6b3c0db62591435c9752f9d38653e703",
      "value": " 0/1803 [00:00&lt;?, ? examples/s]"
     }
    },
    "f176ee46da264916890c84776263b8d6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f24d47724f8744a3bcd643e6660efe45": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_dbb6fa23119443a5bea25bc2c5e560e0",
       "IPY_MODEL_904e62bf412c422d893f1256f494e0a0",
       "IPY_MODEL_f1430049a4934fb4b6e6bd4ffa19ff14"
      ],
      "layout": "IPY_MODEL_e745d8bf7da04409a025204d44961bb0"
     }
    },
    "f385dc04949b4cd48732f35c316e76ac": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f51df18a6d794ec38ec7cc0fb470fea9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "f52e89e54b624ed8923a63aac94b5fe4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e38ee97623474b60900b1110402b62ca",
      "placeholder": "​",
      "style": "IPY_MODEL_e3a1ff7cbd4b470588f20c5aea5c2812",
      "value": "Dl Size...: 100%"
     }
    },
    "f5c625672ff347e9aa8654fcf6c4e8a7": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "f827dcae35604af492d0984ad0ee9c97": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_32c6f8706db54aed9031efb314fa10b1",
      "placeholder": "​",
      "style": "IPY_MODEL_2c531649f8e74016a8c8529c7cf2ba64",
      "value": " 124/124 [00:18&lt;00:00, 11.50 MiB/s]"
     }
    },
    "f9148891a3664014a2a8f4a2ab64a220": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_56a9ea20d9c04820bd2243efa1c8a0ff",
      "placeholder": "​",
      "style": "IPY_MODEL_361ffbd5f2b74f5ba58ddae9a9fcadb5",
      "value": " 1/1 [00:18&lt;00:00, 18.29s/ file]"
     }
    },
    "f9f36b064a1e4858bf4d374f33d92fb9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "fba171f398da41a19ca1ec7d870db662": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_768d778dea1f47ed8a0aab960e384da4",
       "IPY_MODEL_e1750ad0e52540669da5affce95aca74",
       "IPY_MODEL_f9148891a3664014a2a8f4a2ab64a220"
      ],
      "layout": "IPY_MODEL_a194453ab86d4a06ad3f1353bc7a2cec"
     }
    },
    "fbed69c140c044baa60bf37c2bec3857": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fc05cddd40ab4076a5e1fa79fb50468d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ffa91a31b4ad444780645507b3d56453": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6a839391facd410fb9ce3fbc194efede",
       "IPY_MODEL_5abc8a5b299f488ab905e60be7cd0012",
       "IPY_MODEL_5b4bd79e8c7b4b6b8e6d16732a47e050"
      ],
      "layout": "IPY_MODEL_28d32b1732b74a859725717db2749f98"
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}