lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NXTSugt6ieXh"
      },
      "source": [
        "## Training CBoW Model\n",
        "\n",
        "This notebooks is a part of [AI for Beginners Curriculum](http://aka.ms/ai-beginners)\n",
        "\n",
        "In this example, we will look at training CBoW language model to get our own Word2Vec embedding space. We will use AG News dataset as the source of text."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "hvf7izZpieXk"
      },
      "outputs": [],
      "source": [
        "from tensorflow import keras\n",
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds\n",
        "import numpy as np"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will start by loading the dateset:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 299,
          "referenced_widgets": [
            "a6a32befb28542228cde3d444d6411f6",
            "9f5a040885564d41934f6c458761bf33",
            "a7651b06cb974b52a35e34e8f96c226c",
            "ee267b7dcf05457b8e3f545df150f09f",
            "55cc8f6b3b0c49ddbc4bfe526906ccbf",
            "9d4d315121e9440c8578a62fbe88e415",
            "a1f8b53e8a1d4ebd8ed0116219490877",
            "074db709c0a14cd4acfe13ed54e92cbc",
            "713f10e7274e4f0484d34759b5505842",
            "5ac5967b468d4af59cea0693ce9a8217",
            "7916d209cbe04da2912830b16e5f747c",
            "1a467f2a5e4b421db16e4fd4329f9bc1",
            "40ac80ed694940c191002b92e30a9f39",
            "4d62b43b387b45c89471f9820c041718",
            "ee572078162448bd89bd2c52fbe39aa7",
            "2bddf650279242d7a259db4209de3253",
            "9bba163f10a4461ca232b54d79c9b74d",
            "f91bb89f9f144f8e97f8f0c97f7d9f55",
            "bec242f072394c4cabc6f39e43350603",
            "74a70de4caa94d358606308df816725b",
            "be1b974e61b44ecd807a77a94f6f7991",
            "72693d5e2c034fb9ad03a32a8eb2999f",
            "3832f856d7644da2aedc8dc843732269",
            "3e31ac504d4242b8be208b68219bf064",
            "bb16a77431264209935f8e747918e430",
            "b1c32e56d326473db36bcda7abca7010",
            "7b38647b718d4cb58c20480e6387d749",
            "490cee33a6c14ead8209f36ca7dc1351",
            "5eee10ee14ff47f9bea181870bb973e3",
            "15d88a4607524d07be1f3b91345243ba",
            "a585d2e5ac5240679587990dbf53dfd2",
            "bbce51f4b75a4f999f4b3c170083e724",
            "25c18271a4594c1fa8c694d50dd356a7",
            "3edc08cbd6774e7485d42bdd13164ed6",
            "1cf5cb0c39cc4c5ca66be45895aa1860",
            "b70cac0930eb4d2da24a9f5b042f4a9e",
            "b38d4f6271234adfb41e8309a115e95b",
            "d7ef0d4ec19749c3bee8a0be8ad2d468",
            "bfb1e75c5bc744f28544515c660a0b9b",
            "62aa64e5318d445f844b8083ae6c40f4",
            "7ab6c716b4a04052bf048d0ede312365",
            "94db6867a26a4c988f549d20b3cb51f3",
            "30ffd5f13a524b0eabd5d2f20885ce50",
            "5022afdb2026474081a3f54cb4c81351",
            "52d3dd9cf2994e6da25a10ea42be4beb",
            "82ee379245d64fc39d1ed9a2586e20a2",
            "24283dc34e944591877888871a5e584b",
            "027dbec8c35d4ec3ab43ed2878a32eb9",
            "b459e5715d3b44eeb379108510261336",
            "b58f7dc6368b42d0a387e47bce4ce88e",
            "2252a0aeca7c4b7784370704181f1628",
            "dc49356d1ba943ad87b88ee6e451e7fb",
            "7095c6398b0c433db8c4284620c9e335",
            "19ece8654d8149ac87538fd162bd1aeb",
            "e630a16615414ceeba5868d162f55a20",
            "13ee77a308634d928662b651ad2bb9e7",
            "260fcdf1d0404d149732d566b6ccbbab",
            "f3ad889117ba43b783e34a82113b325c",
            "4dace863d2be4961ae72c729405da6cc",
            "2bbd772ad6284273b3cf97c6afeda6e0",
            "ca245734f2f54c4e805e761d23652eca",
            "bcedd81ebcef4d9ca31eea1ae4ab795d",
            "532f40fc7a1e4826b4495be24ee0f8ed",
            "b26304339073463b9f0ba2cce4835d13",
            "abaa80c91f5642649996c844ceb0fcd1",
            "8655ea4b7b6c4399adaf6e04613869ea",
            "fc94257ae5094ce0b04695ad29bdf72b",
            "30224d6b4c274faf85dbd4d2c1892aa7",
            "295d430b24444986a46a9382c5d5f80d",
            "9a4eedfb4c6a466ba6f6f21ce76a64bb",
            "9e28f7897bf142aebd4d374559320812",
            "0798ebda763a40bc86235a40dfc1adec",
            "bcd9ea70684742b6991d4e2c7556efa6",
            "44e94cb4f240446da537579caa8e6d2f",
            "8591f95a707d4214a17e9f187df6e1c4",
            "75dd999664ac40f18168f6e1870a878e",
            "91d15913f17040da828ece1c3b5fa6c6"
          ]
        },
        "id": "pWPCrm2jieXl",
        "outputId": "7ffa325f-d5d2-4044-d318-0a521f4f5c98"
      },
      "outputs": [],
      "source": [
        "ds_train, ds_test = tfds.load('ag_news_subset').values()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## CBoW Model\n",
        "\n",
        "CBoW learns to predict a word based on the $2N$ neighboring words. For example, when $N=1$, we will get the following pairs from the sentence *I like to train networks*: (like,I), (I, like), (to, like), (like,to), (train,to), (to, train), (networks, train), (train,networks). Here, first word is the neighboring word used as an input, and second word is the one we are predicting.\n",
        "\n",
        "To build a network to predict next word, we will need to supply neighboring word as input, and get word number as output. The architecture of CBoW network is the following:\n",
        "\n",
        "* Input word is passed through the embedding layer. This very embedding layer would be our Word2Vec embedding, thus we will define it separately as `embedder` variable. We will use embedding size = 30 in this example, even though you might want to experiment with higher dimensions (real word2vec has 300)\n",
        "* Embedding vector would then be passed to a dense layer that will predict output word. Thus it has the `vocab_size` neurons.\n",
        "\n",
        "Embedding layer in Keras automatically knows how to convert numeric input into one-hot encoding, so that we do not have to one-hot-encode input word separately. We specify `input_length=1` to indicate that we want just one word in the input sequence - normally embedding layer is designed to work with longer sequences.\n",
        "\n",
        "For the output, if we use `sparse_categorical_crossentropy` as loss function, we would also have to provide just word numbers as expected results, without one-hot encoding.\n",
        "\n",
        "We will set `vocab_size` to 5000 to limit computations a bit. We will also define a vectorizer which we will use later. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 68,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6PHiH8oRieXl",
        "outputId": "0259a0d5-b5f1-4bc9-d632-73c31893fa3f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Model: \"sequential_1\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding_1 (Embedding)     (None, 1, 30)             150000    \n",
            "                                                                 \n",
            " dense_1 (Dense)             (None, 1, 5000)           155000    \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 305,000\n",
            "Trainable params: 305,000\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ],
      "source": [
        "vocab_size = 5000\n",
        "\n",
        "vectorizer = keras.layers.experimental.preprocessing.TextVectorization(max_tokens=vocab_size,input_shape=(1,))\n",
        "embedder = keras.layers.Embedding(vocab_size,30,input_length=1)\n",
        "\n",
        "model = keras.Sequential([\n",
        "    embedder,\n",
        "    keras.layers.Dense(vocab_size,activation='softmax')\n",
        "])\n",
        "\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's initialize the vectorizer and get out the vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 69,
      "metadata": {
        "id": "rWnylDAIieXn"
      },
      "outputs": [],
      "source": [
        "def extract_text(x):\n",
        "    return x['title']+' '+x['description']\n",
        "\n",
        "vectorizer.adapt(ds_train.take(500).map(extract_text))\n",
        "vocab = vectorizer.get_vocabulary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Preparing Training Data\n",
        "\n",
        "Now let's program the main function that will compute CBoW word pairs from text. This function will allow us to specify window size, and will return a set of pairs - input and output word. Note that this function can be used on words, as well as on vectors/tensors - which will allow us to encode the text, before passing it to `to_cbow` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 70,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x-dsXygOieXn",
        "outputId": "11828ef5-5961-4909-f777-ff7b9b93adbd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[['like', 'I'], ['to', 'I'], ['I', 'like'], ['to', 'like'], ['train', 'like'], ['I', 'to'], ['like', 'to'], ['train', 'to'], ['networks', 'to'], ['like', 'train'], ['to', 'train'], ['networks', 'train'], ['to', 'networks'], ['train', 'networks']]\n",
            "[[<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=771>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=771>], [<tf.Tensor: shape=(), dtype=int64, numpy=771>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=376>], [<tf.Tensor: shape=(), dtype=int64, numpy=771>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=1045>, <tf.Tensor: shape=(), dtype=int64, numpy=3>], [<tf.Tensor: shape=(), dtype=int64, numpy=376>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=1045>, <tf.Tensor: shape=(), dtype=int64, numpy=1>], [<tf.Tensor: shape=(), dtype=int64, numpy=3>, <tf.Tensor: shape=(), dtype=int64, numpy=1045>], [<tf.Tensor: shape=(), dtype=int64, numpy=1>, <tf.Tensor: shape=(), dtype=int64, numpy=1045>]]\n"
          ]
        }
      ],
      "source": [
        "def to_cbow(sent,window_size=2):\n",
        "    res = []\n",
        "    for i,x in enumerate(sent):\n",
        "        for j in range(max(0,i-window_size),min(i+window_size+1,len(sent))):\n",
        "            if i!=j:\n",
        "                res.append([sent[j],x])\n",
        "    return res\n",
        "\n",
        "print(to_cbow(['I','like','to','train','networks']))\n",
        "print(to_cbow(vectorizer('I like to train networks')))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's prepare the training dataset. We will go through all news, call `to_cbow` to get the list of word pairs, and add those pairs to `X` and `Y`. For the sake of time, we will only consider first 10k news items - you can easily remove the limitation in case you have more time to wait, and want to get better embeddings :)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "id": "54b-Gd9TieXo"
      },
      "outputs": [],
      "source": [
        "X = []\n",
        "Y = []\n",
        "for i,x in zip(range(10000),ds_train.map(extract_text).as_numpy_iterator()):\n",
        "    for w1, w2 in to_cbow(vectorizer(x),window_size=1):\n",
        "        X.append(tf.expand_dims(w1,0))\n",
        "        Y.append(tf.expand_dims(w2,0))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We will also convert that data to one dataset, and batch it for training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "id": "AbLUcojlieXo"
      },
      "outputs": [],
      "source": [
        "ds = tf.data.Dataset.from_tensor_slices((X,Y)).batch(256)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's do the actual training. We will use `SGD` optimizer with pretty high learning rate. You can also try playing around with other optimizers, such as `Adam`. We will train for 200 epochs to begin with - and you can re-run this cell if you want even lower loss."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xAcGAQtVieXp",
        "outputId": "bbab8c44-de25-49b9-ec3f-07db878a0818"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Epoch 1/200\n"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/keras/optimizer_v2/gradient_descent.py:102: UserWarning: The `lr` argument is deprecated, use `learning_rate` instead.\n",
            "  super(SGD, self).__init__(name, **kwargs)\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.6134\n",
            "Epoch 2/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.5431\n",
            "Epoch 3/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.5029\n",
            "Epoch 4/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4754\n",
            "Epoch 5/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4548\n",
            "Epoch 6/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4382\n",
            "Epoch 7/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4243\n",
            "Epoch 8/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4123\n",
            "Epoch 9/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.4019\n",
            "Epoch 10/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3926\n",
            "Epoch 11/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3843\n",
            "Epoch 12/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3767\n",
            "Epoch 13/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3697\n",
            "Epoch 14/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3632\n",
            "Epoch 15/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3571\n",
            "Epoch 16/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3513\n",
            "Epoch 17/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3459\n",
            "Epoch 18/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3408\n",
            "Epoch 19/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3359\n",
            "Epoch 20/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3312\n",
            "Epoch 21/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3266\n",
            "Epoch 22/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3223\n",
            "Epoch 23/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3181\n",
            "Epoch 24/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3140\n",
            "Epoch 25/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3101\n",
            "Epoch 26/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3062\n",
            "Epoch 27/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.3025\n",
            "Epoch 28/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2989\n",
            "Epoch 29/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2953\n",
            "Epoch 30/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2919\n",
            "Epoch 31/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2885\n",
            "Epoch 32/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2852\n",
            "Epoch 33/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2819\n",
            "Epoch 34/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2787\n",
            "Epoch 35/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2756\n",
            "Epoch 36/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2725\n",
            "Epoch 37/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2695\n",
            "Epoch 38/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2665\n",
            "Epoch 39/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2636\n",
            "Epoch 40/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2607\n",
            "Epoch 41/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2578\n",
            "Epoch 42/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2550\n",
            "Epoch 43/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2523\n",
            "Epoch 44/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2495\n",
            "Epoch 45/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2468\n",
            "Epoch 46/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2442\n",
            "Epoch 47/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2416\n",
            "Epoch 48/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2390\n",
            "Epoch 49/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2364\n",
            "Epoch 50/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2339\n",
            "Epoch 51/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2314\n",
            "Epoch 52/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2290\n",
            "Epoch 53/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2266\n",
            "Epoch 54/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2242\n",
            "Epoch 55/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2218\n",
            "Epoch 56/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2195\n",
            "Epoch 57/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2172\n",
            "Epoch 58/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2149\n",
            "Epoch 59/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2126\n",
            "Epoch 60/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2104\n",
            "Epoch 61/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2082\n",
            "Epoch 62/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2060\n",
            "Epoch 63/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2038\n",
            "Epoch 64/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.2017\n",
            "Epoch 65/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1996\n",
            "Epoch 66/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1975\n",
            "Epoch 67/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1954\n",
            "Epoch 68/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1933\n",
            "Epoch 69/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1913\n",
            "Epoch 70/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1893\n",
            "Epoch 71/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1873\n",
            "Epoch 72/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1853\n",
            "Epoch 73/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1833\n",
            "Epoch 74/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1814\n",
            "Epoch 75/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1795\n",
            "Epoch 76/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1775\n",
            "Epoch 77/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1756\n",
            "Epoch 78/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1737\n",
            "Epoch 79/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1719\n",
            "Epoch 80/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1700\n",
            "Epoch 81/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1682\n",
            "Epoch 82/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1663\n",
            "Epoch 83/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1645\n",
            "Epoch 84/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1627\n",
            "Epoch 85/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1609\n",
            "Epoch 86/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1592\n",
            "Epoch 87/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1574\n",
            "Epoch 88/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1557\n",
            "Epoch 89/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1539\n",
            "Epoch 90/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1522\n",
            "Epoch 91/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1505\n",
            "Epoch 92/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1488\n",
            "Epoch 93/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1471\n",
            "Epoch 94/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1454\n",
            "Epoch 95/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1438\n",
            "Epoch 96/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1421\n",
            "Epoch 97/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1405\n",
            "Epoch 98/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1389\n",
            "Epoch 99/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1372\n",
            "Epoch 100/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1356\n",
            "Epoch 101/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1341\n",
            "Epoch 102/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1325\n",
            "Epoch 103/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1309\n",
            "Epoch 104/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1293\n",
            "Epoch 105/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1278\n",
            "Epoch 106/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1263\n",
            "Epoch 107/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1247\n",
            "Epoch 108/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1232\n",
            "Epoch 109/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1217\n",
            "Epoch 110/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1202\n",
            "Epoch 111/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1187\n",
            "Epoch 112/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1173\n",
            "Epoch 113/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1158\n",
            "Epoch 114/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1144\n",
            "Epoch 115/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1129\n",
            "Epoch 116/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1115\n",
            "Epoch 117/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1101\n",
            "Epoch 118/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1086\n",
            "Epoch 119/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1072\n",
            "Epoch 120/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1058\n",
            "Epoch 121/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1045\n",
            "Epoch 122/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1031\n",
            "Epoch 123/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1017\n",
            "Epoch 124/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.1004\n",
            "Epoch 125/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0990\n",
            "Epoch 126/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0977\n",
            "Epoch 127/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0963\n",
            "Epoch 128/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0950\n",
            "Epoch 129/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0937\n",
            "Epoch 130/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0924\n",
            "Epoch 131/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0911\n",
            "Epoch 132/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0898\n",
            "Epoch 133/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0885\n",
            "Epoch 134/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0873\n",
            "Epoch 135/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0860\n",
            "Epoch 136/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0848\n",
            "Epoch 137/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0835\n",
            "Epoch 138/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0823\n",
            "Epoch 139/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0810\n",
            "Epoch 140/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0798\n",
            "Epoch 141/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0786\n",
            "Epoch 142/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0774\n",
            "Epoch 143/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0762\n",
            "Epoch 144/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0750\n",
            "Epoch 145/200\n",
            "2156/2156 [==============================] - 8s 4ms/step - loss: 5.0739\n",
            "Epoch 146/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0727\n",
            "Epoch 147/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0715\n",
            "Epoch 148/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0704\n",
            "Epoch 149/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0692\n",
            "Epoch 150/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0681\n",
            "Epoch 151/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0670\n",
            "Epoch 152/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0658\n",
            "Epoch 153/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0647\n",
            "Epoch 154/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0636\n",
            "Epoch 155/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0625\n",
            "Epoch 156/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0614\n",
            "Epoch 157/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0603\n",
            "Epoch 158/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0593\n",
            "Epoch 159/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0582\n",
            "Epoch 160/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0571\n",
            "Epoch 161/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0561\n",
            "Epoch 162/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0550\n",
            "Epoch 163/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0539\n",
            "Epoch 164/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0529\n",
            "Epoch 165/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0519\n",
            "Epoch 166/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0508\n",
            "Epoch 167/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0498\n",
            "Epoch 168/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0488\n",
            "Epoch 169/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0478\n",
            "Epoch 170/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0468\n",
            "Epoch 171/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0458\n",
            "Epoch 172/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0448\n",
            "Epoch 173/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0438\n",
            "Epoch 174/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0428\n",
            "Epoch 175/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0418\n",
            "Epoch 176/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0409\n",
            "Epoch 177/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0399\n",
            "Epoch 178/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0389\n",
            "Epoch 179/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0380\n",
            "Epoch 180/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0370\n",
            "Epoch 181/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0361\n",
            "Epoch 182/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0351\n",
            "Epoch 183/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0342\n",
            "Epoch 184/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0333\n",
            "Epoch 185/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0323\n",
            "Epoch 186/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0314\n",
            "Epoch 187/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0305\n",
            "Epoch 188/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0296\n",
            "Epoch 189/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0287\n",
            "Epoch 190/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0278\n",
            "Epoch 191/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0269\n",
            "Epoch 192/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0260\n",
            "Epoch 193/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0251\n",
            "Epoch 194/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0242\n",
            "Epoch 195/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0233\n",
            "Epoch 196/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0225\n",
            "Epoch 197/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0216\n",
            "Epoch 198/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0207\n",
            "Epoch 199/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0199\n",
            "Epoch 200/200\n",
            "2156/2156 [==============================] - 7s 3ms/step - loss: 5.0190\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7ff7e52572d0>"
            ]
          },
          "execution_count": 102,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model.compile(optimizer=keras.optimizers.SGD(lr=0.1),loss='sparse_categorical_crossentropy')\n",
        "model.fit(ds,epochs=200)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Trying out Word2Vec\n",
        "\n",
        "To use Word2Vec, let's extract vectors corresponding to all words in our vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "id": "r8TatcXjkU_t"
      },
      "outputs": [],
      "source": [
        "vectors = embedder(vectorizer(vocab))\n",
        "vectors = tf.reshape(vectors,(-1,30)) # we need reshape to get rid of extra dimension"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's see, for example, how the word **Paris** is encoded into a vector:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bz6tAeLzieXp",
        "outputId": "c0422bc7-ca08-4f99-bced-e46d8b9b93e3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "tf.Tensor(\n",
            "[-0.13308628  0.50972325  0.00344684  0.185389   -0.03176536  0.22262476\n",
            " -0.3856765  -0.6854793   0.5185803  -0.7215402  -0.16101503  0.15622072\n",
            "  0.00653811 -0.14954254  0.03379822 -0.01243829  0.27907634 -0.32538188\n",
            "  0.21718933  0.31112966 -0.24142407  0.15589055  0.2915561   0.19029242\n",
            "  0.08425518 -0.0941902  -0.54313695 -0.24854654  0.26196313  0.18027727], shape=(30,), dtype=float32)\n"
          ]
        }
      ],
      "source": [
        "paris_vec = embedder(vectorizer('paris'))[0]\n",
        "print(paris_vec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "It is interesting to use Word2Vec to look for synonyms. The following function will return `n` closest words to a given input. To find them, we compute the norm of $|w_i - v|$, where $v$ is the vector corresponding to our input word, and $w_i$ is the encoding of $i$-th word in the vocabulary. We then sort the array and return corresponding indices using `argsort`, and take first `n` elements of the list, which encode positions of closest words in the vocabulary.  "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NlZyi-_olFar",
        "outputId": "4e4543db-4472-4b46-affd-71f39df4d342"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['paris', 'philippines', 'seoul', 'jakarta', 'zoo']"
            ]
          },
          "execution_count": 105,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def close_words(x,n=5):\n",
        "  vec = embedder(vectorizer(x))[0]\n",
        "  top5 = np.linalg.norm(vectors-vec,axis=1).argsort()[:n]\n",
        "  return [ vocab[x] for x in top5 ]\n",
        "\n",
        "close_words('paris')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-dQq7xeAln0U",
        "outputId": "3fdf5f9b-554c-4546-d84e-b88a96dc0e01"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['china', 'russia', 'pakistan', 'israel', 'turkey']"
            ]
          },
          "execution_count": 112,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "close_words('china')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fJXqK26b29sa",
        "outputId": "7a51e71f-1a1d-409e-c050-cffebb145095"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['official', 'military', 'office', 'police', 'sources']"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "close_words('official')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "My0VeTDd3Ji8"
      },
      "source": [
        "## Takeaway\n",
        "\n",
        "Using clever techniques such as CBoW, we can train Word2Vec model. You may also try to train skip-gram model that is trained to predict the neighboring word given the central one, and see how well it performs. "
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "CBoW-TF.ipynb",
      "provenance": []
    },
    "interpreter": {
      "hash": "16af2a8bbb083ea23e5e41c7f5787656b2ce26968575d8763f2c4b17f9cd711f"
    },
    "kernelspec": {
      "display_name": "Python 3.8.12 ('py38')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.12"
    },
    "orig_nbformat": 4,
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "027dbec8c35d4ec3ab43ed2878a32eb9": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_19ece8654d8149ac87538fd162bd1aeb",
            "placeholder": "​",
            "style": "IPY_MODEL_e630a16615414ceeba5868d162f55a20",
            "value": " 119999/120000 [00:00&lt;00:00, 291181.43 examples/s]"
          }
        },
        "074db709c0a14cd4acfe13ed54e92cbc": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "0798ebda763a40bc86235a40dfc1adec": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "13ee77a308634d928662b651ad2bb9e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_260fcdf1d0404d149732d566b6ccbbab",
              "IPY_MODEL_f3ad889117ba43b783e34a82113b325c",
              "IPY_MODEL_4dace863d2be4961ae72c729405da6cc"
            ],
            "layout": "IPY_MODEL_2bbd772ad6284273b3cf97c6afeda6e0"
          }
        },
        "15d88a4607524d07be1f3b91345243ba": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "19ece8654d8149ac87538fd162bd1aeb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1a467f2a5e4b421db16e4fd4329f9bc1": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_40ac80ed694940c191002b92e30a9f39",
              "IPY_MODEL_4d62b43b387b45c89471f9820c041718",
              "IPY_MODEL_ee572078162448bd89bd2c52fbe39aa7"
            ],
            "layout": "IPY_MODEL_2bddf650279242d7a259db4209de3253"
          }
        },
        "1cf5cb0c39cc4c5ca66be45895aa1860": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bfb1e75c5bc744f28544515c660a0b9b",
            "placeholder": "​",
            "style": "IPY_MODEL_62aa64e5318d445f844b8083ae6c40f4",
            "value": ""
          }
        },
        "2252a0aeca7c4b7784370704181f1628": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "24283dc34e944591877888871a5e584b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "danger",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dc49356d1ba943ad87b88ee6e451e7fb",
            "max": 120000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7095c6398b0c433db8c4284620c9e335",
            "value": 119999
          }
        },
        "25c18271a4594c1fa8c694d50dd356a7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "260fcdf1d0404d149732d566b6ccbbab": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ca245734f2f54c4e805e761d23652eca",
            "placeholder": "​",
            "style": "IPY_MODEL_bcedd81ebcef4d9ca31eea1ae4ab795d",
            "value": ""
          }
        },
        "295d430b24444986a46a9382c5d5f80d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "danger",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_44e94cb4f240446da537579caa8e6d2f",
            "max": 7600,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8591f95a707d4214a17e9f187df6e1c4",
            "value": 7599
          }
        },
        "2bbd772ad6284273b3cf97c6afeda6e0": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2bddf650279242d7a259db4209de3253": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "30224d6b4c274faf85dbd4d2c1892aa7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_0798ebda763a40bc86235a40dfc1adec",
            "placeholder": "​",
            "style": "IPY_MODEL_bcd9ea70684742b6991d4e2c7556efa6",
            "value": "100%"
          }
        },
        "30ffd5f13a524b0eabd5d2f20885ce50": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3832f856d7644da2aedc8dc843732269": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3e31ac504d4242b8be208b68219bf064",
              "IPY_MODEL_bb16a77431264209935f8e747918e430",
              "IPY_MODEL_b1c32e56d326473db36bcda7abca7010"
            ],
            "layout": "IPY_MODEL_7b38647b718d4cb58c20480e6387d749"
          }
        },
        "3e31ac504d4242b8be208b68219bf064": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_490cee33a6c14ead8209f36ca7dc1351",
            "placeholder": "​",
            "style": "IPY_MODEL_5eee10ee14ff47f9bea181870bb973e3",
            "value": "Extraction completed...: 100%"
          }
        },
        "3edc08cbd6774e7485d42bdd13164ed6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1cf5cb0c39cc4c5ca66be45895aa1860",
              "IPY_MODEL_b70cac0930eb4d2da24a9f5b042f4a9e",
              "IPY_MODEL_b38d4f6271234adfb41e8309a115e95b"
            ],
            "layout": "IPY_MODEL_d7ef0d4ec19749c3bee8a0be8ad2d468"
          }
        },
        "40ac80ed694940c191002b92e30a9f39": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9bba163f10a4461ca232b54d79c9b74d",
            "placeholder": "​",
            "style": "IPY_MODEL_f91bb89f9f144f8e97f8f0c97f7d9f55",
            "value": "Dl Size...: "
          }
        },
        "44e94cb4f240446da537579caa8e6d2f": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "490cee33a6c14ead8209f36ca7dc1351": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4d62b43b387b45c89471f9820c041718": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bec242f072394c4cabc6f39e43350603",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_74a70de4caa94d358606308df816725b",
            "value": 0
          }
        },
        "4dace863d2be4961ae72c729405da6cc": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_abaa80c91f5642649996c844ceb0fcd1",
            "placeholder": "​",
            "style": "IPY_MODEL_8655ea4b7b6c4399adaf6e04613869ea",
            "value": " 7221/0 [00:01&lt;00:00, 4326.56 examples/s]"
          }
        },
        "5022afdb2026474081a3f54cb4c81351": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "52d3dd9cf2994e6da25a10ea42be4beb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_82ee379245d64fc39d1ed9a2586e20a2",
              "IPY_MODEL_24283dc34e944591877888871a5e584b",
              "IPY_MODEL_027dbec8c35d4ec3ab43ed2878a32eb9"
            ],
            "layout": "IPY_MODEL_b459e5715d3b44eeb379108510261336"
          }
        },
        "532f40fc7a1e4826b4495be24ee0f8ed": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "55cc8f6b3b0c49ddbc4bfe526906ccbf": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5ac5967b468d4af59cea0693ce9a8217": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5eee10ee14ff47f9bea181870bb973e3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "62aa64e5318d445f844b8083ae6c40f4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7095c6398b0c433db8c4284620c9e335": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "713f10e7274e4f0484d34759b5505842": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "72693d5e2c034fb9ad03a32a8eb2999f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "74a70de4caa94d358606308df816725b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "75dd999664ac40f18168f6e1870a878e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7916d209cbe04da2912830b16e5f747c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7ab6c716b4a04052bf048d0ede312365": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "7b38647b718d4cb58c20480e6387d749": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "82ee379245d64fc39d1ed9a2586e20a2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b58f7dc6368b42d0a387e47bce4ce88e",
            "placeholder": "​",
            "style": "IPY_MODEL_2252a0aeca7c4b7784370704181f1628",
            "value": "100%"
          }
        },
        "8591f95a707d4214a17e9f187df6e1c4": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8655ea4b7b6c4399adaf6e04613869ea": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "91d15913f17040da828ece1c3b5fa6c6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "94db6867a26a4c988f549d20b3cb51f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "9a4eedfb4c6a466ba6f6f21ce76a64bb": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_75dd999664ac40f18168f6e1870a878e",
            "placeholder": "​",
            "style": "IPY_MODEL_91d15913f17040da828ece1c3b5fa6c6",
            "value": " 7599/7600 [00:00&lt;00:00, 106078.71 examples/s]"
          }
        },
        "9bba163f10a4461ca232b54d79c9b74d": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9d4d315121e9440c8578a62fbe88e415": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9e28f7897bf142aebd4d374559320812": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9f5a040885564d41934f6c458761bf33": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9d4d315121e9440c8578a62fbe88e415",
            "placeholder": "​",
            "style": "IPY_MODEL_a1f8b53e8a1d4ebd8ed0116219490877",
            "value": "Dl Completed...: "
          }
        },
        "a1f8b53e8a1d4ebd8ed0116219490877": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a585d2e5ac5240679587990dbf53dfd2": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a6a32befb28542228cde3d444d6411f6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9f5a040885564d41934f6c458761bf33",
              "IPY_MODEL_a7651b06cb974b52a35e34e8f96c226c",
              "IPY_MODEL_ee267b7dcf05457b8e3f545df150f09f"
            ],
            "layout": "IPY_MODEL_55cc8f6b3b0c49ddbc4bfe526906ccbf"
          }
        },
        "a7651b06cb974b52a35e34e8f96c226c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_074db709c0a14cd4acfe13ed54e92cbc",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_713f10e7274e4f0484d34759b5505842",
            "value": 0
          }
        },
        "abaa80c91f5642649996c844ceb0fcd1": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b1c32e56d326473db36bcda7abca7010": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bbce51f4b75a4f999f4b3c170083e724",
            "placeholder": "​",
            "style": "IPY_MODEL_25c18271a4594c1fa8c694d50dd356a7",
            "value": " 1/1 [00:00&lt;00:00,  2.47 file/s]"
          }
        },
        "b26304339073463b9f0ba2cce4835d13": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b38d4f6271234adfb41e8309a115e95b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_30ffd5f13a524b0eabd5d2f20885ce50",
            "placeholder": "​",
            "style": "IPY_MODEL_5022afdb2026474081a3f54cb4c81351",
            "value": " 119817/0 [00:28&lt;00:00, 3712.60 examples/s]"
          }
        },
        "b459e5715d3b44eeb379108510261336": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b58f7dc6368b42d0a387e47bce4ce88e": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b70cac0930eb4d2da24a9f5b042f4a9e": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "info",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7ab6c716b4a04052bf048d0ede312365",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_94db6867a26a4c988f549d20b3cb51f3",
            "value": 1
          }
        },
        "bb16a77431264209935f8e747918e430": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_15d88a4607524d07be1f3b91345243ba",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_a585d2e5ac5240679587990dbf53dfd2",
            "value": 1
          }
        },
        "bbce51f4b75a4f999f4b3c170083e724": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bcd9ea70684742b6991d4e2c7556efa6": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bcedd81ebcef4d9ca31eea1ae4ab795d": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "be1b974e61b44ecd807a77a94f6f7991": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bec242f072394c4cabc6f39e43350603": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "20px"
          }
        },
        "bfb1e75c5bc744f28544515c660a0b9b": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca245734f2f54c4e805e761d23652eca": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d7ef0d4ec19749c3bee8a0be8ad2d468": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc49356d1ba943ad87b88ee6e451e7fb": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e630a16615414ceeba5868d162f55a20": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ee267b7dcf05457b8e3f545df150f09f": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_5ac5967b468d4af59cea0693ce9a8217",
            "placeholder": "​",
            "style": "IPY_MODEL_7916d209cbe04da2912830b16e5f747c",
            "value": " 0/0 [00:00&lt;?, ? url/s]"
          }
        },
        "ee572078162448bd89bd2c52fbe39aa7": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_be1b974e61b44ecd807a77a94f6f7991",
            "placeholder": "​",
            "style": "IPY_MODEL_72693d5e2c034fb9ad03a32a8eb2999f",
            "value": " 0/0 [00:00&lt;?, ? MiB/s]"
          }
        },
        "f3ad889117ba43b783e34a82113b325c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "info",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_532f40fc7a1e4826b4495be24ee0f8ed",
            "max": 1,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_b26304339073463b9f0ba2cce4835d13",
            "value": 1
          }
        },
        "f91bb89f9f144f8e97f8f0c97f7d9f55": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fc94257ae5094ce0b04695ad29bdf72b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_30224d6b4c274faf85dbd4d2c1892aa7",
              "IPY_MODEL_295d430b24444986a46a9382c5d5f80d",
              "IPY_MODEL_9a4eedfb4c6a466ba6f6f21ce76a64bb"
            ],
            "layout": "IPY_MODEL_9e28f7897bf142aebd4d374559320812"
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}