Basics_of_LSTM.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOYlrZ89EXnWz359Yf7hE5D",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/NurFortuna/Deep_Learning_with_Tensorflow_notes/blob/main/Basics_of_LSTM.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## RNN\n",
        "### Recurrent Neural Networks(RNN)'de kısaca bir sonraki katmanın çıktısı(output)  bir sonraki girdiye(input)' eklenir ve aynı katmana geri beslenir. \n",
        "\n",
        "### RNN'lerde girdiler birbiri ile ilişkilidir.\n",
        "\n",
        "### RNN'ler önceki girdileri hatırlar ve her girdi için ilişki kurar.\n",
        "\n",
        "### Belirli bir noktaya kadar yapılan analizi hatırlayabilir. RNN'nin hafızası olarak düşünebiliriz.\n",
        "\n",
        "### Bir veri kümesindeki noktalar, veri kümesindeki diğer noktalara bağımlı olduğunda, verilerin sıralı veriler(Sequential data) olduğu söylenir.\n",
        "\n",
        "### RNN sequential data'yı işleyebilen bir mekanizmaya sahiptir. \n",
        "\n",
        "![resim](https://miro.medium.com/max/1400/1*HgAY1lLMYSANqtgTgwWeXQ.png)"
      ],
      "metadata": {
        "id": "zD1n1tIDlyeo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##LSTM\n",
        "\n",
        "### LSTM  üç lojistik yapıdan oluşur. \n",
        "\n",
        "### LSTM yapısı içerisindeki kapılar(gate) neyin hatırlanacağını, neyin unutulacağını belirler. Yani gelen girdi önemsizse unutulur, Önemliyse bir sonraki aşamaya aktarılır.\n",
        "\n",
        "### Forget gate(unutma kapısı): Hangi bilginin tutulacağını veya unutalacağına karar verir. \n",
        "\n",
        "### Input gate(girdi kapısı) Cell state'i güncellemek için kullanılır.\n",
        "\n",
        "### Output gate(Çıktı kapısı) Bir sonraki katmana gönderilecek değere karar verir.\n",
        "\n",
        "\n",
        "\n",
        "![resim](https://miro.medium.com/max/984/1*Mb_L_slY9rjMr8-IADHvwg.png)\n",
        "\n"
      ],
      "metadata": {
        "id": "v0tfoEbkouiz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install tensorflow==2.2.0"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "74_iwB63pJUa",
        "outputId": "522163a0-f99f-4659-8b9c-13626e094e47"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting tensorflow==2.2.0\n",
            "  Downloading tensorflow-2.2.0-cp38-cp38-manylinux2010_x86_64.whl (516.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m516.3/516.3 MB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting scipy==1.4.1\n",
            "  Downloading scipy-1.4.1-cp38-cp38-manylinux1_x86_64.whl (26.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m26.0/26.0 MB\u001b[0m \u001b[31m19.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.4.0)\n",
            "Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.51.1)\n",
            "Collecting tensorflow-estimator<2.3.0,>=2.2.0\n",
            "  Downloading tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m454.6/454.6 KB\u001b[0m \u001b[31m28.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.6.3)\n",
            "Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (0.38.4)\n",
            "Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (3.19.6)\n",
            "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.15.0)\n",
            "Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.1.2)\n",
            "Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.14.1)\n",
            "Collecting tensorboard<2.3.0,>=2.2.0\n",
            "  Downloading tensorboard-2.2.2-py3-none-any.whl (3.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.0/3.0 MB\u001b[0m \u001b[31m40.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting h5py<2.11.0,>=2.10.0\n",
            "  Downloading h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.9/2.9 MB\u001b[0m \u001b[31m34.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (0.2.0)\n",
            "Collecting gast==0.3.3\n",
            "  Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)\n",
            "Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (1.21.6)\n",
            "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (2.2.0)\n",
            "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.8/dist-packages (from tensorflow==2.2.0) (3.3.0)\n",
            "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.8.1)\n",
            "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.6)\n",
            "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.25.1)\n",
            "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (57.4.0)\n",
            "Collecting google-auth<2,>=1.6.3\n",
            "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 KB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.0.1)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.8/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.4.1)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.8/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.9)\n",
            "Collecting cachetools<5.0,>=2.0.0\n",
            "  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.8/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.2.8)\n",
            "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.3.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.8/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (6.0.0)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.24.3)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.10)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2022.12.7)\n",
            "Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python3.8/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.0.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.12.0)\n",
            "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.8/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.8)\n",
            "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.8/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.2.2)\n",
            "Installing collected packages: tensorflow-estimator, scipy, h5py, gast, cachetools, google-auth, tensorboard, tensorflow\n",
            "  Attempting uninstall: tensorflow-estimator\n",
            "    Found existing installation: tensorflow-estimator 2.9.0\n",
            "    Uninstalling tensorflow-estimator-2.9.0:\n",
            "      Successfully uninstalled tensorflow-estimator-2.9.0\n",
            "  Attempting uninstall: scipy\n",
            "    Found existing installation: scipy 1.7.3\n",
            "    Uninstalling scipy-1.7.3:\n",
            "      Successfully uninstalled scipy-1.7.3\n",
            "  Attempting uninstall: h5py\n",
            "    Found existing installation: h5py 3.1.0\n",
            "    Uninstalling h5py-3.1.0:\n",
            "      Successfully uninstalled h5py-3.1.0\n",
            "  Attempting uninstall: gast\n",
            "    Found existing installation: gast 0.4.0\n",
            "    Uninstalling gast-0.4.0:\n",
            "      Successfully uninstalled gast-0.4.0\n",
            "  Attempting uninstall: cachetools\n",
            "    Found existing installation: cachetools 5.3.0\n",
            "    Uninstalling cachetools-5.3.0:\n",
            "      Successfully uninstalled cachetools-5.3.0\n",
            "  Attempting uninstall: google-auth\n",
            "    Found existing installation: google-auth 2.16.0\n",
            "    Uninstalling google-auth-2.16.0:\n",
            "      Successfully uninstalled google-auth-2.16.0\n",
            "  Attempting uninstall: tensorboard\n",
            "    Found existing installation: tensorboard 2.9.1\n",
            "    Uninstalling tensorboard-2.9.1:\n",
            "      Successfully uninstalled tensorboard-2.9.1\n",
            "  Attempting uninstall: tensorflow\n",
            "    Found existing installation: tensorflow 2.9.2\n",
            "    Uninstalling tensorflow-2.9.2:\n",
            "      Successfully uninstalled tensorflow-2.9.2\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "xarray-einstats 0.5.1 requires scipy>=1.6, but you have scipy 1.4.1 which is incompatible.\n",
            "plotnine 0.8.0 requires scipy>=1.5.0, but you have scipy 1.4.1 which is incompatible.\n",
            "jaxlib 0.3.25+cuda11.cudnn805 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\n",
            "jax 0.3.25 requires scipy>=1.5, but you have scipy 1.4.1 which is incompatible.\n",
            "google-api-core 2.11.0 requires google-auth<3.0dev,>=2.14.1, but you have google-auth 1.35.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed cachetools-4.2.4 gast-0.3.3 google-auth-1.35.0 h5py-2.10.0 scipy-1.4.1 tensorboard-2.2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf"
      ],
      "metadata": {
        "id": "LgE_9M0Zs_us"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Bir LSTM hücresine sahip bir network oluşturuyoruz. "
      ],
      "metadata": {
        "id": "2azapUUQteAo"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "LSTM_CELL_SIZE = 4  \n",
        "\n",
        "state = (tf.zeros([1,LSTM_CELL_SIZE]),)*2\n",
        "state"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LdL5CQD3tGIT",
        "outputId": "1605aaf7-71dd-4b5f-f07f-29173ca6fa1d"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0., 0., 0., 0.]], dtype=float32)>,\n",
              " <tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0., 0., 0., 0.]], dtype=float32)>)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "lstm = tf.keras.layers.LSTM(LSTM_CELL_SIZE, return_sequences=True, return_state=True)\n",
        "\n",
        "lstm.states=state\n",
        "\n",
        "print(lstm.states)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FqvLYRqlubl2",
        "outputId": "35c16cdb-f2da-4c52-d3c2-6d4c320f10af"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(<tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0., 0., 0., 0.]], dtype=float32)>, <tf.Tensor: shape=(1, 4), dtype=float32, numpy=array([[0., 0., 0., 0.]], dtype=float32)>)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Batch size x time steps x features.\n",
        "sample_input = tf.constant([[3,3,2,2,2,2]],dtype=tf.float32)\n",
        "\n",
        "batch_size = 1\n",
        "sentence_max_length = 1\n",
        "n_features = 6\n",
        "\n",
        "new_shape = (batch_size, sentence_max_length, n_features)\n",
        "\n",
        "inputs = tf.constant(np.reshape(sample_input, new_shape), dtype = tf.float32)"
      ],
      "metadata": {
        "id": "EeKTmIvcuimm"
      },
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "output, final_memory_state, final_carry_state = lstm(inputs)"
      ],
      "metadata": {
        "id": "5yuHV0gXvOWD"
      },
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('Output : ', tf.shape(output))\n",
        "\n",
        "print('Memory : ',tf.shape(final_memory_state))\n",
        "\n",
        "print('Carry state : ',tf.shape(final_carry_state))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kpYFnnK7vQHY",
        "outputId": "5e9c7c07-ab17-4ba7-f021-1948479fe0bc"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Output :  tf.Tensor([1 1 4], shape=(3,), dtype=int32)\n",
            "Memory :  tf.Tensor([1 4], shape=(2,), dtype=int32)\n",
            "Carry state :  tf.Tensor([1 4], shape=(2,), dtype=int32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Output shape'imiz (1,1,4) olarak çıktı. 1 batches, 1 dizimiz(sequence) elemente, 4 ise çıktının boyunu"
      ],
      "metadata": {
        "id": "5k0grn9L51az"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Stacked LSTM"
      ],
      "metadata": {
        "id": "3cIkU8Kxvl1N"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cells = []"
      ],
      "metadata": {
        "id": "bAkbqQRuvRxC"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "İlk layer(katman) LTSM'in oluşturulması"
      ],
      "metadata": {
        "id": "t8JTFxP1vwT5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "LSTM_CELL_SIZE_1 = 4 #4 hidden nodes\n",
        "cell1 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_1)\n",
        "cells.append(cell1)"
      ],
      "metadata": {
        "id": "YenOxOTkvvrW"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "ikinci layer(katman) LTSM'in oluşturulması"
      ],
      "metadata": {
        "id": "mpJs_ckOwFsr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "LSTM_CELL_SIZE_2 = 5 #5 hidden nodes\n",
        "cell2 = tf.keras.layers.LSTMCell(LSTM_CELL_SIZE_2)\n",
        "cells.append(cell2)"
      ],
      "metadata": {
        "id": "TZrfv-DRwBZo"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Çok katmanlı LSTM oluşturmak içi `tf.keras.layers.StackedRNNCells` kullanırız."
      ],
      "metadata": {
        "id": "IZxr06_kwgZX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stacked_lstm =  tf.keras.layers.StackedRNNCells(cells)"
      ],
      "metadata": {
        "id": "CEdRahUIwJm-"
      },
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Batch size x time steps x features.\n",
        "sample_input = [[[1,2,3,4,3,2], [1,2,1,1,1,2],[1,2,2,2,2,2]],[[1,2,3,4,3,2],[3,2,2,1,1,2],[0,0,0,0,3,2]]]\n",
        "sample_input\n",
        "\n",
        "batch_size = 2\n",
        "time_steps = 3\n",
        "features = 6\n",
        "new_shape = (batch_size, time_steps, features)\n",
        "\n",
        "x = tf.constant(np.reshape(sample_input, new_shape), dtype = tf.float32)"
      ],
      "metadata": {
        "id": "HNiBs_ZE5M7v"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "output, final_memory_state, final_carry_state  = lstm_layer(x)"
      ],
      "metadata": {
        "id": "lXMGUz_B5QPc"
      },
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print('Output : ', tf.shape(output))\n",
        "\n",
        "print('Memory : ',tf.shape(final_memory_state))\n",
        "\n",
        "print('Carry state : ',tf.shape(final_carry_state))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ep0Ql4JO5R7Y",
        "outputId": "6645b8d5-26a6-41a0-cc36-c87c9b28f93c"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Output :  tf.Tensor([2 3 5], shape=(3,), dtype=int32)\n",
            "Memory :  tf.Tensor([2 2 4], shape=(3,), dtype=int32)\n",
            "Carry state :  tf.Tensor([2 2 5], shape=(3,), dtype=int32)\n"
          ]
        }
      ]
    }
  ]
}