NNs,_CNNs,_RNNs_and_LSTMs.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "**Comprehensive Tutorial on Artificial Neural Networks, Convolutional Neural Networks, Recurrent Neural Networks, and LSTMs using Python**"
      ],
      "metadata": {
        "id": "87516awoKfN6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this tutorial, we will cover the basics of Artificial Neural Networks (ANNs), Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long Short-Term Memory (LSTM) networks using Python. We'll provide both theoretical explanations and practical code examples to help you understand and implement these concepts effectively."
      ],
      "metadata": {
        "id": "JWAggC6kKpj1"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***Table of Contents***\n",
        "1. Introduction to Neural Networks: What are Neural Networks?, Neurons and Activation Functions and Forward Propagation\n",
        "\n",
        "2. Artificial Neural Networks (ANNs): Structure of ANNs, Backpropagation and Gradient Descent, Implementation Example\n",
        "\n",
        "3. Convolutional Neural Networks (CNNs): Motivation for CNNs, Convolutional and Pooling Layers, Implementation Example\n",
        "\n",
        "4. Recurrent Neural Networks (RNNs): Need for RNNs, Structure of RNNs, Backpropagation Through Time (BPTT), Implementation Example\n",
        "\n",
        "5. Long Short-Term Memory (LSTM) Networks: The Problem of Vanishing Gradient,LSTM Architecture, Implementation Example"
      ],
      "metadata": {
        "id": "9qs7wsOtKsby"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**1. Introduction to Neural Networks**\n",
        "\n",
        "***What are Neural Networks?***\n",
        "\n",
        "Neural Networks are a class of machine learning models inspired by the human brain's structure. They consist of interconnected units called neurons that process and transmit information. Neural Networks are widely used for tasks like classification, regression, and pattern recognition.\n",
        "\n",
        "***Neurons and Activation Functions:***\n",
        "\n",
        "A neuron in a neural network consists of an input layer, hidden layers, and an output layer. Each neuron takes inputs, performs a weighted sum, and passes the result through an activation function. Common activation functions include the sigmoid, ReLU (Rectified Linear Unit), and tanh.\n",
        "\n",
        "***Forward Propagation***\n",
        "\n",
        "Forward propagation is the process by which data flows through the neural network, from the input layer to the output layer. Neurons in each layer apply the weighted sum and activation function to produce outputs, which become inputs for the next layer."
      ],
      "metadata": {
        "id": "oC8LeHpOLceT"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "***2. Artificial Neural Networks (ANNs)***\n",
        "\n",
        "***Structure of ANNs:***\n",
        "\n",
        "ANNs consist of an input layer, one or more hidden layers, and an output layer. Neurons in the hidden layers learn to extract relevant features from the data. The number of neurons, layers, and their connectivity influence the network's capacity to learn complex patterns.\n",
        "\n",
        "***Backpropagation and Gradient Descent:***\n",
        "\n",
        "Backpropagation is the process of updating neural network weights to minimize the difference between predicted and actual outputs. This is done using gradient descent, which adjusts weights in the opposite direction of the gradient of the loss function with respect to the weights.\n",
        "\n",
        "***Implementation Example:***\n",
        "\n",
        "Let's create a simple ANN using Python and the Keras library:"
      ],
      "metadata": {
        "id": "QZCcLKVpLm-g"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "\n",
        "# Example data\n",
        "X_train = np.random.random((10000, 20))\n",
        "y_train = np.random.randint(2, size=(10000, 1))\n",
        "\n",
        "# Define input and output dimensions\n",
        "input_dim = X_train.shape[1]\n",
        "output_dim = 1  # Change this based on your problem\n",
        "\n",
        "# Create a Sequential model\n",
        "model = Sequential()\n",
        "\n",
        "# Add layers to the model\n",
        "model.add(Dense(units=64, activation='relu', input_dim=input_dim))\n",
        "model.add(Dense(units=32, activation='relu'))\n",
        "model.add(Dense(units=output_dim, activation='softmax'))\n",
        "\n",
        "# Compile the model\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "\n",
        "# Train the model\n",
        "model.fit(X_train, y_train, epochs=50, batch_size=32)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6fPjuFpeLyvv",
        "outputId": "cdce449d-9a02-497c-8298-1ad5675061fc"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/50\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: SyntaxWarning: In loss categorical_crossentropy, expected y_pred.shape to be (batch_size, num_classes) with num_classes > 1. Received: y_pred.shape=(None, 1). Consider using 'binary_crossentropy' if you only have 2 classes.\n",
            "  return dispatch_target(*args, **kwargs)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "313/313 [==============================] - 2s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 2/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 3/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 4/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 5/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 6/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 7/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 8/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 9/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 10/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 11/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 12/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 13/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 14/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 15/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 16/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 17/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 18/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 19/50\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 20/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 21/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 22/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 23/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 24/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 25/50\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 26/50\n",
            "313/313 [==============================] - 1s 4ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 27/50\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 28/50\n",
            "313/313 [==============================] - 1s 4ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 29/50\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 30/50\n",
            "313/313 [==============================] - 1s 3ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 31/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 32/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 33/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 34/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 35/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 36/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 37/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 38/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 39/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 40/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 41/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 42/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 43/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 44/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 45/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 46/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 47/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 48/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 49/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n",
            "Epoch 50/50\n",
            "313/313 [==============================] - 1s 2ms/step - loss: 0.0000e+00 - accuracy: 0.4990\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7e730411fa00>"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tuning Performance of Artificial Neural Networks (ANNs)\n",
        "To improve the performance of ANNs, you can consider the following techniques:\n",
        "\n",
        "1. Learning Rate Tuning:\n",
        "The learning rate controls the step size during weight updates. It's crucial to find an optimal learning rate to converge faster without overshooting. You can use learning rate schedulers to adjust the learning rate during training.\n",
        "\n",
        "To achieve this:\n",
        "\n",
        "```\n",
        "from keras.optimizers import SGD\n",
        "\n",
        "learning_rate = 0.01\n",
        "sgd = SGD(learning_rate=learning_rate)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['accuracy'])\n",
        "```\n",
        "\n",
        "2. Regularization:\n",
        "Regularization techniques like L1 or L2 regularization can prevent overfitting. They add a penalty term to the loss function based on the weights' magnitude.\n",
        "\n",
        "```\n",
        "from keras.regularizers import l2\n",
        "\n",
        "model.add(Dense(units=64, activation='relu', input_dim=input_dim, kernel_regularizer=l2(0.01)))\n",
        "\n",
        "```\n",
        "\n",
        "3. Dropout:\n",
        "Dropout randomly deactivates a fraction of neurons during each training step. This prevents over-reliance on specific neurons and improves generalization.\n",
        "\n",
        "```\n",
        "from keras.layers import Dropout\n",
        "\n",
        "model.add(Dense(units=64, activation='relu', input_dim=input_dim))\n",
        "model.add(Dropout(0.2))\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "jDDKzBf8Mwof"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "3. **Convolutional Neural Networks (CNNs):**\n",
        "\n",
        "**Motivation for CNNs:**\n",
        "\n",
        "CNNs are designed for image and spatial data. They utilize convolutional layers to automatically learn hierarchical features from the input. This makes them highly effective in tasks like image recognition.\n",
        "\n",
        "**Convolutional and Pooling Layers:**\n",
        "\n",
        "Convolutional layers apply filters to the input, capturing different features. Pooling layers reduce the spatial dimensions of the data, helping to retain important information while reducing computation.\n",
        "\n",
        "**Implementation Example:**\n",
        "\n",
        "Let's build a CNN using Python and Keras:"
      ],
      "metadata": {
        "id": "gjfJKeunRQdr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Conv2D, MaxPooling2D, Flatten, Dense\n",
        "\n",
        "# Example data\n",
        "X_train = np.random.random((10000, 20, 20, 3))  # Assuming image-like data with 3 channels\n",
        "y_train = np.random.randint(2, size=(10000, 10))\n",
        "\n",
        "# Define input shape\n",
        "input_shape = X_train.shape[1:]\n",
        "\n",
        "# Create a Sequential model\n",
        "model = Sequential()\n",
        "\n",
        "# Add convolutional and pooling layers\n",
        "model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "# Compile and train the model\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "model.fit(X_train, y_train, epochs=10, batch_size=32)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I92SMQD-RZpC",
        "outputId": "cd058aef-da8c-437b-990b-262811a39a33"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "313/313 [==============================] - 6s 16ms/step - loss: 15.4783 - accuracy: 0.0915\n",
            "Epoch 2/10\n",
            "313/313 [==============================] - 3s 11ms/step - loss: 33.8991 - accuracy: 0.0897\n",
            "Epoch 3/10\n",
            "313/313 [==============================] - 3s 9ms/step - loss: 73.7265 - accuracy: 0.0954\n",
            "Epoch 4/10\n",
            "313/313 [==============================] - 3s 9ms/step - loss: 110.3974 - accuracy: 0.0938\n",
            "Epoch 5/10\n",
            "313/313 [==============================] - 4s 12ms/step - loss: 127.2374 - accuracy: 0.0924\n",
            "Epoch 6/10\n",
            "313/313 [==============================] - 3s 11ms/step - loss: 179.1361 - accuracy: 0.0990\n",
            "Epoch 7/10\n",
            "313/313 [==============================] - 3s 9ms/step - loss: 211.2084 - accuracy: 0.0961\n",
            "Epoch 8/10\n",
            "313/313 [==============================] - 4s 11ms/step - loss: 239.9257 - accuracy: 0.1056\n",
            "Epoch 9/10\n",
            "313/313 [==============================] - 5s 15ms/step - loss: 275.1620 - accuracy: 0.1050\n",
            "Epoch 10/10\n",
            "313/313 [==============================] - 3s 9ms/step - loss: 269.3490 - accuracy: 0.1026\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7e72f1277df0>"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note the following:\n",
        "1. input_shape: I used the shape (20, 20, 3) for the input data assuming it's image-like data with 3 channels.\n",
        "\n",
        "2. The Dense layer's input size is automatically determined by the previous layer's output size, so you don't need to specify input_dim for it."
      ],
      "metadata": {
        "id": "fN16G-62SZ0e"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Tuning Performance of Convolutional Neural Networks (CNNs):\n",
        "\n",
        "Improving the performance of CNNs involves similar strategies, plus a few specific to image data:\n",
        "\n",
        "1. Data Augmentation:\n",
        "\n",
        "Data augmentation generates new training examples by applying random transformations to the existing data. This increases the diversity of the training set and helps the model generalize better.\n",
        "```\n",
        "from keras.preprocessing.image import ImageDataGenerator\n",
        "\n",
        "datagen = ImageDataGenerator(rotation_range=20, width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)\n",
        "datagen.fit(X_train)\n",
        "```\n",
        "2. Transfer Learning:\n",
        "\n",
        "Transfer learning involves using pre-trained models as a starting point and fine-tuning them for your specific task. This can save training time and improve results.\n",
        "```\n",
        "from keras.applications import VGG16\n",
        "\n",
        "base_model = VGG16(weights='imagenet', include_top=False, input_shape=(224, 224, 3))\n",
        "# Add your custom layers on top of the base model\n",
        "```"
      ],
      "metadata": {
        "id": "DCtp8FkzTiqs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**4. Recurrent Neural Networks (RNNs):**\n",
        "\n",
        "**Need for RNNs:**\n",
        "\n",
        "RNNs are designed for sequential data, where the order of inputs matters. They have a \"memory\" that allows them to consider previous inputs when processing the current input.\n",
        "\n",
        "**Structure of RNNs:**\n",
        "\n",
        "In an RNN, each neuron processes the current input and the output from the previous neuron. This allows RNNs to maintain temporal dependencies.\n",
        "\n",
        "**Backpropagation Through Time (BPTT):**\n",
        "\n",
        "BPTT is the extension of backpropagation for RNNs. It takes into account the sequence of inputs over time and adjusts weights accordingly.\n",
        "\n",
        "Let's implement a basic RNN using Python and Keras:"
      ],
      "metadata": {
        "id": "hTEVMF4NT2dj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from keras.models import Sequential\n",
        "from keras.layers import SimpleRNN, Dense\n",
        "\n",
        "# Example data\n",
        "X_train = np.random.random((10000, 20, 1))  # Assuming sequence data with one feature\n",
        "y_train = np.random.randint(2, size=(10000, 1))\n",
        "\n",
        "# Define input shape\n",
        "input_shape = X_train.shape[1:]\n",
        "\n",
        "# Create a Sequential model\n",
        "model = Sequential()\n",
        "\n",
        "# Add an RNN layer\n",
        "model.add(SimpleRNN(64, input_shape=input_shape))\n",
        "\n",
        "# Add a Dense output layer\n",
        "model.add(Dense(1, activation='sigmoid'))  # Use 'sigmoid' for binary classification\n",
        "\n",
        "# Compile and train the model\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "model.fit(X_train, y_train, epochs=10, batch_size=32)\n",
        "\n"
      ],
      "metadata": {
        "id": "7VHxXtsyM206",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0091fa5e-07ad-4f79-de5e-c192bbbc0e06"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "313/313 [==============================] - 3s 6ms/step - loss: 0.6968 - accuracy: 0.5071\n",
            "Epoch 2/10\n",
            "313/313 [==============================] - 2s 5ms/step - loss: 0.6953 - accuracy: 0.5000\n",
            "Epoch 3/10\n",
            "313/313 [==============================] - 2s 8ms/step - loss: 0.6948 - accuracy: 0.4958\n",
            "Epoch 4/10\n",
            "313/313 [==============================] - 2s 6ms/step - loss: 0.6940 - accuracy: 0.4975\n",
            "Epoch 5/10\n",
            "313/313 [==============================] - 2s 5ms/step - loss: 0.6936 - accuracy: 0.5010\n",
            "Epoch 6/10\n",
            "313/313 [==============================] - 2s 5ms/step - loss: 0.6938 - accuracy: 0.5004\n",
            "Epoch 7/10\n",
            "313/313 [==============================] - 2s 5ms/step - loss: 0.6938 - accuracy: 0.4988\n",
            "Epoch 8/10\n",
            "313/313 [==============================] - 2s 5ms/step - loss: 0.6939 - accuracy: 0.5057\n",
            "Epoch 9/10\n",
            "313/313 [==============================] - 2s 7ms/step - loss: 0.6938 - accuracy: 0.4999\n",
            "Epoch 10/10\n",
            "313/313 [==============================] - 3s 8ms/step - loss: 0.6936 - accuracy: 0.5100\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7e72ef386e00>"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Tuning Performance of Recurrent Neural Networks (RNNs)**\n",
        "\n",
        "\n",
        "To enhance the performance of RNNs, focus on overcoming the vanishing gradient problem and optimizing sequence data:\n",
        "\n",
        "1. Gradient Clipping:\n",
        "\n",
        "Gradient clipping limits the magnitude of gradients during backpropagation, preventing exploding gradients that can occur in RNNs.\n",
        "```\n",
        "from keras.optimizers import Adam\n",
        "\n",
        "optimizer = Adam(clipvalue=0.5)\n",
        "model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
        "```\n",
        "2. Bidirectional RNNs:\n",
        "\n",
        "Bidirectional RNNs process sequences in both forward and backward directions. This can capture contextual information from both past and future states.\n",
        "\n",
        "```\n",
        "from keras.layers import Bidirectional, SimpleRNN\n",
        "\n",
        "model.add(Bidirectional(SimpleRNN(64), input_shape=input_shape))\n",
        "```"
      ],
      "metadata": {
        "id": "AGvia3ZCUoqs"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "5. **Long Short-Term Memory (LSTM) Networks:**\n",
        "\n",
        "**Problem of Vanishing Gradient**:\n",
        "\n",
        "Standard RNNs suffer from the vanishing gradient problem, making it difficult to capture long-range dependencies in sequences. LSTMs address this issue.\n",
        "\n",
        "**LSTM Architecture:**\n",
        "\n",
        "LSTMs have a more complex structure than standard RNNs. They use gates (input, forget, output) to control the flow of information, making them capable of capturing long-term dependencies.\n",
        "\n",
        "\n",
        "Let's see how we can use Python and Keras to create an LSTM model:"
      ],
      "metadata": {
        "id": "Vu3VBWSvU4qZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from keras.layers import LSTM\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "import numpy as np\n",
        "\n",
        "# Create dummy sequential data\n",
        "# Assuming each sequence has 10 time steps and 2 features\n",
        "num_samples = 1000\n",
        "time_steps = 10\n",
        "num_features = 2\n",
        "\n",
        "X_train = np.random.random((num_samples, time_steps, num_features))\n",
        "y_train = np.random.randint(2, size=(num_samples, 1))\n",
        "\n",
        "# Splitting the data into training and validation sets\n",
        "split_ratio = 0.8\n",
        "split_index = int(num_samples * split_ratio)\n",
        "\n",
        "X_train, X_val = X_train[:split_index], X_train[split_index:]\n",
        "y_train, y_val = y_train[:split_index], y_train[split_index:]\n",
        "\n",
        "# Define input shape\n",
        "input_shape = (time_steps, num_features)\n",
        "\n",
        "# Create a Sequential model\n",
        "model = Sequential()\n",
        "\n",
        "# Add an LSTM layer\n",
        "model.add(LSTM(64, input_shape=input_shape))\n",
        "\n",
        "# Add a Dense output layer\n",
        "output_dim = 1  # Change this based on your problem\n",
        "model.add(Dense(output_dim, activation='sigmoid'))  # Use 'sigmoid' for binary classification\n",
        "\n",
        "# Compile and train the model\n",
        "model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "model.fit(X_train, y_train, epochs=10, batch_size=32)\n",
        "\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dimiEiQHVJKb",
        "outputId": "4a65f549-d8d4-4e09-c8f9-846e374a868d"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "25/25 [==============================] - 3s 9ms/step - loss: 0.6935 - accuracy: 0.5025\n",
            "Epoch 2/10\n",
            "25/25 [==============================] - 0s 9ms/step - loss: 0.6948 - accuracy: 0.4863\n",
            "Epoch 3/10\n",
            "25/25 [==============================] - 0s 9ms/step - loss: 0.6919 - accuracy: 0.5225\n",
            "Epoch 4/10\n",
            "25/25 [==============================] - 0s 10ms/step - loss: 0.6926 - accuracy: 0.5225\n",
            "Epoch 5/10\n",
            "25/25 [==============================] - 0s 10ms/step - loss: 0.6929 - accuracy: 0.5225\n",
            "Epoch 6/10\n",
            "25/25 [==============================] - 0s 8ms/step - loss: 0.6929 - accuracy: 0.5225\n",
            "Epoch 7/10\n",
            "25/25 [==============================] - 0s 7ms/step - loss: 0.6935 - accuracy: 0.5225\n",
            "Epoch 8/10\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.6924 - accuracy: 0.5225\n",
            "Epoch 9/10\n",
            "25/25 [==============================] - 0s 7ms/step - loss: 0.6925 - accuracy: 0.5225\n",
            "Epoch 10/10\n",
            "25/25 [==============================] - 0s 6ms/step - loss: 0.6937 - accuracy: 0.4787\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7e72f0171720>"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Tuning Performance of Long Short-Term Memory (LSTM) Networks:**\n",
        "\n",
        "For LSTM networks, focus on architectural enhancements and regularization:\n",
        "\n",
        "**1. Multiple LSTM Layers:**\n",
        "\n",
        "Stacking multiple LSTM layers can allow the network to learn more complex temporal dependencies.\n",
        "```\n",
        "model.add(LSTM(64, return_sequences=True, input_shape=input_shape))\n",
        "model.add(LSTM(64))\n",
        "```\n",
        "**2. Dropout and Recurrent Dropout:**\n",
        "\n",
        "Apply dropout not only to the input but also to the recurrent connections within LSTM cells.\n",
        "\n",
        "```\n",
        "model.add(LSTM(64, dropout=0.2, recurrent_dropout=0.2, input_shape=input_shape))\n",
        "```"
      ],
      "metadata": {
        "id": "ILBY5ANNV_zr"
      }
    }
  ]
}