single_digits_with_MNIST.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "single_digits with MNIST.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "wDlWLbfkJtvu",
        "cellView": "form"
      },
      "source": [
        "#@title Everything not mine is copyright 2020 Google LLC. Double-click here for full information.\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License.\n",
        "\n",
        "# Yann LeCun and Corinna Cortes hold the copyright of MNIST dataset,\n",
        "# which is a derivative work from original NIST datasets. \n",
        "# MNIST dataset is made available under the terms of the \n",
        "# Creative Commons Attribution-Share Alike 3.0 license."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9SBux8qWSzPz"
      },
      "source": [
        "This Notebook is heavily modified from the MLCC programming project with single-digit images here:\n",
        "https://colab.research.google.com/github/google/eng-edu/blob/master/ml/cc/exercises/multi-class_classification_with_MNIST.ipynb"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cxj8yVh4mFl5"
      },
      "source": [
        "**[Taken from the original MLCC \"project\" with single digit images. I will start with these images and then merge them together, pixel-array by pixel-array, to create a new train/test set of 100 double-digit images.]**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9n9_cTveKmse",
        "cellView": "both",
        "outputId": "03245fcb-1c9b-442e-9e90-5eb38ddd6303",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# load some standard utilities.\n",
        "%tensorflow_version 2.x\n",
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "import random as rd\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "# The following lines adjust the granularity of reporting. \n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = \"{:.3f}\".format\n",
        "\n",
        "# The following line improves formatting when ouputting NumPy arrays.\n",
        "np.set_printoptions(linewidth = 200)\n",
        "\n",
        "print(\"Loaded modules.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Loaded modules.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JZlvdpyYKx7V",
        "cellView": "both",
        "outputId": "ba0056d5-ebae-461f-c28c-c535238c9798",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# load the dataset.\n",
        "(x_train, y_train),(x_test, y_test) = tf.keras.datasets.mnist.load_data()\n",
        "print(\"Loaded the train and test sets.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 0s 0us/step\n",
            "Loaded the train and test sets.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "krymMuUyB1e-"
      },
      "source": [
        "train_6000 = np.array(xtn).reshape(28*56)\n",
        "valid_2000 = np.array(xts).reshape(28*56)\n",
        "train_6000[\"target\"] = np.array(ytn) \n",
        "valid_4000[\"target\"] = validation_targets\n",
        "train_6000.to_csv(\"doubledigits_train_6000\")\n",
        "valid_2000.to_csv(\"doubledigits_valid_2000\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3014ezH3C7jT"
      },
      "source": [
        "## Create a deep neural net model\n",
        "\n",
        "[Text from the original project for single digits.]\n",
        "\n",
        "The `create_model` function defines the topography of the deep neural net, specifying the following:\n",
        "\n",
        "* The number of [layers](https://developers.google.com/machine-learning/glossary/#layer) in the deep neural net.\n",
        "* The number of [nodes](https://developers.google.com/machine-learning/glossary/#node) in each layer.\n",
        "* Any [regularization](https://developers.google.com/machine-learning/glossary/#regularization) layers.\n",
        "\n",
        "The `create_model` function also defines the [activation function](https://developers.google.com/machine-learning/glossary/#activation_function) of each layer.  The activation function of the output layer is [softmax](https://developers.google.com/machine-learning/glossary/#softmax), which will yield (100) different outputs for each example. Each of the (100) outputs provides the probability that the input example is a certain digit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M5AI0Drej0KS"
      },
      "source": [
        "# SET UP A DEEP NEURAL NET TO LEARN HANDRITTEN NUMBERS FROM 0 TO 99 #\n",
        "# THIS CODE HERE IS ESSENTIALLY \"THE ALGORITHM\". \n",
        "# THERE'S NOT MUCH TO IT, ONCE EVERYTHING ELSE IS IN PLACE FOR IT!\n",
        "\n",
        "def create_model(learning_rate):\n",
        "    \"\"\"Create and compile a deep neural net.\"\"\"  \n",
        "    # Define the kind of model to use.\n",
        "    model = tf.keras.models.Sequential()\n",
        ".\n",
        "    model.add(tf.keras.layers.Flatten(input_shape=(28, 56)))\n",
        "\n",
        "    model.add(tf.keras.layers.Dense(units=256, activation='relu'))\n",
        " \n",
        "    model.add(tf.keras.layers.Dropout(rate=0.3))\n",
        "\n",
        " \n",
        "    model.add(tf.keras.layers.Dense(units=128, activation='relu'))\n",
        "    model.add(tf.keras.layers.Dropout(rate=0.2))\n",
        "\n",
        "    model.add(tf.keras.layers.Dense(units=100, activation='softmax'))     \n",
        "\n",
        "    model.compile(optimizer=tf.keras.optimizers.Adam(lr=learning_rate),\n",
        "                    loss=\"sparse_categorical_crossentropy\",\n",
        "                    metrics=['accuracy']) \n",
        "    return model \n",
        "\n",
        "# THIS FUNCTION WILL TRAIN THE MODEL ON THE TRAINING SET #\n",
        "def train_model(model, train_features, train_label, epochs,\n",
        "                batch_size=None, validation_split=0.1):\n",
        "\n",
        "    history = model.fit(x=train_features, y=train_label, batch_size=batch_size,\n",
        "                      epochs=epochs, shuffle=True, \n",
        "                      validation_split=validation_split)\n",
        " \n",
        "    # Get a snapshot of the model's metrics after each round of training\n",
        "    # to measure its progress.\n",
        "    epochs = history.epoch\n",
        "    hist = pd.DataFrame(history.history)\n",
        "    return epochs, hist    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D-IXYVfvM4gD"
      },
      "source": [
        "## Invoke the previous functions to train on the train set and evaluate on the test set.\n",
        "\n",
        "Run the following code cell to invoke the preceding functions and actually train the model on the training set. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nj3v5EKQFY8s",
        "cellView": "both"
      },
      "source": [
        "# These variables are the adjustable \"hyperparameters\" of the model.\n",
        "learning_rate = 0.001\n",
        "epochs = 20\n",
        "batch_size = 500\n",
        "validation_split = 0.1\n",
        "\n",
        "# CREATE A NEW NEURAL NETWORK ACCORDING TO SPECIFICATIONS.\n",
        "my_model = create_model(learning_rate)\n",
        "\n",
        "# TRAIN IT ON THE \"NORMALIZED\" TRAINING SET:\n",
        "epochs, hist = train_model(my_model, x_train_normalized, y_train_norm, \n",
        "                           epochs, batch_size, validation_split)\n",
        "\n",
        "# EVALUATE AGAINST THE TEST SET:\n",
        "print(\"\\n Evaluate the new model against the test set:\")\n",
        "my_model.evaluate(x=x_test_normalized, y=y_test_norm, batch_size=batch_size)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gOwFlKU0yjfh"
      },
      "source": [
        "x_train_normalized.shape\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "N6QCoQqRvRfb"
      },
      "source": [
        "# AROUND 90% ACCURACY IS NOT BAD!\n",
        "# Plot a graph of the 'accuracy' metric vs. epochs:\n",
        "plot_curve(epochs, hist, 'accuracy')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CKFOumI6vg4T"
      },
      "source": [
        "# THIS FUNCTION TRANSLATES THE OUTPUT FROM THE NEURAL NETWORK\n",
        "# INTO A \"BEST GUESS\" FOR EACH EXAMPLE BASED ON THE RELATIVE\n",
        "# PROBABILITY IT ESTIMATES FOR EACH NUMBER 0 TO 99.\n",
        "\n",
        "def getAnswers(how_many=10):  \n",
        "    answers = pd.DataFrame(columns=['Answer','Guess','P(A)','P(G)'])  \n",
        "    answers['Answer'] = y_test_norm[:how_many]\n",
        "    predicts = my_model.predict(x_test_normalized).round(5)\n",
        "    for j in range(0,how_many): # how_many is the number of examples to guess\n",
        "        probs = predicts[j] # one row of 100 probabilities \n",
        "        maxr = max(probs)   # top probability\n",
        "        for i in range(0,100):\n",
        "            if probs[i] == maxr:\n",
        "                answers.at[j,'Guess'] = i\n",
        "                answers.at[j,'P(G)'] = maxr*100\n",
        "            if i == answers['Answer'][j]:\n",
        "                answers.at[j,'P(A)'] = probs[i]*100\n",
        "    return answers\n",
        "print(\"Loaded function getAnswers.\")\n",
        "print(\"Getting answers...\" )\n",
        "\n",
        "# LOAD UP ALL THE GUESSES (W/ PROBABILITES) FOR \n",
        "# EACH EXAMPLE IMAGE IN THE NORMALIZED TEST SET\n",
        "answers = getAnswers(len(x_test_normalized))\n",
        "answers"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mkl_yxX8GlEt"
      },
      "source": [
        "# FIND THE *WRONG* GUESSES!\n",
        "wrongs = pd.DataFrame(index=answers.index,columns=['Mask'])\n",
        "for row in answers.index:\n",
        "    if answers['Answer'][row] == answers['Guess'][row]:\n",
        "        wrongs['Mask'][row] = False\n",
        "    else:\n",
        "        wrongs['Mask'][row] = True\n",
        "\n",
        "# LOCATE AND PRINT OUT THE LIST OF WRONGS       \n",
        "a = answers.loc[wrongs['Mask']]\n",
        "w = len(a) \n",
        "t = len(x_test_normalized)\n",
        "print(w,\"wrong out of\",t,\"guesses.\")\n",
        "a"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BZ4riZ5VmQbV"
      },
      "source": [
        "# FIND OUT WHICH NUMBERS IT HAD THE MOST TROUBLE WITH\n",
        "ww = []\n",
        "for i in range(0,100):\n",
        "    w = len((answers.loc[lambda a: a['Answer'] == i]).loc[wrongs['Mask']])\n",
        "    ww += [w]\n",
        "attempts = pd.DataFrame(ww)\n",
        "attempts.sort_values(0).tail(5)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bffPxjYDlEYJ"
      },
      "source": [
        "# LAST BUT NOT LEAST...\n",
        "# THIS CELL SHOWS WHAT SOME OF THE *WRONG* GUESSES WERE.\n",
        "#\n",
        "# IT SHOWS THE IMAGE THE ALGORITHM HAD TO WORK WITH\n",
        "# ALONG WITH ITS \"BEST GUESS\" AND THE LABELED ANSWER\n",
        "\n",
        "# RUN THE CELL A FEW TIMES TO SEE A FEW EXAMPLES OF WHERE IT FAILED!\n",
        "\n",
        "g = answers['Guess'].loc[wrongs['Mask']]\n",
        "a = answers['Answer'].loc[wrongs['Mask']]\n",
        "x = rd.choice(g.index)  #RANDOMLY SELECT ONE OF THE WRONG GUESSES\n",
        "pa = round(answers['P(A)'][x],2)\n",
        "pg = round(answers['P(G)'][x],2)\n",
        "print(\"Answer[\",x,\"] is\",a[x],\"with estimated probability\",pa,\"\\n\")\n",
        "print(\"Guess[\",x,\"] was\",g[x],\"with estimated probability\",pg,\"\\n\")\n",
        "plt.imshow(x_test_normalized[x])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9xo6ne4dOPIl"
      },
      "source": [
        "And that's that, for now!\n",
        "\n",
        "Next step: teach an algorithm to add and subtract just by giving it handwritten digits and handwritten \"+\",\"-\",\"=\" signs!!"
      ]
    }
  ]
}