Mod1_Wk4 Programming Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Programming Assignment\n",
    "\n",
    "## Saving and loading models, with application to the EuroSat dataset\n",
    "\n",
    "### Instructions\n",
    "\n",
    "In this notebook, you will create a neural network that classifies land uses and land covers from satellite imagery. You will save your model using Tensorflow's callbacks and reload it later. You will also load in a pre-trained neural network classifier and compare performance with it. \n",
    "\n",
    "Some code cells are provided for you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line: \n",
    "\n",
    "`#### GRADED CELL ####`\n",
    "\n",
    "Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly. Inside these graded cells, you can use any functions or classes that are imported below, but make sure you don't use any variables that are outside the scope of the function.\n",
    "\n",
    "### How to submit\n",
    "\n",
    "Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.\n",
    "\n",
    "### Let's get started!\n",
    "\n",
    "We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### PACKAGE IMPORTS ####\n",
    "\n",
    "# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook\n",
    "\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.image import load_img, img_to_array\n",
    "from tensorflow.keras.models import Sequential, load_model\n",
    "from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint, EarlyStopping\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# If you would like to make further imports from tensorflow, add them here\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![EuroSAT overview image](data/eurosat_overview.jpg)\n",
    "\n",
    "#### The EuroSAT dataset\n",
    "\n",
    "In this assignment, you will use the [EuroSAT dataset](https://github.com/phelber/EuroSAT). It consists of 27000 labelled Sentinel-2 satellite images of different land uses: residential, industrial, highway, river, forest, pasture, herbaceous vegetation, annual crop, permanent crop and sea/lake. For a reference, see the following papers:\n",
    "- Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Patrick Helber, Benjamin Bischke, Andreas Dengel, Damian Borth. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.\n",
    "- Introducing EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification. Patrick Helber, Benjamin Bischke, Andreas Dengel. 2018 IEEE International Geoscience and Remote Sensing Symposium, 2018.\n",
    "\n",
    "Your goal is to construct a neural network that classifies a satellite image into one of these 10 classes, as well as applying some of the saving and loading techniques you have learned in the previous sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import the data\n",
    "\n",
    "The dataset you will train your model on is a subset of the total data, with 4000 training images and 1000 testing images, with roughly equal numbers of each class. The code to import the data is provided below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to import the Eurosat data\n",
    "\n",
    "def load_eurosat_data():\n",
    "    data_dir = 'data/'\n",
    "    x_train = np.load(os.path.join(data_dir, 'x_train.npy'))\n",
    "    y_train = np.load(os.path.join(data_dir, 'y_train.npy'))\n",
    "    x_test  = np.load(os.path.join(data_dir, 'x_test.npy'))\n",
    "    y_test  = np.load(os.path.join(data_dir, 'y_test.npy'))\n",
    "    return (x_train, y_train), (x_test, y_test)\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = load_eurosat_data()\n",
    "x_train = x_train / 255.0\n",
    "x_test = x_test / 255.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build the neural network model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now construct a model to fit to the data. Using the Sequential API, build your model according to the following specifications:\n",
    "\n",
    "* The model should use the input_shape in the function argument to set the input size in the first layer.\n",
    "* The first layer should be a Conv2D layer with 16 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_1'.\n",
    "* The second layer should also be a Conv2D layer with 8 filters, a 3x3 kernel size, a ReLU activation function and 'SAME' padding. Name this layer 'conv_2'.\n",
    "* The third layer should be a MaxPooling2D layer with a pooling window size of 8x8. Name this layer 'pool_1'.\n",
    "* The fourth layer should be a Flatten layer, named 'flatten'.\n",
    "* The fifth layer should be a Dense layer with 32 units, a ReLU activation. Name this layer 'dense_1'.\n",
    "* The sixth and final layer should be a Dense layer with 10 units and softmax activation. Name this layer 'dense_2'.\n",
    "\n",
    "In total, the network should have 6 layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### GRADED CELL ####\n",
    "\n",
    "# Complete the following function. \n",
    "# Make sure to not change the function name or arguments.\n",
    "\n",
    "def get_new_model(input_shape):\n",
    "    \"\"\"\n",
    "    This function should build a Sequential model according to the above specification. Ensure the \n",
    "    weights are initialised by providing the input_shape argument in the first layer, given by the\n",
    "    function argument.\n",
    "    Your function should also compile the model with the Adam optimiser, sparse categorical cross\n",
    "    entropy loss function, and a single accuracy metric.\n",
    "    \"\"\"\n",
    "    model = Sequential([\n",
    "        Conv2D(filters=16,input_shape=(64,64,3),kernel_size=(3,3),activation='relu',padding='same',name='conv_1'),\n",
    "        Conv2D(filters=8,kernel_size=(3,3),activation='relu',padding='same',name='conv_2'),\n",
    "        MaxPooling2D(pool_size=(8,8),name='pool_1'),\n",
    "        Flatten(name='Flatten'),\n",
    "        Dense(units=32, activation='relu', name='dense_1'),\n",
    "        Dense(units=10, activation='relu', name='dense_2')\n",
    "    ])\n",
    "    model.compile(optimizer='adam',\n",
    "                  loss='sparse_categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compile and evaluate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run your function to create the model\n",
    "\n",
    "model = get_new_model(x_train[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to define a function to evaluate a model's test accuracy\n",
    "\n",
    "def get_test_accuracy(model, x_test, y_test):\n",
    "    \"\"\"Test model classification accuracy\"\"\"\n",
    "    test_loss, test_acc = model.evaluate(x=x_test, y=y_test, verbose=0)\n",
    "    print('accuracy: {acc:0.3f}'.format(acc=test_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv_1 (Conv2D)              (None, 64, 64, 16)        448       \n",
      "_________________________________________________________________\n",
      "conv_2 (Conv2D)              (None, 64, 64, 8)         1160      \n",
      "_________________________________________________________________\n",
      "pool_1 (MaxPooling2D)        (None, 8, 8, 8)           0         \n",
      "_________________________________________________________________\n",
      "Flatten (Flatten)            (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 32)                16416     \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 18,354\n",
      "Trainable params: 18,354\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "accuracy: 0.107\n"
     ]
    }
   ],
   "source": [
    "# Print the model summary and calculate its initialised test accuracy\n",
    "\n",
    "model.summary()\n",
    "get_test_accuracy(model, x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create checkpoints to save model during training, with a criterion\n",
    "\n",
    "You will now create three callbacks:\n",
    "- `checkpoint_every_epoch`: checkpoint that saves the model weights every epoch during training\n",
    "- `checkpoint_best_only`: checkpoint that saves only the weights with the highest validation accuracy. Use the testing data as the validation data.\n",
    "- `early_stopping`: early stopping object that ends training if the validation accuracy has not improved in 3 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### GRADED CELL ####\n",
    "\n",
    "# Complete the following functions. \n",
    "# Make sure to not change the function names or arguments.\n",
    "\n",
    "def get_checkpoint_every_epoch():\n",
    "    \"\"\"\n",
    "    This function should return a ModelCheckpoint object that:\n",
    "    - saves the weights only at the end of every epoch\n",
    "    - saves into a directory called 'checkpoints_every_epoch' inside the current working directory\n",
    "    - generates filenames in that directory like 'checkpoint_XXX' where\n",
    "      XXX is the epoch number formatted to have three digits, e.g. 001, 002, 003, etc.\n",
    "    \"\"\"\n",
    "    return ModelCheckpoint(\n",
    "        filepath = 'checkpoints_every_epoch/checkpoint_{epoch:03d}',\n",
    "        frequency='epoch',\n",
    "        save_weights_only=True, \n",
    "        verbose=1)\n",
    "    \n",
    "    \n",
    "\n",
    "\n",
    "def get_checkpoint_best_only():\n",
    "    \"\"\"\n",
    "    This function should return a ModelCheckpoint object that:\n",
    "    - saves only the weights that generate the highest validation (testing) accuracy\n",
    "    - saves into a directory called 'checkpoints_best_only' inside the current working directory\n",
    "    - generates a file called 'checkpoints_best_only/checkpoint' \n",
    "    \"\"\"\n",
    "    return ModelCheckpoint(\n",
    "        filepath = 'checkpoints_best_only/checkpoint',\n",
    "        frequency='epoch',\n",
    "        save_best_only=True,\n",
    "        monitor='val_accuracy',\n",
    "        save_weights_only=True, \n",
    "        verbose=1)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### GRADED CELL ####\n",
    "\n",
    "# Complete the following function. \n",
    "# Make sure to not change the function name or arguments.\n",
    "\n",
    "def get_early_stopping():\n",
    "    \"\"\"\n",
    "    This function should return an EarlyStopping callback that stops training when\n",
    "    the validation (testing) accuracy has not improved in the last 3 epochs.\n",
    "    HINT: use the EarlyStopping callback with the correct 'monitor' and 'patience'\n",
    "    \"\"\"\n",
    "    return EarlyStopping(patience=3, monitor='val_accuracy')\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell to create the callbacks\n",
    "\n",
    "checkpoint_every_epoch = get_checkpoint_every_epoch()\n",
    "checkpoint_best_only = get_checkpoint_best_only()\n",
    "early_stopping = get_early_stopping()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train model using the callbacks\n",
    "\n",
    "Now, you will train the model using the three callbacks you created. If you created the callbacks correctly, three things should happen:\n",
    "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_every_epoch`\n",
    "- At the end of every epoch, the model weights are saved into a directory called `checkpoints_best_only` **only** if those weights lead to the highest test accuracy\n",
    "- Training stops when the testing accuracy has not improved in three epochs.\n",
    "\n",
    "You should then have two directories:\n",
    "- A directory called `checkpoints_every_epoch` containing filenames that include `checkpoint_001`, `checkpoint_002`, etc with the `001`, `002` corresponding to the epoch\n",
    "- A directory called `checkpoints_best_only` containing filenames that include `checkpoint`, which contain only the weights leading to the highest testing accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 4000 samples, validate on 1000 samples\n",
      "Epoch 1/50\n",
      "3968/4000 [============================>.] - ETA: 0s - loss: 3.1754 - accuracy: 0.1094\n",
      "Epoch 00001: saving model to checkpoints_every_epoch/checkpoint_001\n",
      "\n",
      "Epoch 00001: val_accuracy improved from -inf to 0.09000, saving model to checkpoints_best_only/checkpoint\n",
      "4000/4000 [==============================] - 84s 21ms/sample - loss: 3.1684 - accuracy: 0.1093 - val_loss: 2.3026 - val_accuracy: 0.0900\n",
      "Epoch 2/50\n",
      "3968/4000 [============================>.] - ETA: 0s - loss: 2.3026 - accuracy: 0.1011\n",
      "Epoch 00002: saving model to checkpoints_every_epoch/checkpoint_002\n",
      "\n",
      "Epoch 00002: val_accuracy did not improve from 0.09000\n",
      "4000/4000 [==============================] - 84s 21ms/sample - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.0900\n",
      "Epoch 3/50\n",
      "3968/4000 [============================>.] - ETA: 0s - loss: 2.3026 - accuracy: 0.1016\n",
      "Epoch 00003: saving model to checkpoints_every_epoch/checkpoint_003\n",
      "\n",
      "Epoch 00003: val_accuracy did not improve from 0.09000\n",
      "4000/4000 [==============================] - 82s 21ms/sample - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.0900\n",
      "Epoch 4/50\n",
      "3968/4000 [============================>.] - ETA: 0s - loss: 2.3026 - accuracy: 0.1018\n",
      "Epoch 00004: saving model to checkpoints_every_epoch/checkpoint_004\n",
      "\n",
      "Epoch 00004: val_accuracy did not improve from 0.09000\n",
      "4000/4000 [==============================] - 80s 20ms/sample - loss: 2.3026 - accuracy: 0.1015 - val_loss: 2.3026 - val_accuracy: 0.0900\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fdd15c7c400>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Train model using the callbacks you just created\n",
    "\n",
    "callbacks = [checkpoint_every_epoch, checkpoint_best_only, early_stopping]\n",
    "model.fit(x_train, y_train, epochs=50, validation_data=(x_test, y_test), callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create new instance of model and load on both sets of weights\n",
    "\n",
    "Now you will use the weights you just saved in a fresh model. You should create two functions, both of which take a freshly instantiated model instance:\n",
    "- `model_last_epoch` should contain the weights from the latest saved epoch\n",
    "- `model_best_epoch` should contain the weights from the saved epoch with the highest testing accuracy\n",
    "\n",
    "_Hint: use the_ `tf.train.latest_checkpoint` _function to get the filename of the latest saved checkpoint file. Check the docs_ [_here_](https://www.tensorflow.org/api_docs/python/tf/train/latest_checkpoint)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### GRADED CELL ####\n",
    "\n",
    "# Complete the following functions. \n",
    "# Make sure to not change the function name or arguments.\n",
    "\n",
    "def get_model_last_epoch(model):\n",
    "    \"\"\"\n",
    "    This function should create a new instance of the CNN you created earlier,\n",
    "    load on the weights from the last training epoch, and return this model.\n",
    "    \"\"\"\n",
    "    model.load_weights(tf.train.latest_checkpoint(checkpoint_dir='checkpoints_every_epoch'))\n",
    "    return model\n",
    "    \n",
    "    \n",
    "def get_model_best_epoch(model):\n",
    "    \"\"\"\n",
    "    This function should create a new instance of the CNN you created earlier, load \n",
    "    on the weights leading to the highest validation accuracy, and return this model.\n",
    "    \"\"\"\n",
    "    model.load_weights('checkpoints_best_only/checkpoint')\n",
    "    return model\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model with last epoch weights:\n",
      "accuracy: 0.090\n",
      "\n",
      "Model with best epoch weights:\n",
      "accuracy: 0.090\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to create two models: one with the weights from the last training\n",
    "# epoch, and one with the weights leading to the highest validation (testing) accuracy.\n",
    "# Verify that the second has a higher validation (testing) accuarcy.\n",
    "\n",
    "model_last_epoch = get_model_last_epoch(get_new_model(x_train[0].shape))\n",
    "model_best_epoch = get_model_best_epoch(get_new_model(x_train[0].shape))\n",
    "print('Model with last epoch weights:')\n",
    "get_test_accuracy(model_last_epoch, x_test, y_test)\n",
    "print('')\n",
    "print('Model with best epoch weights:')\n",
    "get_test_accuracy(model_best_epoch, x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load, from scratch, a model trained on the EuroSat dataset.\n",
    "\n",
    "In your workspace, you will find another model trained on the `EuroSAT` dataset in `.h5` format. This model is trained on a larger subset of the EuroSAT dataset and has a more complex architecture. The path to the model is `models/EuroSatNet.h5`. See how its testing accuracy compares to your model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### GRADED CELL ####\n",
    "\n",
    "# Complete the following functions. \n",
    "# Make sure to not change the function name or arguments.\n",
    "\n",
    "def get_model_eurosatnet():\n",
    "    \"\"\"\n",
    "    This function should return the pretrained EuroSatNet.h5 model.\n",
    "    \"\"\"\n",
    "    model= load_model('models/EuroSatNet.h5')\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_21\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv_1 (Conv2D)              (None, 64, 64, 16)        448       \n",
      "_________________________________________________________________\n",
      "conv_2 (Conv2D)              (None, 64, 64, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_1 (MaxPooling2D)        (None, 32, 32, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv_3 (Conv2D)              (None, 32, 32, 16)        2320      \n",
      "_________________________________________________________________\n",
      "conv_4 (Conv2D)              (None, 32, 32, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_2 (MaxPooling2D)        (None, 16, 16, 16)        0         \n",
      "_________________________________________________________________\n",
      "conv_5 (Conv2D)              (None, 16, 16, 16)        2320      \n",
      "_________________________________________________________________\n",
      "conv_6 (Conv2D)              (None, 16, 16, 16)        6416      \n",
      "_________________________________________________________________\n",
      "pool_3 (MaxPooling2D)        (None, 8, 8, 16)          0         \n",
      "_________________________________________________________________\n",
      "conv_7 (Conv2D)              (None, 8, 8, 16)          2320      \n",
      "_________________________________________________________________\n",
      "conv_8 (Conv2D)              (None, 8, 8, 16)          6416      \n",
      "_________________________________________________________________\n",
      "pool_4 (MaxPooling2D)        (None, 4, 4, 16)          0         \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 256)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 32)                8224      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                330       \n",
      "=================================================================\n",
      "Total params: 41,626\n",
      "Trainable params: 41,626\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "accuracy: 0.810\n"
     ]
    }
   ],
   "source": [
    "# Run this cell to print a summary of the EuroSatNet model, along with its validation accuracy.\n",
    "\n",
    "model_eurosatnet = get_model_eurosatnet()\n",
    "model_eurosatnet.summary()\n",
    "get_test_accuracy(model_eurosatnet, x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations for completing this programming assignment! You're now ready to move on to the capstone project for this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "tensor-flow-2-1",
   "graded_item_id": "JaRY0",
   "launcher_item_id": "mJ8fg"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}