intro_to_neural_nets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "wDlWLbfkJtvu"
   },
   "outputs": [],
   "source": [
    "#@title Copyright 2020 Google LLC. Double-click here for license information.\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TL5y5fY9Jy_x"
   },
   "source": [
    "# Introduction to Neural Nets\n",
    "\n",
    "This Colab builds a deep neural network to perform more sophisticated linear regression than the earlier Colabs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7RDY3EeAluPd"
   },
   "source": [
    "## Learning Objectives:\n",
    "\n",
    "After doing this Colab, you'll know how to do the following:\n",
    "\n",
    "  * Create a simple deep neural network.\n",
    "  * Tune the hyperparameters for a simple deep neural network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XGj0PNaJlubZ"
   },
   "source": [
    "## The Dataset\n",
    "  \n",
    "Like several of the previous Colabs, this Colab uses the [California Housing Dataset](https://developers.google.com/machine-learning/crash-course/california-housing-data-description)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tX_umRMMsa3z"
   },
   "source": [
    "## Use the right version of TensorFlow\n",
    "\n",
    "The following hidden code cell ensures that the Colab will run on TensorFlow 2.X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "lM75uNH-sTv2"
   },
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "from __future__ imports must occur at the beginning of the file (<ipython-input-3-18e18b979c5f>, line 6)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-3-18e18b979c5f>\"\u001b[1;36m, line \u001b[1;32m6\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m from __future__ imports must occur at the beginning of the file\n"
     ]
    }
   ],
   "source": [
    "#@title Run on TensorFlow 2.x\n",
    "%tensorflow_version 2.x\n",
    "from __future__ import absolute_import, division, print_function, unicode_literals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xchnxAsaKKqO"
   },
   "source": [
    "## Import relevant modules\n",
    "\n",
    "The following hidden code cell imports the necessary code to run the code in the rest of this Colaboratory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "9n9_cTveKmse"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'tensorflow'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-56f707c154ab>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mnumpy\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mnp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mpandas\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mpd\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mtensorflow\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mtf\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkeras\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mlayers\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[0mmatplotlib\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mpyplot\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0mplt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'tensorflow'"
     ]
    }
   ],
   "source": [
    "#@title Import relevant modules\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras import layers\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# The following lines adjust the granularity of reporting. \n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = \"{:.1f}\".format\n",
    "\n",
    "print(\"Imported modules.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X_TaJhU4KcuY"
   },
   "source": [
    "## Load the dataset\n",
    "\n",
    "Like most of the previous Colab exercises, this exercise uses the California Housing Dataset.  The following code cell loads the separate .csv files and creates the following two pandas DataFrames:\n",
    "\n",
    "* `train_df`, which contains the training set\n",
    "* `test_df`, which contains the test set\n",
    "   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JZlvdpyYKx7V"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\")\n",
    "train_df = train_df.reindex(np.random.permutation(train_df.index)) # shuffle the examples\n",
    "test_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8ldP-5z1B2vL"
   },
   "source": [
    "## Normalize values\n",
    "\n",
    "When building a model with multiple features, the values of each feature should cover roughly the same range.  The following code cell normalizes datasets by converting each raw value to its Z-score. (For more information about Z-scores, see the Classification exercise.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "g8HC-TDgB1D1"
   },
   "outputs": [],
   "source": [
    "#@title Convert raw values to their Z-scores \n",
    "\n",
    "# Calculate the Z-scores of each column in the training set:\n",
    "train_df_mean = train_df.mean()\n",
    "train_df_std = train_df.std()\n",
    "train_df_norm = (train_df - train_df_mean)/train_df_std\n",
    "\n",
    "# Calculate the Z-scores of each column in the test set.\n",
    "test_df_mean = test_df.mean()\n",
    "test_df_std = test_df.std()\n",
    "test_df_norm = (test_df - test_df_mean)/test_df_std\n",
    "\n",
    "print(\"Normalized the values.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "b9ehCgIRjTxy"
   },
   "source": [
    "## Represent data\n",
    "\n",
    "The following code cell creates a feature layer containing three features:\n",
    "\n",
    "* `latitude` X `longitude` (a feature cross)\n",
    "* `median_income`\n",
    "* `population`\n",
    "\n",
    "This code cell specifies the features that you'll ultimately train the model on and how each of those features will be represented. The transformations (collected in `my_feature_layer`) don't actually get applied until you pass a DataFrame to it, which will happen when we train the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "8EkNAQhnjSu-"
   },
   "outputs": [],
   "source": [
    "# Create an empty list that will eventually hold all created feature columns.\n",
    "feature_columns = []\n",
    "\n",
    "# We scaled all the columns, including latitude and longitude, into their\n",
    "# Z scores. So, instead of picking a resolution in degrees, we're going\n",
    "# to use resolution_in_Zs.  A resolution_in_Zs of 1 corresponds to \n",
    "# a full standard deviation. \n",
    "resolution_in_Zs = 0.3  # 3/10 of a standard deviation.\n",
    "\n",
    "# Create a bucket feature column for latitude.\n",
    "latitude_as_a_numeric_column = tf.feature_column.numeric_column(\"latitude\")\n",
    "latitude_boundaries = list(np.arange(int(min(train_df_norm['latitude'])), \n",
    "                                     int(max(train_df_norm['latitude'])), \n",
    "                                     resolution_in_Zs))\n",
    "latitude = tf.feature_column.bucketized_column(latitude_as_a_numeric_column, latitude_boundaries)\n",
    "\n",
    "# Create a bucket feature column for longitude.\n",
    "longitude_as_a_numeric_column = tf.feature_column.numeric_column(\"longitude\")\n",
    "longitude_boundaries = list(np.arange(int(min(train_df_norm['longitude'])), \n",
    "                                      int(max(train_df_norm['longitude'])), \n",
    "                                      resolution_in_Zs))\n",
    "longitude = tf.feature_column.bucketized_column(longitude_as_a_numeric_column, \n",
    "                                                longitude_boundaries)\n",
    "\n",
    "# Create a feature cross of latitude and longitude.\n",
    "latitude_x_longitude = tf.feature_column.crossed_column([latitude, longitude], hash_bucket_size=100)\n",
    "crossed_feature = tf.feature_column.indicator_column(latitude_x_longitude)\n",
    "feature_columns.append(crossed_feature)  \n",
    "\n",
    "# Represent median_income as a floating-point value.\n",
    "median_income = tf.feature_column.numeric_column(\"median_income\")\n",
    "feature_columns.append(median_income)\n",
    "\n",
    "# Represent population as a floating-point value.\n",
    "population = tf.feature_column.numeric_column(\"population\")\n",
    "feature_columns.append(population)\n",
    "\n",
    "# Convert the list of feature columns into a layer that will later be fed into\n",
    "# the model. \n",
    "my_feature_layer = tf.keras.layers.DenseFeatures(feature_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ak_TMAzGOIFq"
   },
   "source": [
    "## Build a linear regression model as a baseline\n",
    "\n",
    "Before creating a deep neural net, find a [baseline](https://developers.google.com/machine-learning/glossary/#baseline) loss by running a simple linear regression model that uses the feature layer you just created. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "QF0BFRXTOeR3"
   },
   "outputs": [],
   "source": [
    "#@title Define the plotting function.\n",
    "\n",
    "def plot_the_loss_curve(epochs, mse):\n",
    "  \"\"\"Plot a curve of loss vs. epoch.\"\"\"\n",
    "\n",
    "  plt.figure()\n",
    "  plt.xlabel(\"Epoch\")\n",
    "  plt.ylabel(\"Mean Squared Error\")\n",
    "\n",
    "  plt.plot(epochs, mse, label=\"Loss\")\n",
    "  plt.legend()\n",
    "  plt.ylim([mse.min()*0.95, mse.max() * 1.03])\n",
    "  plt.show()  \n",
    "\n",
    "print(\"Defined the plot_the_loss_curve function.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "RW4Qe710LgnG"
   },
   "outputs": [],
   "source": [
    "#@title Define functions to create and train a linear regression model\n",
    "def create_model(my_learning_rate, feature_layer):\n",
    "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
    "  # Most simple tf.keras models are sequential.\n",
    "  model = tf.keras.models.Sequential()\n",
    "\n",
    "  # Add the layer containing the feature columns to the model.\n",
    "  model.add(feature_layer)\n",
    "\n",
    "  # Add one linear layer to the model to yield a simple linear regressor.\n",
    "  model.add(tf.keras.layers.Dense(units=1, input_shape=(1,)))\n",
    "\n",
    "  # Construct the layers into a model that TensorFlow can execute.\n",
    "  model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=my_learning_rate),\n",
    "                loss=\"mean_squared_error\",\n",
    "                metrics=[tf.keras.metrics.MeanSquaredError()])\n",
    "\n",
    "  return model           \n",
    "\n",
    "\n",
    "def train_model(model, dataset, epochs, batch_size, label_name):\n",
    "  \"\"\"Feed a dataset into the model in order to train it.\"\"\"\n",
    "\n",
    "  # Split the dataset into features and label.\n",
    "  features = {name:np.array(value) for name, value in dataset.items()}\n",
    "  label = np.array(features.pop(label_name))\n",
    "  history = model.fit(x=features, y=label, batch_size=batch_size,\n",
    "                      epochs=epochs, shuffle=True)\n",
    "\n",
    "  # Get details that will be useful for plotting the loss curve.\n",
    "  epochs = history.epoch\n",
    "  hist = pd.DataFrame(history.history)\n",
    "  rmse = hist[\"mean_squared_error\"]\n",
    "\n",
    "  return epochs, rmse   \n",
    "\n",
    "print(\"Defined the create_model and train_model functions.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "f47LmxF5X_pu"
   },
   "source": [
    "Run the following code cell to invoke the the functions defined in the preceding two code cells. (Ignore the warning messages.)\n",
    "\n",
    "**Note:** Because we've scaled all the input data, **including the label**, the resulting loss values will be *much less* than previous models. \n",
    "\n",
    "**Note:** Depending on the version of TensorFlow, running this cell might generate WARNING messages. Please ignore these warnings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "tsfE4ujDL4ju"
   },
   "outputs": [],
   "source": [
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.01\n",
    "epochs = 15\n",
    "batch_size = 1000\n",
    "label_name = \"median_house_value\"\n",
    "\n",
    "# Establish the model's topography.\n",
    "my_model = create_model(learning_rate, my_feature_layer)\n",
    "\n",
    "# Train the model on the normalized training set.\n",
    "epochs, mse = train_model(my_model, train_df_norm, epochs, batch_size, label_name)\n",
    "plot_the_loss_curve(epochs, mse)\n",
    "\n",
    "test_features = {name:np.array(value) for name, value in test_df_norm.items()}\n",
    "test_label = np.array(test_features.pop(label_name)) # isolate the label\n",
    "print(\"\\n Evaluate the linear regression model against the test set:\")\n",
    "my_model.evaluate(x = test_features, y = test_label, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3014ezH3C7jT"
   },
   "source": [
    "## Define a deep neural net model\n",
    "\n",
    "The `create_model` function defines the topography of the deep neural net, specifying the following:\n",
    "\n",
    "* The number of [layers](https://developers.google.com/machine-learning/glossary/#layer) in the deep neural net.\n",
    "* The number of [nodes](https://developers.google.com/machine-learning/glossary/#node) in each layer.\n",
    "\n",
    "The `create_model` function also defines the [activation function](https://developers.google.com/machine-learning/glossary/#activation_function) of each layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "pedD5GhlDC-y"
   },
   "outputs": [],
   "source": [
    "def create_model(my_learning_rate, my_feature_layer):\n",
    "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
    "  # Most simple tf.keras models are sequential.\n",
    "  model = tf.keras.models.Sequential()\n",
    "\n",
    "  # Add the layer containing the feature columns to the model.\n",
    "  model.add(my_feature_layer)\n",
    "\n",
    "  # Describe the topography of the model by calling the tf.keras.layers.Dense\n",
    "  # method once for each layer. We've specified the following arguments:\n",
    "  #   * units specifies the number of nodes in this layer.\n",
    "  #   * activation specifies the activation function (Rectified Linear Unit).\n",
    "  #   * name is just a string that can be useful when debugging.\n",
    "\n",
    "  # Define the first hidden layer with 20 nodes.   \n",
    "  model.add(tf.keras.layers.Dense(units=20, \n",
    "                                  activation='relu', \n",
    "                                  name='Hidden1'))\n",
    "  \n",
    "  # Define the second hidden layer with 12 nodes. \n",
    "  model.add(tf.keras.layers.Dense(units=12, \n",
    "                                  activation='relu', \n",
    "                                  name='Hidden2'))\n",
    "  \n",
    "  # Define the output layer.\n",
    "  model.add(tf.keras.layers.Dense(units=1,  \n",
    "                                  name='Output'))                              \n",
    "  \n",
    "  model.compile(optimizer=tf.keras.optimizers.Adam(lr=my_learning_rate),\n",
    "                loss=\"mean_squared_error\",\n",
    "                metrics=[tf.keras.metrics.MeanSquaredError()])\n",
    "\n",
    "  return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "anH4A_yCcZx2"
   },
   "source": [
    "## Define a training function\n",
    "\n",
    "The `train_model` function trains the model from the input features and labels. The [tf.keras.Model.fit](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential#fit) method performs the actual training. The `x` parameter of the `fit` method is very flexible, enabling you to pass feature data in a variety of ways. The following implementation passes a Python dictionary in which:\n",
    "\n",
    "* The *keys* are the names of each feature (for example, `longitude`, `latitude`, and so on).\n",
    "* The *value* of each key is a NumPy array containing the values of that feature. \n",
    "\n",
    "**Note:** Although you are passing *every* feature to `model.fit`, most of those values will be ignored. Only the features accessed by `my_feature_layer` will actually be used to train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4jv_lJYTcrEF"
   },
   "outputs": [],
   "source": [
    "def train_model(model, dataset, epochs, label_name,\n",
    "                batch_size=None):\n",
    "  \"\"\"Train the model by feeding it data.\"\"\"\n",
    "\n",
    "  # Split the dataset into features and label.\n",
    "  features = {name:np.array(value) for name, value in dataset.items()}\n",
    "  label = np.array(features.pop(label_name))\n",
    "  history = model.fit(x=features, y=label, batch_size=batch_size,\n",
    "                      epochs=epochs, shuffle=True) \n",
    "\n",
    "  # The list of epochs is stored separately from the rest of history.\n",
    "  epochs = history.epoch\n",
    "  \n",
    "  # To track the progression of training, gather a snapshot\n",
    "  # of the model's mean squared error at each epoch. \n",
    "  hist = pd.DataFrame(history.history)\n",
    "  mse = hist[\"mean_squared_error\"]\n",
    "\n",
    "  return epochs, mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D-IXYVfvM4gD"
   },
   "source": [
    "## Call the functions to build and train a deep neural net\n",
    "\n",
    "Okay, it is time to actually train the deep neural net.  If time permits, experiment with the three hyperparameters to see if you can reduce the loss\n",
    "against the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "both",
    "colab": {},
    "colab_type": "code",
    "id": "nj3v5EKQFY8s"
   },
   "outputs": [],
   "source": [
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.01\n",
    "epochs = 20\n",
    "batch_size = 1000\n",
    "\n",
    "# Specify the label\n",
    "label_name = \"median_house_value\"\n",
    "\n",
    "# Establish the model's topography.\n",
    "my_model = create_model(learning_rate, my_feature_layer)\n",
    "\n",
    "# Train the model on the normalized training set. We're passing the entire\n",
    "# normalized training set, but the model will only use the features\n",
    "# defined by the feature_layer.\n",
    "epochs, mse = train_model(my_model, train_df_norm, epochs, \n",
    "                          label_name, batch_size)\n",
    "plot_the_loss_curve(epochs, mse)\n",
    "\n",
    "# After building a model against the training set, test that model\n",
    "# against the test set.\n",
    "test_features = {name:np.array(value) for name, value in test_df_norm.items()}\n",
    "test_label = np.array(test_features.pop(label_name)) # isolate the label\n",
    "print(\"\\n Evaluate the new model against the test set:\")\n",
    "my_model.evaluate(x = test_features, y = test_label, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wlPXK-SmmjQ2"
   },
   "source": [
    "## Task 1: Compare the two models\n",
    "\n",
    "How did the deep neural net perform against the baseline linear regression model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "hI7ojsL7nnBE"
   },
   "outputs": [],
   "source": [
    "#@title Double-click to view a possible answer\n",
    "\n",
    "# Assuming that the linear model converged and\n",
    "# the deep neural net model also converged, please \n",
    "# compare the test set loss for each.\n",
    "# In our experiments, the loss of the deep neural \n",
    "# network model was consistently lower than \n",
    "# that of the linear regression model, which \n",
    "# suggests that the deep neural network model \n",
    "# will make better predictions than the \n",
    "# linear regression model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y5IKmk7D49_n"
   },
   "source": [
    "## Task 2: Optimize the deep neural network's topography\n",
    "\n",
    "Experiment with the number of layers of the deep neural network and the number of nodes in each layer.  Aim to achieve both of the following goals:\n",
    "\n",
    "*  Lower the loss against the test set.\n",
    "*  Minimize the overall number of nodes in the deep neural net. \n",
    "\n",
    "The two goals may be in conflict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "wYG5qXpP5a9n"
   },
   "outputs": [],
   "source": [
    "#@title Double-click to view a possible answer\n",
    "\n",
    "# Many answers are possible.  We noticed the \n",
    "# following trends:\n",
    "#   * Two layers outperformed one layer, but \n",
    "#     three layers did not perform significantly \n",
    "#     better than two layers; two layers \n",
    "#     outperformed one layer.\n",
    "#     In other words, two layers seemed best. \n",
    "#   * Setting the topography as follows produced \n",
    "#     reasonably good results with relatively few \n",
    "#     nodes:\n",
    "#       * 10 nodes in the first layer.\n",
    "#       *  6 nodes in the second layer.\n",
    "#     As the number of nodes in each layer dropped\n",
    "#     below the preceding, test loss increased.  \n",
    "#     However, depending on your application, hardware\n",
    "#     constraints, and the relative pain inflicted \n",
    "#     by a less accurate model, a smaller network \n",
    "#     (for example, 6 nodes in the first layer and \n",
    "#     4 nodes in the second layer) might be \n",
    "#     acceptable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Pu7R_ZpDopIj"
   },
   "source": [
    "## Task 3: Regularize the deep neural network (if you have enough time)\n",
    "\n",
    "Notice that the model's loss against the test set is *much higher* than the loss against the training set.  In other words, the deep neural network is [overfitting](https://developers.google.com/machine-learning/glossary/#overfitting) to the data in the training set.  To reduce overfitting, regularize the model.  The course has suggested several different ways to regularize a model, including:\n",
    "\n",
    "  * [L1 regularization](https://developers.google.com/machine-learning/glossary/#L1_regularization)\n",
    "  * [L2 regularization](https://developers.google.com/machine-learning/glossary/#L2_regularization)\n",
    "  * [Dropout regularization](https://developers.google.com/machine-learning/glossary/#dropout_regularization)\n",
    "\n",
    "Your task is to experiment with one or more regularization mechanisms to bring the test loss closer to the training loss (while still keeping test loss relatively low).  \n",
    "\n",
    "**Note:** When you add a regularization function to a model, you might need to tweak other hyperparameters. \n",
    "\n",
    "### Implementing L1 or L2 regularization\n",
    "\n",
    "To use L1 or L2 regularization on a hidden layer, specify the `kernel_regularizer` argument to [tf.keras.layers.Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense). Assign one of the following methods to this argument:\n",
    "\n",
    "* `tf.keras.regularizers.l1` for L1 regularization\n",
    "* `tf.keras.regularizers.l2` for L2 regularization\n",
    "\n",
    "Each of the preceding methods takes an `l` parameter, which adjusts the [regularization rate](https://developers.google.com/machine-learning/glossary/#regularization_rate). Assign a decimal value between 0 and 1.0 to `l`; the higher the decimal, the greater the regularization. For example, the following applies L2 regularization at a strength of 0.05. \n",
    "\n",
    "```\n",
    "model.add(tf.keras.layers.Dense(units=20, \n",
    "                                activation='relu',\n",
    "                                kernel_regularizer=tf.keras.regularizers.l2(l=0.01),\n",
    "                                name='Hidden1'))\n",
    "```\n",
    "\n",
    "### Implementing Dropout regularization\n",
    "\n",
    "You implement dropout regularization as a separate layer in the topography. For example, the following code demonstrates how to add a dropout regularization layer between the first hidden layer and the second hidden layer:\n",
    "\n",
    "```\n",
    "model.add(tf.keras.layers.Dense( *define first hidden layer*)\n",
    " \n",
    "model.add(tf.keras.layers.Dropout(rate=0.25))\n",
    "\n",
    "model.add(tf.keras.layers.Dense( *define second hidden layer*)\n",
    "```\n",
    "\n",
    "The `rate` parameter to [tf.keras.layers.Dropout](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dropout) specifies the fraction of nodes that the model should drop out during training. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "tflt9TZEDARW"
   },
   "outputs": [],
   "source": [
    "#@title Double-click for a possible solution\n",
    "\n",
    "# The following \"solution\" uses L2 regularization to bring training loss\n",
    "# and test loss closer to each other. Many, many other solutions are possible.\n",
    "\n",
    "\n",
    "def create_model(my_learning_rate, my_feature_layer):\n",
    "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
    "\n",
    "  # Discard any pre-existing version of the model.\n",
    "  model = None\n",
    "\n",
    "  # Most simple tf.keras models are sequential.\n",
    "  model = tf.keras.models.Sequential()\n",
    "\n",
    "  # Add the layer containing the feature columns to the model.\n",
    "  model.add(my_feature_layer)\n",
    "\n",
    "  # Describe the topography of the model. \n",
    "\n",
    "  # Implement L2 regularization in the first hidden layer.\n",
    "  model.add(tf.keras.layers.Dense(units=20, \n",
    "                                  activation='relu',\n",
    "                                  kernel_regularizer=tf.keras.regularizers.l2(0.04),\n",
    "                                  name='Hidden1'))\n",
    "  \n",
    "  # Implement L2 regularization in the second hidden layer.\n",
    "  model.add(tf.keras.layers.Dense(units=12, \n",
    "                                  activation='relu', \n",
    "                                  kernel_regularizer=tf.keras.regularizers.l2(0.04),\n",
    "                                  name='Hidden2'))\n",
    "\n",
    "  # Define the output layer.\n",
    "  model.add(tf.keras.layers.Dense(units=1,  \n",
    "                                  name='Output'))                              \n",
    "  \n",
    "  model.compile(optimizer=tf.keras.optimizers.Adam(lr=my_learning_rate),\n",
    "                loss=\"mean_squared_error\",\n",
    "                metrics=[tf.keras.metrics.MeanSquaredError()])\n",
    "\n",
    "  return model     \n",
    "\n",
    "# Call the new create_model function and the other (unchanged) functions.\n",
    "\n",
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.007\n",
    "epochs = 140\n",
    "batch_size = 1000\n",
    "\n",
    "label_name = \"median_house_value\"\n",
    "\n",
    "# Establish the model's topography.\n",
    "my_model = create_model(learning_rate, my_feature_layer)\n",
    "\n",
    "# Train the model on the normalized training set.\n",
    "epochs, mse = train_model(my_model, train_df_norm, epochs, \n",
    "                          label_name, batch_size)\n",
    "plot_the_loss_curve(epochs, mse)\n",
    "\n",
    "test_features = {name:np.array(value) for name, value in test_df_norm.items()}\n",
    "test_label = np.array(test_features.pop(label_name)) # isolate the label\n",
    "print(\"\\n Evaluate the new model against the test set:\")\n",
    "my_model.evaluate(x = test_features, y = test_label, batch_size=batch_size) "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Intro to Neural Nets.ipynb",
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}