linear_regression_with_a_real_dataset.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Linear Regression with a Real Dataset.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "metadata": {
        "id": "wDlWLbfkJtvu",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Copyright 2020 Google LLC. Double-click here for license information.\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TL5y5fY9Jy_x",
        "colab_type": "text"
      },
      "source": [
        "# Linear Regression with a Real Dataset\n",
        "\n",
        "This Colab uses a real dataset to predict the prices of houses in California.   \n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8wtceyJj2uX",
        "colab_type": "text"
      },
      "source": [
        "## Learning Objectives:\n",
        "\n",
        "After doing this Colab, you'll know how to do the following:\n",
        "\n",
        "  * Read a .csv file into a [pandas](https://developers.google.com/machine-learning/glossary/#pandas) DataFrame.\n",
        "  * Examine a [dataset](https://developers.google.com/machine-learning/glossary/#data_set). \n",
        "  * Experiment with different [features](https://developers.google.com/machine-learning/glossary/#feature) in building a model.\n",
        "  * Tune the model's [hyperparameters](https://developers.google.com/machine-learning/glossary/#hyperparameter)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JJZEgJQSjyK4",
        "colab_type": "text"
      },
      "source": [
        "## The Dataset\n",
        "  \n",
        "The [dataset for this exercise](https://developers.google.com/machine-learning/crash-course/california-housing-data-description) is based on 1990 census data from California. The dataset is old but still provides a great opportunity to learn about machine learning programming."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tX_umRMMsa3z",
        "colab_type": "text"
      },
      "source": [
        "## Use the right version of TensorFlow\n",
        "\n",
        "The following hidden code cell ensures that the Colab will run on TensorFlow 2.X."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lM75uNH-sTv2",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Run on TensorFlow 2.x\n",
        "%tensorflow_version 2.x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xchnxAsaKKqO",
        "colab_type": "text"
      },
      "source": [
        "## Import relevant modules\n",
        "\n",
        "The following hidden code cell imports the necessary code to run the code in the rest of this Colaboratory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9n9_cTveKmse",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Import relevant modules\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "from matplotlib import pyplot as plt\n",
        "\n",
        "# The following lines adjust the granularity of reporting. \n",
        "pd.options.display.max_rows = 10\n",
        "pd.options.display.float_format = \"{:.1f}\".format"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X_TaJhU4KcuY",
        "colab_type": "text"
      },
      "source": [
        "## The dataset\n",
        "\n",
        "Datasets are often stored on disk or at a URL in [.csv format](https://wikipedia.org/wiki/Comma-separated_values). \n",
        "\n",
        "A well-formed .csv file contains column names in the first row, followed by many rows of data.  A comma divides each value in each row. For example, here are the first five rows of the .csv file file holding the California Housing Dataset:\n",
        "\n",
        "```\n",
        "\"longitude\",\"latitude\",\"housing_median_age\",\"total_rooms\",\"total_bedrooms\",\"population\",\"households\",\"median_income\",\"median_house_value\"\n",
        "-114.310000,34.190000,15.000000,5612.000000,1283.000000,1015.000000,472.000000,1.493600,66900.000000\n",
        "-114.470000,34.400000,19.000000,7650.000000,1901.000000,1129.000000,463.000000,1.820000,80100.000000\n",
        "-114.560000,33.690000,17.000000,720.000000,174.000000,333.000000,117.000000,1.650900,85700.000000\n",
        "-114.570000,33.640000,14.000000,1501.000000,337.000000,515.000000,226.000000,3.191700,73400.000000\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sSFQkzNlj-l6",
        "colab_type": "text"
      },
      "source": [
        "### Load the .csv file into a pandas DataFrame\n",
        "\n",
        "This Colab, like many machine learning programs, gathers the .csv file and stores the data in memory as a pandas Dataframe.  pandas is an open source Python library.  The primary datatype in pandas is a DataFrame.  You can imagine a pandas DataFrame as a spreadsheet in which each row is identified by a number and each column by a name. pandas is itself built on another open source Python library called NumPy. If you aren't familiar with these technologies, please view these two quick tutorials:\n",
        "\n",
        "*   [NumPy](https://colab.research.google.com/github/google/eng-edu/blob/master/ml/cc/exercises/numpy_ultraquick_tutorial.ipynb?utm_source=linearregressionreal-colab&utm_medium=colab&utm_campaign=colab-external&utm_content=numpy_tf2-colab&hl=en)\n",
        "*   [Pandas DataFrames](https://colab.research.google.com/github/google/eng-edu/blob/master/ml/cc/exercises/pandas_dataframe_ultraquick_tutorial.ipynb?utm_source=linearregressionreal-colab&utm_medium=colab&utm_campaign=colab-external&utm_content=pandas_tf2-colab&hl=en)\n",
        "\n",
        "The following code cell imports the .csv file into a pandas DataFrame and scales the values in the label (`median_house_value`):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JZlvdpyYKx7V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import the dataset.\n",
        "training_df = pd.read_csv(filepath_or_buffer=\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\")\n",
        "\n",
        "# Scale the label.\n",
        "training_df[\"median_house_value\"] /= 1000.0\n",
        "\n",
        "# Print the first rows of the pandas DataFrame.\n",
        "training_df.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5inxx49n4U9u",
        "colab_type": "text"
      },
      "source": [
        "Scaling `median_house_value` puts the value of each house in units of thousands. Scaling will keep loss values and learning rates in a friendlier range.  \n",
        "\n",
        "Although scaling a label is usually *not* essential, scaling features in a multi-feature model usually *is* essential."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMysi6-3IAbu",
        "colab_type": "text"
      },
      "source": [
        "## Examine the dataset\n",
        "\n",
        "A large part of most machine learning projects is getting to know your data. The pandas API provides a `describe` function that outputs the following statistics about every column in the DataFrame:\n",
        "\n",
        "* `count`, which is the number of rows in that column. Ideally, `count` contains the same value for every column. \n",
        "\n",
        "* `mean` and `std`, which contain the mean and standard deviation of the values in each column. \n",
        "\n",
        "* `min` and `max`, which contain the lowest and highest values in each column.\n",
        "\n",
        "* `25%`, `50%`, `75%`, which contain various [quantiles](https://developers.google.com/machine-learning/glossary/#quantile)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rnUSYKw4LUuh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get statistics on the dataset.\n",
        "training_df.describe()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9pcW_Yjtoo8",
        "colab_type": "text"
      },
      "source": [
        "### Task 1: Identify anomalies in the dataset\n",
        "\n",
        "Do you see any anomalies (strange values) in the data? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UoS7NWRXEs1H",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click to view a possible answer.\n",
        "\n",
        "# The maximum value (max) of several columns seems very\n",
        "# high compared to the other quantiles. For example,\n",
        "# example the total_rooms column. Given the quantile\n",
        "# values (25%, 50%, and 75%), you might expect the \n",
        "# max value of total_rooms to be approximately \n",
        "# 5,000 or possibly 10,000. However, the max value \n",
        "# is actually 37,937.\n",
        "\n",
        "# When you see anomalies in a column, become more careful\n",
        "# about using that column as a feature. That said,\n",
        "# anomalies in potential features sometimes mirror \n",
        "# anomalies in the label, which could make the column \n",
        "# be (or seem to be) a powerful feature.\n",
        "# Also, as you will see later in the course, you \n",
        "# might be able to represent (pre-process) raw data \n",
        "# in order to make columns into useful features."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3014ezH3C7jT",
        "colab_type": "text"
      },
      "source": [
        "## Define functions that build and train a model\n",
        "\n",
        "The following code defines two functions:\n",
        "\n",
        "  * `build_model(my_learning_rate)`, which builds a randomly-initialized model.\n",
        "  * `train_model(model, feature, label, epochs)`, which trains the model from the examples (feature and label) you pass. \n",
        "\n",
        "Since you don't need to understand model building code right now, we've hidden this code cell.  You may optionally double-click the following headline to see the code that builds and trains a model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pedD5GhlDC-y",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Define the functions that build and train a model\n",
        "def build_model(my_learning_rate):\n",
        "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
        "  # Most simple tf.keras models are sequential.\n",
        "  model = tf.keras.models.Sequential()\n",
        "\n",
        "  # Describe the topography of the model.\n",
        "  # The topography of a simple linear regression model\n",
        "  # is a single node in a single layer.\n",
        "  model.add(tf.keras.layers.Dense(units=1, \n",
        "                                  input_shape=(1,)))\n",
        "\n",
        "  # Compile the model topography into code that TensorFlow can efficiently\n",
        "  # execute. Configure training to minimize the model's mean squared error. \n",
        "  model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=my_learning_rate),\n",
        "                loss=\"mean_squared_error\",\n",
        "                metrics=[tf.keras.metrics.RootMeanSquaredError()])\n",
        "\n",
        "  return model        \n",
        "\n",
        "\n",
        "def train_model(model, df, feature, label, epochs, batch_size):\n",
        "  \"\"\"Train the model by feeding it data.\"\"\"\n",
        "\n",
        "  # Feed the model the feature and the label.\n",
        "  # The model will train for the specified number of epochs. \n",
        "  history = model.fit(x=df[feature],\n",
        "                      y=df[label],\n",
        "                      batch_size=None,\n",
        "                      epochs=epochs)\n",
        "\n",
        "  # Gather the trained model's weight and bias.\n",
        "  trained_weight = model.get_weights()[0]\n",
        "  trained_bias = model.get_weights()[1]\n",
        "\n",
        "  # The list of epochs is stored separately from the rest of history.\n",
        "  epochs = history.epoch\n",
        "  \n",
        "  # Isolate the error for each epoch.\n",
        "  hist = pd.DataFrame(history.history)\n",
        "\n",
        "  # To track the progression of training, we're going to take a snapshot\n",
        "  # of the model's root mean squared error at each epoch. \n",
        "  rmse = hist[\"root_mean_squared_error\"]\n",
        "\n",
        "  return trained_weight, trained_bias, epochs, rmse\n",
        "\n",
        "print(\"Defined the create_model and traing_model functions.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ak_TMAzGOIFq",
        "colab_type": "text"
      },
      "source": [
        "## Define plotting functions\n",
        "\n",
        "The following [matplotlib](https://developers.google.com/machine-learning/glossary/#matplotlib) functions create the following plots:\n",
        "\n",
        "*  a scatter plot of the feature vs. the label, and a line showing the output of the trained model\n",
        "*  a loss curve\n",
        "\n",
        "You may optionally double-click the headline to see the matplotlib code, but note that writing matplotlib code is not an important part of learning ML programming."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QF0BFRXTOeR3",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Define the plotting functions\n",
        "def plot_the_model(trained_weight, trained_bias, feature, label):\n",
        "  \"\"\"Plot the trained model against 200 random training examples.\"\"\"\n",
        "\n",
        "  # Label the axes.\n",
        "  plt.xlabel(feature)\n",
        "  plt.ylabel(label)\n",
        "\n",
        "  # Create a scatter plot from 200 random points of the dataset.\n",
        "  random_examples = training_df.sample(n=200)\n",
        "  plt.scatter(random_examples[feature], random_examples[label])\n",
        "\n",
        "  # Create a red line representing the model. The red line starts\n",
        "  # at coordinates (x0, y0) and ends at coordinates (x1, y1).\n",
        "  x0 = 0\n",
        "  y0 = trained_bias\n",
        "  x1 = 10000\n",
        "  y1 = trained_bias + (trained_weight * x1)\n",
        "  plt.plot([x0, x1], [y0, y1], c='r')\n",
        "\n",
        "  # Render the scatter plot and the red line.\n",
        "  plt.show()\n",
        "\n",
        "\n",
        "def plot_the_loss_curve(epochs, rmse):\n",
        "  \"\"\"Plot a curve of loss vs. epoch.\"\"\"\n",
        "\n",
        "  plt.figure()\n",
        "  plt.xlabel(\"Epoch\")\n",
        "  plt.ylabel(\"Root Mean Squared Error\")\n",
        "\n",
        "  plt.plot(epochs, rmse, label=\"Loss\")\n",
        "  plt.legend()\n",
        "  plt.ylim([rmse.min()*0.97, rmse.max()])\n",
        "  plt.show()  \n",
        "\n",
        "print(\"Defined the plot_the_model and plot_the_loss_curve functions.\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D-IXYVfvM4gD",
        "colab_type": "text"
      },
      "source": [
        "## Call the model functions\n",
        "\n",
        "An important part of machine learning is determining which [features](https://developers.google.com/machine-learning/glossary/#feature) correlate with the [label](https://developers.google.com/machine-learning/glossary/#label). For example, real-life home-value prediction models typically rely on hundreds of features and synthetic features. However, this model relies on only one feature. For now, you'll arbitrarily use `total_rooms` as that feature. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nj3v5EKQFY8s",
        "colab_type": "code",
        "cellView": "both",
        "colab": {}
      },
      "source": [
        "# The following variables are the hyperparameters.\n",
        "learning_rate = 0.01\n",
        "epochs = 30\n",
        "batch_size = 30\n",
        "\n",
        "# Specify the feature and the label.\n",
        "my_feature = \"total_rooms\"  # the total number of rooms on a specific city block.\n",
        "my_label=\"median_house_value\" # the median value of a house on a specific city block.\n",
        "# That is, you're going to create a model that predicts house value based \n",
        "# solely on total_rooms.  \n",
        "\n",
        "# Discard any pre-existing version of the model.\n",
        "my_model = None\n",
        "\n",
        "# Invoke the functions.\n",
        "my_model = build_model(learning_rate)\n",
        "weight, bias, epochs, rmse = train_model(my_model, training_df, \n",
        "                                         my_feature, my_label,\n",
        "                                         epochs, batch_size)\n",
        "\n",
        "print(\"\\nThe learned weight for your model is %.4f\" % weight)\n",
        "print(\"The learned bias for your model is %.4f\\n\" % bias )\n",
        "\n",
        "plot_the_model(weight, bias, my_feature, my_label)\n",
        "plot_the_loss_curve(epochs, rmse)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Btp8zUNbYOcd",
        "colab_type": "text"
      },
      "source": [
        "A certain amount of randomness plays into training a model. Consequently, you'll get different results each time you train the model. That said, given the dataset and the hyperparameters, the trained model will generally do a poor job describing the feature's relation to the label."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xNqWWos_zyk",
        "colab_type": "text"
      },
      "source": [
        "## Use the model to make predictions\n",
        "\n",
        "You can use the trained model to make predictions. In practice, [you should make predictions on examples that are not used in training](https://developers.google.com/machine-learning/crash-course/training-and-test-sets/splitting-data). However, for this exercise, you'll just work with a subset of the same training dataset. A later Colab exercise will explore ways to make predictions on examples not used in training.\n",
        "\n",
        "First, run the following code to define the house prediction function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nH63BmncAcab",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def predict_house_values(n, feature, label):\n",
        "  \"\"\"Predict house values based on a feature.\"\"\"\n",
        "\n",
        "  batch = training_df[feature][10000:10000 + n]\n",
        "  predicted_values = my_model.predict_on_batch(x=batch)\n",
        "\n",
        "  print(\"feature   label          predicted\")\n",
        "  print(\"  value   value          value\")\n",
        "  print(\"          in thousand$   in thousand$\")\n",
        "  print(\"--------------------------------------\")\n",
        "  for i in range(n):\n",
        "    print (\"%5.0f %6.0f %15.0f\" % (training_df[feature][i],\n",
        "                                   training_df[label][i],\n",
        "                                   predicted_values[i][0] ))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NbBNQujU5WjK",
        "colab_type": "text"
      },
      "source": [
        "Now, invoke the house prediction function on 10 examples:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Y_0DGBt0Kz_N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "predict_house_values(10, my_feature, my_label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-gGaqArcpqY3",
        "colab_type": "text"
      },
      "source": [
        "### Task 2: Judge the predictive power of the model\n",
        "\n",
        "Look at the preceding table. How close is the predicted value to the label value?  In other words, does your model accurately predict house values?  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yVpjhUFm9uID",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click to view the answer.\n",
        "\n",
        "# Most of the predicted values differ significantly\n",
        "# from the label value, so the trained model probably \n",
        "# doesn't have much predictive power. However, the\n",
        "# first 10 examples might not be representative of \n",
        "# the rest of the examples.  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wLoqis3IUPSd",
        "colab_type": "text"
      },
      "source": [
        "## Task 3: Try a different feature\n",
        "\n",
        "The `total_rooms` feature had only a little predictive power. Would a different feature have greater predictive power?  Try using `population` as the feature instead of `total_rooms`. \n",
        "\n",
        "Note: When you change features, you might also need to change the hyperparameters."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H0ab6HD4ZO75",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "my_feature = \"?\"   # Replace the ? with population or possibly\n",
        "                   # a different column name.\n",
        "\n",
        "# Experiment with the hyperparameters.\n",
        "learning_rate = 2\n",
        "epochs = 3\n",
        "batch_size = 120\n",
        "\n",
        "# Don't change anything below this line.\n",
        "model = build_model(learning_rate)\n",
        "weight, bias, epochs, rmse = train_model(my_model, training_df, \n",
        "                                         my_feature, my_label,\n",
        "                                         epochs, batch_size)\n",
        "plot_the_model(weight, bias, my_feature, my_label)\n",
        "plot_the_loss_curve(epochs, rmse)\n",
        "\n",
        "predict_house_values(15, my_feature, my_label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "107mDkW7U6mg",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click to view a possible solution.\n",
        "\n",
        "my_feature = \"population\" # Pick a feature other than \"total_rooms\"\n",
        "\n",
        "# Possibly, experiment with the hyperparameters.\n",
        "learning_rate = 0.05\n",
        "epochs = 18\n",
        "batch_size = 3\n",
        "\n",
        "# Don't change anything below.\n",
        "my_model = build_model(learning_rate)\n",
        "weight, bias, epochs, rmse = train_model(my_model, training_df, \n",
        "                                         my_feature, my_label,\n",
        "                                         epochs, batch_size)\n",
        "\n",
        "plot_the_model(weight, bias, my_feature, my_label)\n",
        "plot_the_loss_curve(epochs, rmse)\n",
        "\n",
        "predict_house_values(10, my_feature, my_label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nd_rHJ59AUtk",
        "colab_type": "text"
      },
      "source": [
        "Did `population` produce better predictions than `total_rooms`?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F0tPEtzcC-vK",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click to view the answer.\n",
        "\n",
        "# Training is not entirely deterministic, but population \n",
        "# typically converges at a slightly higher RMSE than \n",
        "# total_rooms.  So, population appears to be about \n",
        "# the same or slightly worse at making predictions \n",
        "# than total_rooms."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C8uYpyGacsIg",
        "colab_type": "text"
      },
      "source": [
        "## Task 4: Define a synthetic feature\n",
        "\n",
        "You have determined that `total_rooms` and `population` were not useful features.  That is, neither the total number of rooms in a neighborhood nor the neighborhood's population successfully predicted the median house price of that neighborhood. Perhaps though, the *ratio* of `total_rooms` to `population` might have some predictive power. That is, perhaps block density relates to median house value.\n",
        "\n",
        "To explore this hypothesis, do the following: \n",
        "\n",
        "1. Create a [synthetic feature](https://developers.google.com/machine-learning/glossary/#synthetic_feature) that's a ratio of `total_rooms` to `population`. (If you are new to pandas DataFrames, please study the [Pandas DataFrame Ultraquick Tutorial](https://colab.research.google.com/github/google/eng-edu/blob/master/ml/cc/exercises/pandas_dataframe_ultraquick_tutorial.ipynb?utm_source=linearregressionreal-colab&utm_medium=colab&utm_campaign=colab-external&utm_content=pandas_tf2-colab&hl=en).)\n",
        "2. Tune the three hyperparameters.\n",
        "3. Determine whether this synthetic feature produces \n",
        "   a lower loss value than any of the single features you \n",
        "   tried earlier in this exercise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Kx2xHSgdcpg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Define a synthetic feature named rooms_per_person\n",
        "training_df[\"rooms_per_person\"] = ? # write your code here.\n",
        "\n",
        "# Don't change the next two lines.\n",
        "my_feature = \"rooms_per_person\"\n",
        "my_label=\"median_house_value\" \n",
        "\n",
        "# Assign values to these three hyperparameters.\n",
        "learning_rate = ?\n",
        "epochs = ?\n",
        "batch_size =  ?\n",
        "\n",
        "# Don't change anything below this line.\n",
        "my_model = build_model(learning_rate)\n",
        "weight, bias, epochs, rmse = train_model(my_model, training_df,\n",
        "                                         my_feature, my_label,\n",
        "                                         epochs, batch_size)\n",
        "\n",
        "plot_the_loss_curve(epochs, rmse)\n",
        "predict_house_values(15, my_feature, my_label)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xRfxp_3yofe3",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click to view a possible solution to Task 4.\n",
        "\n",
        "# Define a synthetic feature\n",
        "training_df[\"rooms_per_person\"] = training_df[\"total_rooms\"] / training_df[\"population\"]\n",
        "my_feature = \"rooms_per_person\"\n",
        "my_label=\"median_house_value\" \n",
        "\n",
        "# Tune the hyperparameters.\n",
        "learning_rate = 0.06\n",
        "epochs = 24\n",
        "batch_size = 30\n",
        "\n",
        "# Don't change anything below this line.\n",
        "my_model = build_model(learning_rate)\n",
        "weight, bias, epochs, mae = train_model(my_model, training_df,\n",
        "                                        my_feature, my_label,\n",
        "                                        epochs, batch_size)\n",
        "\n",
        "plot_the_loss_curve(epochs, mae)\n",
        "predict_house_values(15, my_feature, my_label)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HBiDWursB1Wi",
        "colab_type": "text"
      },
      "source": [
        "Based on the loss values, this synthetic feature produces a better model than the individual features you tried in Task 2 and Task 3. However, the model still isn't creating great predictions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XEG_9oU9O54u",
        "colab_type": "text"
      },
      "source": [
        "## Task 5. Find feature(s) whose raw values correlate with the label\n",
        "\n",
        "So far, we've relied on trial-and-error to identify possible features for the model.  Let's rely on statistics instead.\n",
        "\n",
        "A **correlation matrix** indicates how each attribute's raw values relate to the other attributes' raw values. Correlation values have the following meanings:\n",
        "\n",
        "  * `1.0`: perfect positive correlation; that is, when one attribute rises, the other attribute rises.\n",
        "  * `-1.0`: perfect negative correlation; that is, when one attribute rises, the other attribute falls. \n",
        "  * `0.0`: no correlation; the two column's [are not linearly related](https://en.wikipedia.org/wiki/Correlation_and_dependence#/media/File:Correlation_examples2.svg).\n",
        "\n",
        "In general, the higher the absolute value of a correlation value, the greater its predictive power. For example, a correlation value of -0.8 implies far more predictive power than a correlation of -0.2.\n",
        "\n",
        "The following code cell generates the correlation matrix for attributes of the California Housing Dataset:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zFGKL45LO8Tt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Generate a correlation matrix.\n",
        "training_df.corr()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hp0r3NAVPEdt",
        "colab_type": "text"
      },
      "source": [
        "The correlation matrix shows nine potential features (including a synthetic\n",
        "feature) and one label (`median_house_value`).  A strong negative correlation or strong positive correlation with the label suggests a potentially good feature.  \n",
        "\n",
        "**Your Task:** Determine which of the nine potential features appears to be the best candidate for a feature?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RomQTd1OPVd0",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Double-click here for the solution to Task 5\n",
        "\n",
        "# The `median_income` correlates 0.7 with the label \n",
        "# (median_house_value), so median_income` might be a \n",
        "# good feature. The other seven potential features\n",
        "# all have a correlation relatively close to 0. \n",
        "\n",
        "# If time permits, try median_income as the feature\n",
        "# and see whether the model improves."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8RqvEbaVSlRt",
        "colab_type": "text"
      },
      "source": [
        "Correlation matrices don't tell the entire story. In later exercises, you'll find additional ways to unlock predictive power from potential features.\n",
        "\n",
        "**Note:** Using `median_income` as a feature may raise some ethical and fairness\n",
        "issues. Towards the end of the course, we'll explore ethical and fairness issues."
      ]
    }
  ]
}