W6BLectureExercises_sol.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data analysis tools 2\n",
    "**Lecture and exercises for week 6B** in Digital Methods, University of Copenhagen\n",
    "\n",
    "*Note: Due to the sudden demand for remote learning, we are experimenting with different ways of creating a good\n",
    "remote learning experience. Therefore your feedback and suggestions are encouraged. You can give feedback\n",
    "anonymously through [this web form](https://ulfaslak.com/vent) or reach out to a teacher. This notebook contains\n",
    "bits of lecturing and exercises interweaved. There is no seperate lecture file, everything you need is in here.\n",
    "Later this week I will upload solutions possibly in video format, explaining the solutions. It is important to note\n",
    "that we will probably iterate on this format, so we ask for your patience if we mess up.*\n",
    "\n",
    "***Also, due to this special format, it is important you DON'T SKIP AHEAD and go straight for the exercises. Try\n",
    "to consume the content of this notebook in the order it is presented, otherwise you may miss important points\n",
    "that we, for obvious reasons, cannot communicate in person.***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T08:25:41.784828Z",
     "start_time": "2020-03-13T08:25:41.781164Z"
    }
   },
   "outputs": [],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "def video_html(url):\n",
    "    return f\"\"\"\n",
    "    <div align=\"middle\">\n",
    "    <video width=\"50%\" controls>\n",
    "      <source src=\"{url}\" type=\"video/mp4\">\n",
    "    </video></div>\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. `NumPy` + `pandas` = ❤️\n",
    "### *(a brief primer)*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When manipulating data in Python, NumPy and pandas are your go-to tools. Numpy is an extremely fast *low-level*\n",
    "library for manipulating N-dimensional *arrays* (from lists (1D), to tables (2D), to boxes (3D), etc.). It also\n",
    "contains a bunch of mathematical tools for advanced analysis. pandas is a fast *high-level* library for manipulating\n",
    "series (1D) and tables (2D). It let's you do a lot of very sophisticated data manipulation operations without\n",
    "a whole lot of code (hence the term *high-level*). In summary:\n",
    "* **NumPy** is the fundamental package for scientific computing with Python.<br>\n",
    "* **pandas** is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool\n",
    "\n",
    "Friedolin gave you a nice introduction to these libraries last week, but **for a short recap, watch the video below**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T09:52:06.398921Z",
     "start_time": "2020-03-13T09:52:06.394874Z"
    },
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/5vbwkjzd6nrk24g/1_numpy_and_pandas_min.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/5vbwkjzd6nrk24g/1_numpy_and_pandas_min.mov?raw=1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here is a quick demonstration of how to use NumPy and pandas in Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T09:38:58.914165Z",
     "start_time": "2020-03-13T09:38:58.909123Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/1d0l26ilr80ez31/2_numpy_and_pandas_example_min.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/1d0l26ilr80ez31/2_numpy_and_pandas_example_min.mov?raw=1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 1**: Import `numpy`. Use the `np.random.randint` function to create a 10 by 2 array of random integers.\n",
    "Display this array.\n",
    ">\n",
    "> *Hint: Remember you can do`?np.random.randint` to display the documentation. Read about the `size` argument.\n",
    "It says to pass it a \"tuple\". A tuple is like a list, only it uses parentheses, not square brackets. Try passing\n",
    "it the tuple `(10, 2)`.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T04:59:19.862301Z",
     "start_time": "2020-03-22T04:59:19.849776Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 7],\n",
       "       [1, 4],\n",
       "       [8, 1],\n",
       "       [6, 1],\n",
       "       [0, 1],\n",
       "       [8, 0],\n",
       "       [9, 0],\n",
       "       [0, 6],\n",
       "       [0, 9],\n",
       "       [1, 5]])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "arr = np.random.randint(0, 10, size=(10, 2))\n",
    "arr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 2**: Import `pandas`. Load the array you just created as a `pd.DataFrame`. Name the row and column indices\n",
    "(anything you like). Display this DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T04:59:44.838537Z",
     "start_time": "2020-03-22T04:59:44.829473Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   0  1\n",
       "0  3  7\n",
       "1  1  4\n",
       "2  8  1\n",
       "3  6  1\n",
       "4  0  1\n",
       "5  8  0\n",
       "6  9  0\n",
       "7  0  6\n",
       "8  0  9\n",
       "9  1  5"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame(arr)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 3**: Turn the DataFrame back into an array! Display the type of the resulting array. What does this reveal\n",
    "about the connection between pandas and NumPy?\n",
    ">\n",
    "> *Hint: You can do this by getting it's `values` attribute. To get an attribute from an object, you use the \"dot-\n",
    "operator\". Like `object.attribute`.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T04:59:52.113023Z",
     "start_time": "2020-03-22T04:59:52.108879Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[3, 7],\n",
       "       [1, 4],\n",
       "       [8, 1],\n",
       "       [6, 1],\n",
       "       [0, 1],\n",
       "       [8, 0],\n",
       "       [9, 0],\n",
       "       [0, 6],\n",
       "       [0, 9],\n",
       "       [1, 5]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merging/joining datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's not uncommon that we need to stick to pieces of data together to make a new table. Sometimes merging data is\n",
    "straight forward, both pieces have the same shape and we can just slab one on top of the other and all is good. But\n",
    "other times, one piece has more, fewer or different columns. Sometimes, data points in either dataset needs to be\n",
    "merged somehow. **In the video below we have look at different cases and how to manage them**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T09:41:32.797133Z",
     "start_time": "2020-03-13T09:41:32.791593Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/qprehqimhkp9lpe/3_joining_data_min.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/qprehqimhkp9lpe/3_joining_data_min.mov?raw=1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also watch this video where I give **some examples** of how to merge data in pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T11:23:01.466747Z",
     "start_time": "2020-03-13T11:23:01.462029Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/qj85zi8yq6jbrxp/4_joining_data_example.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/qj85zi8yq6jbrxp/4_joining_data_example.mov?raw=1'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 4**: Consider the two pandas dataframes `df1` and `df2` below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:00:35.339660Z",
     "start_time": "2020-03-22T05:00:35.331972Z"
    }
   },
   "outputs": [],
   "source": [
    "df1 = pd.DataFrame(\n",
    "    np.random.randint(0, 10, size=(3, 4)),\n",
    "    index=[0, 1, 2], columns=['a', 'b', 'c', 'd']\n",
    ")\n",
    "\n",
    "df2 = pd.DataFrame(\n",
    "    np.random.randint(0, 10, size=(5, 3)),\n",
    "    index=[1, 2, 3, 4, 5], columns=['c', 'd', 'e']\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Join `df1` and `df2` (using `pd.concat`) with argument `axis=1`.\n",
    ">\n",
    "> 1. What is the resulting shape if you merge them using an outer join?\n",
    "> 1. What is the resulting shape if you merge them using an inner join?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:01:58.717085Z",
     "start_time": "2020-03-22T05:01:58.705502Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b    c    d    c    d    e\n",
       "0  4.0  2.0  7.0  5.0  NaN  NaN  NaN\n",
       "1  2.0  7.0  0.0  7.0  1.0  4.0  4.0\n",
       "2  5.0  7.0  1.0  3.0  2.0  0.0  4.0\n",
       "3  NaN  NaN  NaN  NaN  0.0  8.0  0.0\n",
       "4  NaN  NaN  NaN  NaN  9.0  7.0  2.0\n",
       "5  NaN  NaN  NaN  NaN  4.0  6.0  7.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df1, df2], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the outer join we get 5 rows and 7 columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the inner join we get 2 rows and 7 columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 5**: Again, use `pd.concat` to outer join `df1` and `df2`, but this time use `axis=0`.\n",
    "> 1. Qualitatively, what is the difference now that `axis=1`, compared to when `axis=0`? You can for examlpe\n",
    "display the two results side by size and try to spot the pattern difference.\n",
    "> 1. Read about the `axis` argument (remember you can use `?pd.concat` to launch the docs). Does this correspond\n",
    "with your understanding? Can you predict what an inner join of `df1` and `df2` with `axis=0` would look like?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:02:17.770536Z",
     "start_time": "2020-03-22T05:02:17.758616Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:1: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>a</th>\n",
       "      <th>b</th>\n",
       "      <th>c</th>\n",
       "      <th>d</th>\n",
       "      <th>e</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     a    b  c  d    e\n",
       "0  4.0  2.0  7  5  NaN\n",
       "1  2.0  7.0  0  7  NaN\n",
       "2  5.0  7.0  1  3  NaN\n",
       "1  NaN  NaN  1  4  4.0\n",
       "2  NaN  NaN  2  0  4.0\n",
       "3  NaN  NaN  0  8  0.0\n",
       "4  NaN  NaN  9  7  2.0\n",
       "5  NaN  NaN  4  6  7.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df1, df2], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This data frame is taller. It has 7 rows and 5 columns, the inverse of when `axis=1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When reading about `pd.concat`, I learn that the axis argument\n",
    "dictates whether dataframes are stacked horizontally (axis=1) or\n",
    "vertically (axis=0). So if we stack vertically, like here, and\n",
    "make the inner join, then it is only *columns* that exist in both\n",
    "dataframes that are kept in the result. So the shape of the resulting\n",
    "dataframe would then be 7 by 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 6**: So we have learned that in pandas, anything can be concatenated. That's not the case in NumPy!\n",
    "Consider the two arrays below `arr1` and `arr2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:06:18.451403Z",
     "start_time": "2020-03-22T05:06:18.444519Z"
    }
   },
   "outputs": [],
   "source": [
    "arr1 = np.random.randint(0, 10, size=(3, 4))\n",
    "arr2 = np.random.randint(0, 10, size=(3, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 1. Use `np.vstack` to concatenate `arr1` and `arr2` below. What does the error say? And why?\n",
    "> 1. Now use `np.hstack` to concatenate `arr1` and `arr2`. It works! But why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:06:31.544972Z",
     "start_time": "2020-03-22T05:06:31.530112Z"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 4 and the array at index 1 has size 3",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-4803ef01c61c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvstack\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0marr1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marr2\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mvstack\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;32m/anaconda3/lib/python3.7/site-packages/numpy/core/shape_base.py\u001b[0m in \u001b[0;36mvstack\u001b[0;34m(tup)\u001b[0m\n\u001b[1;32m    280\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marrs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    281\u001b[0m         \u001b[0marrs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0marrs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 282\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_nx\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcatenate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marrs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    283\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    284\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mconcatenate\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 4 and the array at index 1 has size 3"
     ]
    }
   ],
   "source": [
    "np.vstack([arr1, arr2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In essence, the message is telling us that the arrays we are trying\n",
    "to merge do not match in size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:07:27.891307Z",
     "start_time": "2020-03-22T05:07:27.887161Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5, 6, 6, 9, 5, 0, 4],\n",
       "       [3, 2, 2, 7, 5, 2, 0],\n",
       "       [9, 3, 4, 5, 7, 2, 5]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.hstack([arr1, arr2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This works because both arr1 and arr2 have three rows, so stacking\n",
    "then horizontally is doable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is useful to be able to summarize a dataset in a few key numbers. *Summary statistics*\n",
    "does that for us. pandas has a set of nice tools to make this very easy! First let's load\n",
    "some data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:08:21.519593Z",
     "start_time": "2020-03-22T05:08:20.844376Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.23</td>\n",
       "      <td>1.71</td>\n",
       "      <td>2.43</td>\n",
       "      <td>15.6</td>\n",
       "      <td>127.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.06</td>\n",
       "      <td>0.28</td>\n",
       "      <td>2.29</td>\n",
       "      <td>5.64</td>\n",
       "      <td>1.04</td>\n",
       "      <td>3.92</td>\n",
       "      <td>1065.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>13.20</td>\n",
       "      <td>1.78</td>\n",
       "      <td>2.14</td>\n",
       "      <td>11.2</td>\n",
       "      <td>100.0</td>\n",
       "      <td>2.65</td>\n",
       "      <td>2.76</td>\n",
       "      <td>0.26</td>\n",
       "      <td>1.28</td>\n",
       "      <td>4.38</td>\n",
       "      <td>1.05</td>\n",
       "      <td>3.40</td>\n",
       "      <td>1050.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.16</td>\n",
       "      <td>2.36</td>\n",
       "      <td>2.67</td>\n",
       "      <td>18.6</td>\n",
       "      <td>101.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>3.24</td>\n",
       "      <td>0.30</td>\n",
       "      <td>2.81</td>\n",
       "      <td>5.68</td>\n",
       "      <td>1.03</td>\n",
       "      <td>3.17</td>\n",
       "      <td>1185.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>14.37</td>\n",
       "      <td>1.95</td>\n",
       "      <td>2.50</td>\n",
       "      <td>16.8</td>\n",
       "      <td>113.0</td>\n",
       "      <td>3.85</td>\n",
       "      <td>3.49</td>\n",
       "      <td>0.24</td>\n",
       "      <td>2.18</td>\n",
       "      <td>7.80</td>\n",
       "      <td>0.86</td>\n",
       "      <td>3.45</td>\n",
       "      <td>1480.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13.24</td>\n",
       "      <td>2.59</td>\n",
       "      <td>2.87</td>\n",
       "      <td>21.0</td>\n",
       "      <td>118.0</td>\n",
       "      <td>2.80</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0.39</td>\n",
       "      <td>1.82</td>\n",
       "      <td>4.32</td>\n",
       "      <td>1.04</td>\n",
       "      <td>2.93</td>\n",
       "      <td>735.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   alcohol  malic_acid   ash  alcalinity_of_ash  magnesium  total_phenols  \\\n",
       "0    14.23        1.71  2.43               15.6      127.0           2.80   \n",
       "1    13.20        1.78  2.14               11.2      100.0           2.65   \n",
       "2    13.16        2.36  2.67               18.6      101.0           2.80   \n",
       "3    14.37        1.95  2.50               16.8      113.0           3.85   \n",
       "4    13.24        2.59  2.87               21.0      118.0           2.80   \n",
       "\n",
       "   flavanoids  nonflavanoid_phenols  proanthocyanins  color_intensity   hue  \\\n",
       "0        3.06                  0.28             2.29             5.64  1.04   \n",
       "1        2.76                  0.26             1.28             4.38  1.05   \n",
       "2        3.24                  0.30             2.81             5.68  1.03   \n",
       "3        3.49                  0.24             2.18             7.80  0.86   \n",
       "4        2.69                  0.39             1.82             4.32  1.04   \n",
       "\n",
       "   od280/od315_of_diluted_wines  proline  \n",
       "0                          3.92   1065.0  \n",
       "1                          3.40   1050.0  \n",
       "2                          3.17   1185.0  \n",
       "3                          3.45   1480.0  \n",
       "4                          2.93    735.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import\n",
    "from sklearn.datasets import load_wine\n",
    "\n",
    "# Structure as pd.DataFrame\n",
    "data = load_wine()\n",
    "wine_df = pd.DataFrame(data['data'], columns=data['feature_names'])\n",
    "\n",
    "# display\n",
    "wine_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset contains information and measurements on 173 Italian wines. Read the column\n",
    "names and look at the values and try to make sense of it. What are typical values for each\n",
    "feature? How much do they deviate? What's the lowest they get? etc. Here's a short video \n",
    "where I explain different statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T11:50:25.771084Z",
     "start_time": "2020-03-13T11:50:25.765516Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/bwlhgpwcbq3652l/5_summary_statistics.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/bwlhgpwcbq3652l/5_summary_statistics.mov?raw=1'))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 7**: If you followed my instructions above, you should hopefully felt that staring\n",
    "at data like that isn't the most meaningful way to spend your time. So go ahead and run\n",
    "`wine_df.describe()`. What does it show?\n",
    "1. Explain what the *mean* and *std* values tell you about. Inside which range of alcohol\n",
    "content does 68% of the data lie?\n",
    "1. Explain what *min*, *max* and the *25%*, *50%*, and *75%* values mean.\n",
    "1. What is another common word for the *50%* value?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:08:38.123166Z",
     "start_time": "2020-03-22T05:08:38.071151Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alcohol</th>\n",
       "      <th>malic_acid</th>\n",
       "      <th>ash</th>\n",
       "      <th>alcalinity_of_ash</th>\n",
       "      <th>magnesium</th>\n",
       "      <th>total_phenols</th>\n",
       "      <th>flavanoids</th>\n",
       "      <th>nonflavanoid_phenols</th>\n",
       "      <th>proanthocyanins</th>\n",
       "      <th>color_intensity</th>\n",
       "      <th>hue</th>\n",
       "      <th>od280/od315_of_diluted_wines</th>\n",
       "      <th>proline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "      <td>178.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>13.000618</td>\n",
       "      <td>2.336348</td>\n",
       "      <td>2.366517</td>\n",
       "      <td>19.494944</td>\n",
       "      <td>99.741573</td>\n",
       "      <td>2.295112</td>\n",
       "      <td>2.029270</td>\n",
       "      <td>0.361854</td>\n",
       "      <td>1.590899</td>\n",
       "      <td>5.058090</td>\n",
       "      <td>0.957449</td>\n",
       "      <td>2.611685</td>\n",
       "      <td>746.893258</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.811827</td>\n",
       "      <td>1.117146</td>\n",
       "      <td>0.274344</td>\n",
       "      <td>3.339564</td>\n",
       "      <td>14.282484</td>\n",
       "      <td>0.625851</td>\n",
       "      <td>0.998859</td>\n",
       "      <td>0.124453</td>\n",
       "      <td>0.572359</td>\n",
       "      <td>2.318286</td>\n",
       "      <td>0.228572</td>\n",
       "      <td>0.709990</td>\n",
       "      <td>314.907474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>11.030000</td>\n",
       "      <td>0.740000</td>\n",
       "      <td>1.360000</td>\n",
       "      <td>10.600000</td>\n",
       "      <td>70.000000</td>\n",
       "      <td>0.980000</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>0.130000</td>\n",
       "      <td>0.410000</td>\n",
       "      <td>1.280000</td>\n",
       "      <td>0.480000</td>\n",
       "      <td>1.270000</td>\n",
       "      <td>278.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>12.362500</td>\n",
       "      <td>1.602500</td>\n",
       "      <td>2.210000</td>\n",
       "      <td>17.200000</td>\n",
       "      <td>88.000000</td>\n",
       "      <td>1.742500</td>\n",
       "      <td>1.205000</td>\n",
       "      <td>0.270000</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>3.220000</td>\n",
       "      <td>0.782500</td>\n",
       "      <td>1.937500</td>\n",
       "      <td>500.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>13.050000</td>\n",
       "      <td>1.865000</td>\n",
       "      <td>2.360000</td>\n",
       "      <td>19.500000</td>\n",
       "      <td>98.000000</td>\n",
       "      <td>2.355000</td>\n",
       "      <td>2.135000</td>\n",
       "      <td>0.340000</td>\n",
       "      <td>1.555000</td>\n",
       "      <td>4.690000</td>\n",
       "      <td>0.965000</td>\n",
       "      <td>2.780000</td>\n",
       "      <td>673.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>13.677500</td>\n",
       "      <td>3.082500</td>\n",
       "      <td>2.557500</td>\n",
       "      <td>21.500000</td>\n",
       "      <td>107.000000</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>2.875000</td>\n",
       "      <td>0.437500</td>\n",
       "      <td>1.950000</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>1.120000</td>\n",
       "      <td>3.170000</td>\n",
       "      <td>985.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>14.830000</td>\n",
       "      <td>5.800000</td>\n",
       "      <td>3.230000</td>\n",
       "      <td>30.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>3.880000</td>\n",
       "      <td>5.080000</td>\n",
       "      <td>0.660000</td>\n",
       "      <td>3.580000</td>\n",
       "      <td>13.000000</td>\n",
       "      <td>1.710000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1680.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          alcohol  malic_acid         ash  alcalinity_of_ash   magnesium  \\\n",
       "count  178.000000  178.000000  178.000000         178.000000  178.000000   \n",
       "mean    13.000618    2.336348    2.366517          19.494944   99.741573   \n",
       "std      0.811827    1.117146    0.274344           3.339564   14.282484   \n",
       "min     11.030000    0.740000    1.360000          10.600000   70.000000   \n",
       "25%     12.362500    1.602500    2.210000          17.200000   88.000000   \n",
       "50%     13.050000    1.865000    2.360000          19.500000   98.000000   \n",
       "75%     13.677500    3.082500    2.557500          21.500000  107.000000   \n",
       "max     14.830000    5.800000    3.230000          30.000000  162.000000   \n",
       "\n",
       "       total_phenols  flavanoids  nonflavanoid_phenols  proanthocyanins  \\\n",
       "count     178.000000  178.000000            178.000000       178.000000   \n",
       "mean        2.295112    2.029270              0.361854         1.590899   \n",
       "std         0.625851    0.998859              0.124453         0.572359   \n",
       "min         0.980000    0.340000              0.130000         0.410000   \n",
       "25%         1.742500    1.205000              0.270000         1.250000   \n",
       "50%         2.355000    2.135000              0.340000         1.555000   \n",
       "75%         2.800000    2.875000              0.437500         1.950000   \n",
       "max         3.880000    5.080000              0.660000         3.580000   \n",
       "\n",
       "       color_intensity         hue  od280/od315_of_diluted_wines      proline  \n",
       "count       178.000000  178.000000                    178.000000   178.000000  \n",
       "mean          5.058090    0.957449                      2.611685   746.893258  \n",
       "std           2.318286    0.228572                      0.709990   314.907474  \n",
       "min           1.280000    0.480000                      1.270000   278.000000  \n",
       "25%           3.220000    0.782500                      1.937500   500.500000  \n",
       "50%           4.690000    0.965000                      2.780000   673.500000  \n",
       "75%           6.200000    1.120000                      3.170000   985.000000  \n",
       "max          13.000000    1.710000                      4.000000  1680.000000  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wine_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a summary statistics table. It's giving us the most important\n",
    "stats for each variable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 1**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **mean** is the average. You attain it by adding all the values\n",
    "and dividing by number of values.<br>\n",
    "The **std** is the *average deviation from the average*. 68% of all the\n",
    "data lies within one standard deviation around the mean. For the alcohol\n",
    "variable it means that 68% of the wines have an alcohold content between\n",
    "12.2 and 13.8. So it gives us a sense of how widely spread the data is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **min** is the smallest value of a variable. The **max** is the\n",
    "greatest.<br>\n",
    "The **percentiles** are statistics we can use to access how the data is\n",
    "distributed. Techically, to attain them for a variable, we sort the values\n",
    "then move **25%** into the list of sorted values and pick whatever value is\n",
    "there, then the same for **50%** and ***75%**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS 3**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 50% percentile is also called the **median**. This is somewhat similar\n",
    "to the *mean*, and for normally distributed (bell curves-like) data \n",
    "mean and median are usually the same. But in highly skewed variable (such \n",
    "as yearly income of individuals) the median will usually be much greater\n",
    "than the mean, because a few samples are *dragging* it towards higher values.\n",
    "In *skewed* variables like this, the median is a good measure to report,\n",
    "because it gives you a clearer picture of where most of the data is \n",
    "(what most mortals are making in a year)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Group-by operations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often times we have some data, and need to know e.g. the average of every datapoint\n",
    "that matched a certain criteria. Or our datapoints are categorized by some variable\n",
    "(e.g. *sex* categorizes people into *male*, *female* and sometimes *non-binary*), and we want\n",
    "to count how many datapoints that fall in each category. Then we use *group-by*!\n",
    "\n",
    "We are going to use the Titanic dataset for this exercise. Each row is a passanger, and each\n",
    "column describes a feature of the passangers. Please stare at it in silence for 2 minutes, and\n",
    "then watch the video below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:51:25.671878Z",
     "start_time": "2020-03-22T05:51:24.645342Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Name</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Siblings/Spouses Aboard</th>\n",
       "      <th>Parents/Children Aboard</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Owen Harris Braund</td>\n",
       "      <td>male</td>\n",
       "      <td>22.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. John Bradley (Florence Briggs Thayer) Cum...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Laina Heikkinen</td>\n",
       "      <td>female</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. Jacques Heath (Lily May Peel) Futrelle</td>\n",
       "      <td>female</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. William Henry Allen</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. James Moran</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.4583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Timothy J McCarthy</td>\n",
       "      <td>male</td>\n",
       "      <td>54.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>51.8625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Master. Gosta Leonard Palsson</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>21.0750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. Oscar W (Elisabeth Vilhelmina Berg) Johnson</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11.1333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. Nicholas (Adele Achem) Nasser</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Marguerite Rut Sandstrom</td>\n",
       "      <td>female</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>16.7000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Miss. Elizabeth Bonnell</td>\n",
       "      <td>female</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. William Henry Saundercock</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Anders Johan Andersson</td>\n",
       "      <td>male</td>\n",
       "      <td>39.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>31.2750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Hulda Amanda Adolfina Vestrom</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. (Mary D Kingcome) Hewlett</td>\n",
       "      <td>female</td>\n",
       "      <td>55.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Master. Eugene Rice</td>\n",
       "      <td>male</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>29.1250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. Charles Eugene Williams</td>\n",
       "      <td>male</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. Julius (Emelia Maria Vandemoortele) Vande...</td>\n",
       "      <td>female</td>\n",
       "      <td>31.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. Fatima Masselmani</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. Joseph J Fynney</td>\n",
       "      <td>male</td>\n",
       "      <td>35.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. Lawrence Beesley</td>\n",
       "      <td>male</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Anna McGowan</td>\n",
       "      <td>female</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.0292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. William Thompson Sloper</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Torborg Danira Palsson</td>\n",
       "      <td>female</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>21.0750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. Carl Oscar (Selma Augusta Emilia Johansso...</td>\n",
       "      <td>female</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>31.3875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Farred Chehab Emir</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Charles Alexander Fortune</td>\n",
       "      <td>male</td>\n",
       "      <td>19.0</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>263.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Ellen O'Dwyer</td>\n",
       "      <td>female</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Lalio Todoroff</td>\n",
       "      <td>male</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>857</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. Frederick Edward Giles</td>\n",
       "      <td>male</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>11.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>858</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. Frederick Joel (Margaret Welles Barron) S...</td>\n",
       "      <td>female</td>\n",
       "      <td>48.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25.9292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>859</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Dorothy Edith Sage</td>\n",
       "      <td>female</td>\n",
       "      <td>14.0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>69.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. John William Gill</td>\n",
       "      <td>male</td>\n",
       "      <td>24.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>861</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. (Karolina) Bystrom</td>\n",
       "      <td>female</td>\n",
       "      <td>42.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>862</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Miss. Asuncion Duran y More</td>\n",
       "      <td>female</td>\n",
       "      <td>27.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>863</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Washington Augustus II Roebling</td>\n",
       "      <td>male</td>\n",
       "      <td>31.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>50.4958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>864</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Philemon van Melkebeke</td>\n",
       "      <td>male</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>865</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Master. Harold Theodor Johnson</td>\n",
       "      <td>male</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11.1333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>866</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Cerin Balkic</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>867</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. Richard Leonard (Sallie Monypeny) Beckwith</td>\n",
       "      <td>female</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>52.5542</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>868</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Frans Olof Carlsson</td>\n",
       "      <td>male</td>\n",
       "      <td>33.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>869</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Victor Vander Cruyssen</td>\n",
       "      <td>male</td>\n",
       "      <td>47.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>870</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. Samuel (Hannah Wizosky) Abelson</td>\n",
       "      <td>female</td>\n",
       "      <td>28.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>24.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>871</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Adele Kiamie Najib</td>\n",
       "      <td>female</td>\n",
       "      <td>15.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.2250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>872</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Alfred Ossian Gustafsson</td>\n",
       "      <td>male</td>\n",
       "      <td>20.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9.8458</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Nedelio Petroff</td>\n",
       "      <td>male</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>874</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Kristo Laleff</td>\n",
       "      <td>male</td>\n",
       "      <td>23.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>875</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mrs. Thomas Jr (Lily Alexenia Wilson) Potter</td>\n",
       "      <td>female</td>\n",
       "      <td>56.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>83.1583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>876</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Mrs. William (Imanita Parrish Hall) Shelley</td>\n",
       "      <td>female</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>26.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>877</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Johann Markun</td>\n",
       "      <td>male</td>\n",
       "      <td>33.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.8958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>878</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Gerda Ulrika Dahlberg</td>\n",
       "      <td>female</td>\n",
       "      <td>22.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.5167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>879</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Mr. Frederick James Banfield</td>\n",
       "      <td>male</td>\n",
       "      <td>28.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10.5000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>880</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Henry Jr Sutehall</td>\n",
       "      <td>male</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.0500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>881</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mrs. William (Margaret Norton) Rice</td>\n",
       "      <td>female</td>\n",
       "      <td>39.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>29.1250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>882</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Rev. Juozas Montvila</td>\n",
       "      <td>male</td>\n",
       "      <td>27.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>13.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>883</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Miss. Margaret Edith Graham</td>\n",
       "      <td>female</td>\n",
       "      <td>19.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>884</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Miss. Catherine Helen Johnston</td>\n",
       "      <td>female</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>23.4500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>885</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Mr. Karl Howell Behr</td>\n",
       "      <td>male</td>\n",
       "      <td>26.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>886</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>Mr. Patrick Dooley</td>\n",
       "      <td>male</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>887 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Survived  Pclass                                               Name  \\\n",
       "0           0       3                             Mr. Owen Harris Braund   \n",
       "1           1       1  Mrs. John Bradley (Florence Briggs Thayer) Cum...   \n",
       "2           1       3                              Miss. Laina Heikkinen   \n",
       "3           1       1        Mrs. Jacques Heath (Lily May Peel) Futrelle   \n",
       "4           0       3                            Mr. William Henry Allen   \n",
       "5           0       3                                    Mr. James Moran   \n",
       "6           0       1                             Mr. Timothy J McCarthy   \n",
       "7           0       3                      Master. Gosta Leonard Palsson   \n",
       "8           1       3   Mrs. Oscar W (Elisabeth Vilhelmina Berg) Johnson   \n",
       "9           1       2                 Mrs. Nicholas (Adele Achem) Nasser   \n",
       "10          1       3                     Miss. Marguerite Rut Sandstrom   \n",
       "11          1       1                            Miss. Elizabeth Bonnell   \n",
       "12          0       3                      Mr. William Henry Saundercock   \n",
       "13          0       3                         Mr. Anders Johan Andersson   \n",
       "14          0       3                Miss. Hulda Amanda Adolfina Vestrom   \n",
       "15          1       2                     Mrs. (Mary D Kingcome) Hewlett   \n",
       "16          0       3                                Master. Eugene Rice   \n",
       "17          1       2                        Mr. Charles Eugene Williams   \n",
       "18          0       3  Mrs. Julius (Emelia Maria Vandemoortele) Vande...   \n",
       "19          1       3                             Mrs. Fatima Masselmani   \n",
       "20          0       2                                Mr. Joseph J Fynney   \n",
       "21          1       2                               Mr. Lawrence Beesley   \n",
       "22          1       3                                 Miss. Anna McGowan   \n",
       "23          1       1                        Mr. William Thompson Sloper   \n",
       "24          0       3                       Miss. Torborg Danira Palsson   \n",
       "25          1       3  Mrs. Carl Oscar (Selma Augusta Emilia Johansso...   \n",
       "26          0       3                             Mr. Farred Chehab Emir   \n",
       "27          0       1                      Mr. Charles Alexander Fortune   \n",
       "28          1       3                                Miss. Ellen O'Dwyer   \n",
       "29          0       3                                 Mr. Lalio Todoroff   \n",
       "..        ...     ...                                                ...   \n",
       "857         0       2                         Mr. Frederick Edward Giles   \n",
       "858         1       1  Mrs. Frederick Joel (Margaret Welles Barron) S...   \n",
       "859         0       3                           Miss. Dorothy Edith Sage   \n",
       "860         0       2                              Mr. John William Gill   \n",
       "861         1       2                            Mrs. (Karolina) Bystrom   \n",
       "862         1       2                        Miss. Asuncion Duran y More   \n",
       "863         0       1                Mr. Washington Augustus II Roebling   \n",
       "864         0       3                         Mr. Philemon van Melkebeke   \n",
       "865         1       3                     Master. Harold Theodor Johnson   \n",
       "866         0       3                                   Mr. Cerin Balkic   \n",
       "867         1       1    Mrs. Richard Leonard (Sallie Monypeny) Beckwith   \n",
       "868         0       1                            Mr. Frans Olof Carlsson   \n",
       "869         0       3                         Mr. Victor Vander Cruyssen   \n",
       "870         1       2               Mrs. Samuel (Hannah Wizosky) Abelson   \n",
       "871         1       3                           Miss. Adele Kiamie Najib   \n",
       "872         0       3                       Mr. Alfred Ossian Gustafsson   \n",
       "873         0       3                                Mr. Nedelio Petroff   \n",
       "874         0       3                                  Mr. Kristo Laleff   \n",
       "875         1       1       Mrs. Thomas Jr (Lily Alexenia Wilson) Potter   \n",
       "876         1       2        Mrs. William (Imanita Parrish Hall) Shelley   \n",
       "877         0       3                                  Mr. Johann Markun   \n",
       "878         0       3                        Miss. Gerda Ulrika Dahlberg   \n",
       "879         0       2                       Mr. Frederick James Banfield   \n",
       "880         0       3                              Mr. Henry Jr Sutehall   \n",
       "881         0       3                Mrs. William (Margaret Norton) Rice   \n",
       "882         0       2                               Rev. Juozas Montvila   \n",
       "883         1       1                        Miss. Margaret Edith Graham   \n",
       "884         0       3                     Miss. Catherine Helen Johnston   \n",
       "885         1       1                               Mr. Karl Howell Behr   \n",
       "886         0       3                                 Mr. Patrick Dooley   \n",
       "\n",
       "        Sex   Age  Siblings/Spouses Aboard  Parents/Children Aboard      Fare  \n",
       "0      male  22.0                        1                        0    7.2500  \n",
       "1    female  38.0                        1                        0   71.2833  \n",
       "2    female  26.0                        0                        0    7.9250  \n",
       "3    female  35.0                        1                        0   53.1000  \n",
       "4      male  35.0                        0                        0    8.0500  \n",
       "5      male  27.0                        0                        0    8.4583  \n",
       "6      male  54.0                        0                        0   51.8625  \n",
       "7      male   2.0                        3                        1   21.0750  \n",
       "8    female  27.0                        0                        2   11.1333  \n",
       "9    female  14.0                        1                        0   30.0708  \n",
       "10   female   4.0                        1                        1   16.7000  \n",
       "11   female  58.0                        0                        0   26.5500  \n",
       "12     male  20.0                        0                        0    8.0500  \n",
       "13     male  39.0                        1                        5   31.2750  \n",
       "14   female  14.0                        0                        0    7.8542  \n",
       "15   female  55.0                        0                        0   16.0000  \n",
       "16     male   2.0                        4                        1   29.1250  \n",
       "17     male  23.0                        0                        0   13.0000  \n",
       "18   female  31.0                        1                        0   18.0000  \n",
       "19   female  22.0                        0                        0    7.2250  \n",
       "20     male  35.0                        0                        0   26.0000  \n",
       "21     male  34.0                        0                        0   13.0000  \n",
       "22   female  15.0                        0                        0    8.0292  \n",
       "23     male  28.0                        0                        0   35.5000  \n",
       "24   female   8.0                        3                        1   21.0750  \n",
       "25   female  38.0                        1                        5   31.3875  \n",
       "26     male  26.0                        0                        0    7.2250  \n",
       "27     male  19.0                        3                        2  263.0000  \n",
       "28   female  24.0                        0                        0    7.8792  \n",
       "29     male  23.0                        0                        0    7.8958  \n",
       "..      ...   ...                      ...                      ...       ...  \n",
       "857    male  21.0                        1                        0   11.5000  \n",
       "858  female  48.0                        0                        0   25.9292  \n",
       "859  female  14.0                        8                        2   69.5500  \n",
       "860    male  24.0                        0                        0   13.0000  \n",
       "861  female  42.0                        0                        0   13.0000  \n",
       "862  female  27.0                        1                        0   13.8583  \n",
       "863    male  31.0                        0                        0   50.4958  \n",
       "864    male  23.0                        0                        0    9.5000  \n",
       "865    male   4.0                        1                        1   11.1333  \n",
       "866    male  26.0                        0                        0    7.8958  \n",
       "867  female  47.0                        1                        1   52.5542  \n",
       "868    male  33.0                        0                        0    5.0000  \n",
       "869    male  47.0                        0                        0    9.0000  \n",
       "870  female  28.0                        1                        0   24.0000  \n",
       "871  female  15.0                        0                        0    7.2250  \n",
       "872    male  20.0                        0                        0    9.8458  \n",
       "873    male  19.0                        0                        0    7.8958  \n",
       "874    male  23.0                        0                        0    7.8958  \n",
       "875  female  56.0                        0                        1   83.1583  \n",
       "876  female  25.0                        0                        1   26.0000  \n",
       "877    male  33.0                        0                        0    7.8958  \n",
       "878  female  22.0                        0                        0   10.5167  \n",
       "879    male  28.0                        0                        0   10.5000  \n",
       "880    male  25.0                        0                        0    7.0500  \n",
       "881  female  39.0                        0                        5   29.1250  \n",
       "882    male  27.0                        0                        0   13.0000  \n",
       "883  female  19.0                        0                        0   30.0000  \n",
       "884  female   7.0                        1                        2   23.4500  \n",
       "885    male  26.0                        0                        0   30.0000  \n",
       "886    male  32.0                        0                        0    7.7500  \n",
       "\n",
       "[887 rows x 8 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic_df = pd.read_csv('https://web.stanford.edu/class/archive/cs/cs109/cs109.1166/stuff/titanic.csv')\n",
    "titanic_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-13T12:11:04.253062Z",
     "start_time": "2020-03-13T12:11:04.243432Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div align=\"middle\">\n",
       "    <video width=\"50%\" controls>\n",
       "      <source src=\"https://www.dropbox.com/s/yejuzmw7x0vw14r/6_groupby.mov?raw=1\" type=\"video/mp4\">\n",
       "    </video></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(video_html('https://www.dropbox.com/s/yejuzmw7x0vw14r/6_groupby.mov?raw=1'))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 8**: Use the `groupby` operation to get the *mean* feature values of survivors and non-survivors.\n",
    "* How big is the age difference?\n",
    "* What about the 'Pclass' (low values closer to first class)? \n",
    "* Number of siblings/spouses aboard?\n",
    "* Parents/children aboard?\n",
    "* Ticket price (fare)?\n",
    ">\n",
    "> Do you see any systematic differences between survivors and non-survivors?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**ANS**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:52:25.000658Z",
     "start_time": "2020-03-22T05:52:24.984143Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Siblings/Spouses Aboard</th>\n",
       "      <th>Parents/Children Aboard</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2.528440</td>\n",
       "      <td>30.138532</td>\n",
       "      <td>0.557798</td>\n",
       "      <td>0.332110</td>\n",
       "      <td>22.208584</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.950292</td>\n",
       "      <td>28.408392</td>\n",
       "      <td>0.473684</td>\n",
       "      <td>0.464912</td>\n",
       "      <td>48.395408</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Pclass        Age  Siblings/Spouses Aboard  \\\n",
       "Survived                                                 \n",
       "0         2.528440  30.138532                 0.557798   \n",
       "1         1.950292  28.408392                 0.473684   \n",
       "\n",
       "          Parents/Children Aboard       Fare  \n",
       "Survived                                      \n",
       "0                        0.332110  22.208584  \n",
       "1                        0.464912  48.395408  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic_df.groupby('Survived').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Survivors are slightly younger, boarded in better class, alone or \n",
    "traveling with parent/child, and paid significantly more for their\n",
    "ticket."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Ex. 9**: Try grouping on both 'Survived' and 'Sex', and then `count`ing the number of\n",
    "rows in either group and subgroup. Do you see a pattern in who survives and who does not?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-03-22T05:57:09.141983Z",
     "start_time": "2020-03-22T05:57:09.125247Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Pclass</th>\n",
       "      <th>Age</th>\n",
       "      <th>Siblings/Spouses Aboard</th>\n",
       "      <th>Parents/Children Aboard</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Survived</th>\n",
       "      <th>Sex</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">0</th>\n",
       "      <th>female</th>\n",
       "      <td>2.851852</td>\n",
       "      <td>24.419753</td>\n",
       "      <td>1.209877</td>\n",
       "      <td>1.037037</td>\n",
       "      <td>23.024385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>2.471983</td>\n",
       "      <td>31.136853</td>\n",
       "      <td>0.443966</td>\n",
       "      <td>0.209052</td>\n",
       "      <td>22.066170</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>female</th>\n",
       "      <td>1.918455</td>\n",
       "      <td>28.866953</td>\n",
       "      <td>0.515021</td>\n",
       "      <td>0.515021</td>\n",
       "      <td>51.938573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>male</th>\n",
       "      <td>2.018349</td>\n",
       "      <td>27.428165</td>\n",
       "      <td>0.385321</td>\n",
       "      <td>0.357798</td>\n",
       "      <td>40.821484</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Pclass        Age  Siblings/Spouses Aboard  \\\n",
       "Survived Sex                                                    \n",
       "0        female  2.851852  24.419753                 1.209877   \n",
       "         male    2.471983  31.136853                 0.443966   \n",
       "1        female  1.918455  28.866953                 0.515021   \n",
       "         male    2.018349  27.428165                 0.385321   \n",
       "\n",
       "                 Parents/Children Aboard       Fare  \n",
       "Survived Sex                                         \n",
       "0        female                 1.037037  23.024385  \n",
       "         male                   0.209052  22.066170  \n",
       "1        female                 0.515021  51.938573  \n",
       "         male                   0.357798  40.821484  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "titanic_df.groupby(['Survived', 'Sex']).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nuancing the picture by also grouping on sex reveals most strikingly\n",
    "that among survivors, females were much better classed and pair higher\n",
    "fare prices. In other words, it appears that as a woman your chances \n",
    "of surviving were to a much higher extend dictaged by the price you\n",
    "paid for your ticked. Surviving females are also older and less likely\n",
    "to have sibling or spouses on board, indicating that even as a female\n",
    "it's better to be less attached to family during the sink."
   ]
  }
 ],
 "metadata": {
  "display_name": "Python 3",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}