data-types-and-missing-values.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[Pandas Micro-Course Home Page](https://www.kaggle.com/learn/pandas)**\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data types and missing data reference\n",
    "\n",
    "This is the reference section of the \"Data types and missing data\" section of the tutorial.\n",
    "\n",
    "In this short section we will look at two inter-related concepts, data types and missing data. This section draws from the [Intro to data structures](https://pandas.pydata.org/pandas-docs/stable/dsintro.html) and [Working with missing data](https://pandas.pydata.org/pandas-docs/stable/missing_data.html) sections of the comprehensive official tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "reviews = pd.read_csv(\"../input/wine-reviews/winemag-data-130k-v2.csv\", index_col=0)\n",
    "pd.set_option('max_rows', 5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data types\n",
    "\n",
    "The data type for a column in a `DataFrame` or a `Series` is known as the `dtype`.\n",
    "\n",
    "You can use the `dtype` property to grab the type of a specific column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.price.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, the `dtypes` property returns the `dtype` of _every_ column in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country        object\n",
       "description    object\n",
       "                ...  \n",
       "variety        object\n",
       "winery         object\n",
       "Length: 13, dtype: object"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data types tell us something about how `pandas` is storing the data internally. `float64` means that it's using a 64-bit floating point number; `int64` means a similarly sized integer instead, and so on.\n",
    "\n",
    "One peculiarity to keep in mind (and on display very clearly here) is that columns consisting entirely of strings do not get their own type; they are instead given the `object` type.\n",
    "\n",
    "It's possible to convert a column of one type into another wherever such a conversion makes sense by using the `astype` function. For example, we may transform the `points` column from its existing `int64` data type into a `float64` data type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         87.0\n",
       "1         87.0\n",
       "          ... \n",
       "129969    90.0\n",
       "129970    90.0\n",
       "Name: points, Length: 129971, dtype: float64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.points.astype('float64')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A `DataFrame` or `Series` index has its own `dtype`, too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('int64')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.index.dtype"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pandas` also supports more exotic data types: categorical data and timeseries data. Because these data types are more rarely used, we will omit them until a much later section of this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing data\n",
    "\n",
    "Entries missing values are given the value `NaN`, short for \"Not a Number\". For technical reasons these `NaN` values are always of the `float64` dtype.\n",
    "\n",
    "`pandas` provides some methods specific to missing data. To select `NaN` entreis you can use `pd.isnull` (or its companion `pd.notnull`). This is meant to be used thusly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>description</th>\n",
       "      <th>designation</th>\n",
       "      <th>points</th>\n",
       "      <th>price</th>\n",
       "      <th>province</th>\n",
       "      <th>region_1</th>\n",
       "      <th>region_2</th>\n",
       "      <th>taster_name</th>\n",
       "      <th>taster_twitter_handle</th>\n",
       "      <th>title</th>\n",
       "      <th>variety</th>\n",
       "      <th>winery</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>913</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Amber in color, this wine has aromas of peach ...</td>\n",
       "      <td>Asureti Valley</td>\n",
       "      <td>87</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mike DeSimone</td>\n",
       "      <td>@worldwineguys</td>\n",
       "      <td>Gotsa Family Wines 2014 Asureti Valley Chinuri</td>\n",
       "      <td>Chinuri</td>\n",
       "      <td>Gotsa Family Wines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3131</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Soft, fruity and juicy, this is a pleasant, si...</td>\n",
       "      <td>Partager</td>\n",
       "      <td>83</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Roger Voss</td>\n",
       "      <td>@vossroger</td>\n",
       "      <td>Barton &amp; Guestier NV Partager Red</td>\n",
       "      <td>Red Blend</td>\n",
       "      <td>Barton &amp; Guestier</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129590</th>\n",
       "      <td>NaN</td>\n",
       "      <td>A blend of 60% Syrah, 30% Cabernet Sauvignon a...</td>\n",
       "      <td>Shah</td>\n",
       "      <td>90</td>\n",
       "      <td>30.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mike DeSimone</td>\n",
       "      <td>@worldwineguys</td>\n",
       "      <td>Büyülübağ 2012 Shah Red</td>\n",
       "      <td>Red Blend</td>\n",
       "      <td>Büyülübağ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>129900</th>\n",
       "      <td>NaN</td>\n",
       "      <td>This wine offers a delightful bouquet of black...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>91</td>\n",
       "      <td>32.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Mike DeSimone</td>\n",
       "      <td>@worldwineguys</td>\n",
       "      <td>Psagot 2014 Merlot</td>\n",
       "      <td>Merlot</td>\n",
       "      <td>Psagot</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>63 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       country         ...                      winery\n",
       "913        NaN         ...          Gotsa Family Wines\n",
       "3131       NaN         ...           Barton & Guestier\n",
       "...        ...         ...                         ...\n",
       "129590     NaN         ...                   Büyülübağ\n",
       "129900     NaN         ...                      Psagot\n",
       "\n",
       "[63 rows x 13 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews[reviews.country.isnull()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replacing missing values is a common operation.  `pandas` provides a really handy method for this problem: `fillna`. `fillna` provides a few different strategies for mitigating such data. For example, we can simply replace each `NaN` with an `\"Unknown\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0         Unknown\n",
       "1         Unknown\n",
       "           ...   \n",
       "129969    Unknown\n",
       "129970    Unknown\n",
       "Name: region_2, Length: 129971, dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.region_2.fillna(\"Unknown\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we could fill each missing value with the first non-null value that appears sometime after the given record in the database. This is known as the backfill strategy:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fillna` supports a few strategies for imputing missing values. For more on that read [the official function documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.fillna.html).\n",
    "\n",
    "Alternatively, we may have a non-null value that we would like to replace. For example, suppose that since this dataset was published, reviewer Kerin O'Keefe has changed her Twitter handle from `@kerinokeefe` to `@kerino`. One way to reflect this in the dataset is using the `replace` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0            @kerino\n",
       "1         @vossroger\n",
       "             ...    \n",
       "129969    @vossroger\n",
       "129970    @vossroger\n",
       "Name: taster_twitter_handle, Length: 129971, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.taster_twitter_handle.replace(\"@kerinokeefe\", \"@kerino\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `replace` method is worth mentioning here because it's handy for replacing missing data which is given some kind of sentinel value in the dataset: things like `\"Unknown\"`, `\"Undisclosed\"`, `\"Invalid\"`, and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "**[Pandas Micro-Course Home Page](https://www.kaggle.com/learn/pandas)**\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}