DM_05_04.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DM_05_04"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import pylab\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.svm import OneClassSVM\n",
    "from sklearn.covariance import EllipticEnvelope\n",
    "\n",
    "pylab.rcParams.update({'font.size': 14})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"AnomalyData.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save state_code to label outliers. \"data\" contains just quantitative variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "state_code = df[\"state_code\"]\n",
    "data = df.loc[:, \"data science\": \"Openness\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Univariate Outliers \n",
    "Create a box plot to display univariate outliers on \"modern dance.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "param = \"modern dance\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get quantile values and IQR for outlier limits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "qv1 = data[param].quantile(0.25)\n",
    "qv2 = data[param].quantile(0.5)\n",
    "qv3 = data[param].quantile(0.75)\n",
    "qv_limit = 1.5 * (qv3 - qv1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get positions of outliers and use state_code for labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "un_outliers_mask = (data[param] > qv3 + qv_limit) | (data[param] < qv1 - qv_limit)\n",
    "un_outliers_data = data[param][un_outliers_mask]\n",
    "un_outliers_name = state_code[un_outliers_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create box plot for \"modern dance.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = pylab.figure(figsize=(4,6))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "for name, y in zip(un_outliers_name, un_outliers_data):\n",
    "    ax.text(1, y, name)\n",
    "ax.boxplot(data[param])\n",
    "ax.set_ylabel(param)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Bivariate Outliers\n",
    "Create a scatterplot with an ellipse as a boundary for outliers.\n",
    "\n",
    "Use the Google search terms \"data science\" and \"ceo\" for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = [\"data science\", \"ceo\"]\n",
    "params_data = np.array([df[params[0]], df[params[1]]]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the \"elliptical envelope.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ee = EllipticEnvelope()\n",
    "ee.fit(params_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the names and positions of outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "biv_outliers_mask = ee.predict(params_data) == -1\n",
    "biv_outliers_data = params_data[biv_outliers_mask]\n",
    "biv_outliers_name = state_code[biv_outliers_mask]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate the decision boundary for the scatterplot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "xx, yy = np.meshgrid(np.linspace(params_data[:, 0].min(), params_data[:, 0].max(), 100),\n",
    "                     np.linspace(params_data[:, 1].min(), params_data[:, 1].max(), 100))\n",
    "zz = ee.decision_function(np.c_[xx.ravel(), yy.ravel()])\n",
    "zz = zz.reshape(xx.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Draw the scatterplot with the elliptical envelope and label the outliers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fig = pylab.figure(figsize=(10,10))\n",
    "ax = fig.add_subplot(1, 1, 1)\n",
    "for name, xy in zip(biv_outliers_name, biv_outliers_data):\n",
    "    ax.text(xy[0], xy[1], name)\n",
    "ax.contour(xx, yy, zz, levels=[0], linewidths=2)\n",
    "ax.scatter(params_data[:, 0], params_data[:, 1], color='black')\n",
    "ax.set_xlabel(params[0])\n",
    "ax.set_ylabel(params[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Multivariate Outliers\n",
    "Use the one-class support vector machine (SVM) algorithm to classify unusual cases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ocsvm = OneClassSVM(nu=0.25, gamma=0.05)\n",
    "ocsvm.fit(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "List the names of the outlying states based on the one-class SVM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# \n",
    "state_code[ocsvm.predict(data) == -1]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}