getting-started-spatial-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started with Spatial Data Science in Python\n",
    "\n",
    "Bryan C. Runck // runck014@umn.edu // Department of Geography, Environment and Society\n",
    "\n",
    "**Overview**\n",
    "How can we use python to do spatial data science? This jam session will provide a hands-on overview of basic mapping in Python with GeoPandas and how to perform basic spatial analysis using PySAL. No programming experience is required.\n",
    "\n",
    "## Objectives\n",
    "1. Make simple maps with [GeoPandas](http://geopandas.org) and AirBnB data\n",
    "    - Data I/O\n",
    "    - Make chloropleth maps\n",
    "    - Make scatterplots\n",
    "    - Rate mapping\n",
    "    - Recognize the importance of projections\n",
    "2. Perform an exploratory visual analysis of the data to identify potential places you would want to hone an AirBnB stay\n",
    "3. Use [PySAL](http://pysal.readthedocs.io/en/latest/) to compute global spatial autocorrelation \n",
    "    - Constructing spatial weights\n",
    "    - Moran's I (Global)\n",
    "    - Visually check result\n",
    "4. Use Moran's I to determine which AirBnB variables have high levels of spatial autocorrelation\n",
    "    \n",
    "\n",
    "\n",
    "# Table of Contents\n",
    "\n",
    "1. [Motivation for spatial data science](https://docs.google.com/presentation/d/1_RuL1EHp7sOn5yLnCuBqRWW8eGo-Z8YemyEzjS2KpXU/edit?usp=sharing) (link to slides)\n",
    "2. [Getting spatial data](#get_data)\n",
    "3. [Data exploration](#esda)\n",
    "4. [Basic spatial analysis](#bda)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='get_data'></a>\n",
    "# Getting Spatial Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "#wget is a linux tool; Jupyter allows access through ! character; https://www.gnu.org/software/wget/\n",
    "Download a file from Minnesota Geospatial Commons\n",
    "'''\n",
    "#!wget ftp://ftp.gisdata.mn.gov/pub/gdrs/data/pub/us_mn_state_dnr/bdry_dnr_wildlife_mgmt_areas_pub/shp_bdry_dnr_wildlife_mgmt_areas_pub.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "linux tool to unzip files; may need to install on system\n",
    "'''\n",
    "#!unzip shp_bdry_dnr_wildlife_mgmt_areas_pub.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "'''\n",
    "linux command; \n",
    "ls lists all of the file, -l is a formatting designator; \n",
    "*.shp tells linux, list all of the files in the current directory that end in\n",
    ".shp <- is a common spatial data format; comes with 3 other files\n",
    "https://en.wikipedia.org/wiki/Shapefile#Shapefile_shape_index_format_(.shx)\n",
    "'''\n",
    "#!ls -l *.shp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='esda'></a>\n",
    "# Spatial Data Exploration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import geopandas as gpd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Data\n",
    "\n",
    "AirBnB's in Chicago - the data set comes from Luc Anselin's spatial data science group in at the University of Chicago. \n",
    "\n",
    "**Metadata can be found [here](https://geodacenter.github.io/data-and-lab//airbnb_Chicago-2015/).***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb = gpd.read_file('data/airbnb_Chicago 2015.shp')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check import to make sure it looks OK\n",
    "chicago_bnb.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb.crs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "^^ crs = coordinate reference system; it makes all of the spatial stuff work correctly; see [wikipedia](https://en.wikipedia.org/wiki/Spatial_reference_system) for a nice overview; EPSG = European Petroleum Survey Group, and is a prominent spatial reference system identifier; 4326 is the code for WGS84 a global datum (e.g. how points in GIS are connected to real places on the earth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summarizing and Basic Plotting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# describe feature works just as it does in pandas; \n",
    "chicago_bnb.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline \n",
    "#tells matplotlib to print to Jupyter\n",
    "\n",
    "chicago_bnb.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(chicago_bnb['community'][0])\n",
    "chicago_bnb['geometry'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb['geometry'][0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb['geometry'][0:10].plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chloropleth Map\n",
    "The histogram of maps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# columns that could be mapped\n",
    "print(chicago_bnb.columns.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb.plot(column='price_pp', cmap='magma', legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scatterplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.scatter(x=chicago_bnb['rev_rating'], y=chicago_bnb['response_r'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing Multiple Scatteplots with Seaborn\n",
    "The goal is to identify interesting relationships that could guide exploratory mapping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(chicago_bnb.columns.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(chicago_bnb[['response_r','accept_r', 'price_pp']].dropna())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Rate Mapping\n",
    "\n",
    "Raw numbers are incomparable across jurisdictions; to say that there were 100 homicides in one US county and 10 in another makes it seem like there were a lot in county with 100, until this is converting into a rate by placing total population in the denomentator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_thefts_map = chicago_bnb.plot(column='num_theft', cmap='magma', legend=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb['thefts_per_capita'] = chicago_bnb['num_theft']/chicago_bnb['population']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "per_person_thefts_map = chicago_bnb.plot(column='thefts_per_capita', cmap='magma', legend=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb['crimes_per_capita'] = chicago_bnb['num_crimes']/chicago_bnb['population']\n",
    "num_crimes_map = chicago_bnb.plot(column='crimes_per_capita', cmap='magma', legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity: Identify Three Potential Places in Chicago Where You Would Want to Stay\n",
    "You and a friend are planning to head to Chicago on a budget. You want to identify the top three communities to look for an AirBnB in. You’ve been provided with a dataset to aid in you in your decision-making.\n",
    "\n",
    "Utilize the basic ideas we explored related to mapping to:\n",
    "1. Identify three potential communities where you would want to stay\n",
    "2. Make a map with these three communities highlighted\n",
    "3. **Challenge:** create a linear combination of variables to create an index score of where you would want to stay. For example, the value of community to you, $v(community)$, could be modeled as:\n",
    "\n",
    "$ v(community) = weight_1 * norm(price_{pp}) + weight_2 * norm(accept_r) + ... + weight_n * norm(variable_n)$\n",
    "\n",
    "Variables would need to be normalized, and subjective weights can be assigned based on what you personally value.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='bda'></a>\n",
    "# Basic Spatial Data Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pysal\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing spatial weights\n",
    "\n",
    "There are three different types of spatial weights:\n",
    "1. Contiguity Based Weights\n",
    "2. Distance Based Weights\n",
    "3. Kernel Weights\n",
    "\n",
    "This demonstration only performs contiguity-based weights, namely queen and rook. Here is a [link](http://pysal.readthedocs.io/en/latest/users/tutorials/weights.html#pysal-spatial-weight-types) to the documentation with many more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb = chicago_bnb.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_weights = pysal.weights.Rook.from_dataframe(chicago_bnb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"%.4f\"%chicago_weights.pct_nonzero)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_Qweights = pysal.weights.Queen.from_dataframe(chicago_bnb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"%.4f\"%chicago_Qweights.pct_nonzero)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_Qweights.weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(chicago_Qweights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Moran's I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y=chicago_bnb['price_pp']\n",
    "w=chicago_Qweights\n",
    "chicago_Qweights_moran = pysal.Moran(y, w, two_tailed=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"%.3f\"%chicago_Qweights_moran.I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"%.8f\"%chicago_Qweights_moran.p_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chicago_bnb.plot(column='price_pp', legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Activity\n",
    "Utilize the basic ideas we explored related to spatial autocorrelation to:\n",
    "1. Test spatial autocorrelation across multiple variables and weights\n",
    "2. Which variable is the most spatially autocorrelated?\n",
    "3. Do you have any hunches as to why there is or isn’t spatial autocorrelation in different variables?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}