1_opening_files.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Opening files with python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Author:** Ties de Kok ([Personal Website](http://www.tiesdekok.com))  \n",
    "**Last updated:** 18 May 2018  \n",
    "**Python version:** Python 3.6  \n",
    "**License:** MIT License  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** Some features (like the ToC) will only work if you run it locally, use Binder, or use nbviewer by clicking this link: \n",
    "https://nbviewer.jupyter.org/github/TiesdeKok/LearnPythonforResearch/blob/master/1_opening_files.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Introduction*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Python you can open and save a wide variety of files.  \n",
    "There are often multiple ways to open a particular file format, the examples below are in my experience the most convenient\n",
    "\n",
    "**Note:** All the sample files are in the `example_data` folder. The code to generate these files is at the end of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Table of Contents* <a id='toc'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [Indexing a folder](#indexing)   \n",
    "* [Text files](#text-files)\n",
    "* [Excel files](#excel-files)   \n",
    "* [CSV files](#csv-files) \n",
    "* [Stata files](#stata-files)   \n",
    "* [SAS files](#sas-files) \n",
    "* [JSON files](#json-files)   \n",
    "* [HDF files: Pandas](#hdf-pandas-files)   \n",
    "* [HDF files: General Python objects](#hdf-general-files)\n",
    "* [Code to generate examples files](#example-files) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from glob import glob\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The below is for convenience so we can type `join` instead of `os.path.join`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from os.path import join"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Indexing a folder</span><a id='indexing'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Irrespective of the type of file you are trying to open it is useful to be able to index all files in a folder.  \n",
    "This is necessary if you, for example, want to loop over all files in a folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are multiple ways to go about this, but I will show `os.listdir`, `glob`, and `os.walk`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First define the path to the folder that we want to index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our examples files are in the `example_data` folder so we can set the directory as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all the files in the root of the folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note:* this will ignore files in sub-folders!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['excel_sample.xlsx', 'csv_sample.csv']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filenames = os.listdir(data_path)\n",
    "filenames[:2] # show first two items"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\example_data\\\\excel_sample.xlsx',\n",
       " 'G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\example_data\\\\csv_sample.csv']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepaths = [join(data_path, filename) for filename in filenames]\n",
    "filepaths[:2] # show first two items"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can alternatively use `glob` as this directly allows to include pathname matching.  \n",
    "For example if we only want Excel `.xlsx` files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\example_data\\\\excel_sample.xlsx']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "glob(join(data_path, '*.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get all files, also those in sub-folders:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the folder contains multiple levels we need to either use `os.walk()` or `glob`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\README.md',\n",
       " 'G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\environment.yml']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "folder = os.getcwd()\n",
    "filepaths = []\n",
    "for root,dirs,files in os.walk(folder):\n",
    "    for i in files:\n",
    "        filepaths.append(join(root,i))\n",
    "filepaths[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Personally, using `glob` yields cleaner code although it is a bit harder to understand:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\README.md',\n",
       " 'G:\\\\My Drive\\\\Work\\\\Programming\\\\Active\\\\PythonAccountingResearch\\\\environment.yml']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filepaths_glob = glob(join(folder, '**/*'), recursive=True)\n",
    "filepaths_glob[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Text files</span><a id='text-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Opening text files is done using the default Python library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open a file with different file modes:  \n",
    "w -> write only  \n",
    "r -> read only  \n",
    "w+ -> read and write + completely overwrite file  \n",
    "a+ -> read and write + append at the bottom  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Opening a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(join(data_path, 'text_sample.txt'), 'r') as file:\n",
    "    file_content = file.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning Python is great. \n",
      "Good luck!\n"
     ]
    }
   ],
   "source": [
    "print(file_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to a file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(join(data_path, 'text_sample.txt'), 'w+') as file:\n",
    "    file.write('Learning Python is great. \\nGood luck!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that I am using a `with` statement when opening files.  \n",
    "Another method is to use `open` and `close`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open(join(data_path, 'text_sample.txt'), 'r')\n",
    "file_content = f.read()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `with` method is preferred as it automatically closes the file.  \n",
    "This prevents the file from being 'in use' if you forget to use `.close()`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Looping over indexed files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_files = glob(join(data_path, '*.txt'))\n",
    "text_list = []\n",
    "\n",
    "for i in text_files:\n",
    "    with open(i, 'r') as f:\n",
    "        text_list.append(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Learning Python is great. \\nGood luck!']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Excel files</span><a id='excel-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open `Excel`, `csv`, `Stata`, `SAS` files in multiple ways, I like to use `Pandas` as it is the most convenient. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "excel_file = pd.read_excel(join(data_path, 'excel_sample.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function has a lot of options, see:  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html  \n",
    "\n",
    "*Note:* You often want to specify the encoding to prevent errors, for example: `, encoding='utf-8'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Excel file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "excel_file.to_excel(join(data_path, 'excel_sample.xlsx'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This saves a `Pandas` dataframe object, see the data handling file.  \n",
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_excel.html  \n",
    "\n",
    "*Note:* You can save as `.xls` but also `.xlsx`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">CSV files</span><a id='csv-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can open `Excel`, `csv`, `Stata`, `SAS` files in multiple ways, I like to use `Pandas` as it is the most convenient. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "csv_file = pd.read_csv(join(data_path, 'csv_sample.csv'), sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "csv_file.to_csv(join(data_path, 'csv_sample.csv'), sep=',')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_csv.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">Stata files</span><a id='stata-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Open Stata file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stata_file = pd.read_stata(join(data_path, 'stata_sample.dta'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_stata.html  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save Stata file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stata_file.to_stata(join(data_path, 'stata_sample.dta'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_stata.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Note: make sure you have the latest version of Pandas for new Stata versions*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">SAS files</span><a id='sas-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can only read SAS files but cannot write them:  \n",
    "\n",
    "```\n",
    "sas_file = pd.read_sas(r'C:\\file.sas7bdat', format='sas7bdat')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sas.html  \n",
    "This function works in most cases but files with text are likely to throw hard to fix encoding errors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">JSON files</span><a id='json-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files using pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_json.html  \n",
    "\n",
    "*Note:* The path can also be a url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Read JSON file to dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "json_df = pd.read_json(join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save dataframe to JSON file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "json_df.to_json(join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### JSON files using the `JSON` module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Read JSON:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(join(data_path, 'json_sample.json'), 'r') as f:\n",
    "    json_data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Write JSON:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(join(data_path, 'json_sample.json'), 'w') as f:\n",
    "    json.dump(json_data, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">HDF files for Pandas</span><a id='hdf-pandas-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You often run into the problem of having to store large amounts of data.  \n",
    "The traditional formats such as .csv are not very efficient as big-data file formats.  \n",
    "\n",
    "I like to use the `Hierarchical Data Format` or `HDF` in short.\n",
    "This `.hdf` file format is designed to store and organize large amounts of data. \n",
    "\n",
    "Writing and reading `.hdf` files is extremely fast compared to `.csv`:\n",
    "\n",
    "**Writing:**\n",
    "\n",
    "```\n",
    "%timeit test_hdf_fixed_write(df)\n",
    "1 loops, best of 3: 237 ms per loop\n",
    "\n",
    "%timeit test_hdf_table_write(df)\n",
    "1 loops, best of 3: 901 ms per loop\n",
    "\n",
    "%timeit test_csv_write(df)\n",
    "1 loops, best of 3: 3.44 s per loop\n",
    "```\n",
    "\n",
    "**Reading:**\n",
    "\n",
    "```\n",
    "%timeit test_hdf_fixed_read()\n",
    "10 loops, best of 3: 19.1 ms per loop\n",
    "\n",
    "%timeit test_hdf_table_read()\n",
    "10 loops, best of 3: 39 ms per loop\n",
    "\n",
    "%timeit test_csv_read()\n",
    "1 loops, best of 3: 620 ms per loop\n",
    "```\n",
    "\n",
    "The downside of `HDF` is that the file sizes tend to be larger. However, storage space is cheap, working memory isn't."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read HDF files using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_df = pd.read_hdf(join(data_path, 'hdf_sample.h5'), 'hdf_sample') # Second argument is the key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_hdf.html  \n",
    "\n",
    "*Note*: You can give it any `key` you like. I recommend to use the filename without `.h5` as `key`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write HDF files using Pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hdf_df.to_hdf(join(data_path, 'hdf_sample.h5'), 'hdf_sample')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.to_hdf.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <span style=\"text-decoration: underline;\">HDF files for general Python objects</span><a id='hdf-general-files'></a> [(to top)](#toc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above works great for Python dataframes, but sometimes you want to save general Python objects to HDF files as well.  \n",
    "Personally, I use a package called `deepdish`:  \n",
    "http://deepdish.readthedocs.io/en/latest/api_io.html\n",
    "\n",
    "It is possible to install this by running `pip install deepdish` in the command line. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import deepdish as dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e' : 5}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save data using `deepdish`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "dd.io.save(join(data_path, 'dd_example.h5'), test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load data using `deepdish`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = dd.io.load(join(data_path, 'dd_example.h5'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <span style=\"text-decoration: underline;\">Code used to generate example files</span><a id='example-files'></a> [(to top)](#toc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionary with random data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "raw_data = {'foreign':{1:'Domestic',2:'Domestic',3:'Domestic',6:'Domestic',7:'Domestic',8:'Domestic',9:'Domestic',14:'Domestic',21:'Domestic',23:'Domestic',24:'Domestic',30:'Domestic',31:'Domestic',33:'Domestic',37:'Domestic',38:'Domestic',43:'Domestic',48:'Domestic',50:'Domestic',51:'Domestic',53:'Foreign',56:'Foreign',57:'Foreign',66:'Foreign',70:'Foreign'},\n",
    "            'make':{1:'AMCPacer',2:'AMCSpirit',3:'BuickCentury',6:'BuickOpel',7:'BuickRegal',8:'BuickRiviera',9:'BuickSkylark',14:'Chev.Impala',21:'DodgeMagnum',23:'FordFiesta',24:'FordMustang',30:'Merc.Marquis',31:'Merc.Monarch',33:'Merc.Zephyr',37:'OldsDelta88',38:'OldsOmega',43:'Plym.Horizon',48:'Pont.GrandPrix',50:'Pont.Phoenix',51:'Pont.Sunbird',53:'AudiFox',56:'Datsun210',57:'Datsun510',66:'ToyotaCelica',70:'VWDiesel'},\n",
    "            'price': {1:4749,2:3799,3:4816,6:4453,7:5189,8:10372,9:4082,14:5705,21:5886,23:4389,24:4187,30:6165,31:4516,33:3291,37:4890,38:4181,43:4482,48:5222,50:4424,51:4172,53:6295,56:4589,57:5079,66:5899,70:5397},\n",
    "            'weight':{1:3350,2:2640,3:3250,6:2230,7:3280,8:3880,9:3400,14:3690,21:3600,23:1800,24:2650,30:3720,31:3370,33:2830,37:3690,38:3370,43:2200,48:3210,50:3420,51:2690,53:2070,56:2020,57:2280,66:2410,70:2040}}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert dictionary to Pandas dataframe for easy saving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>foreign</th>\n",
       "      <th>make</th>\n",
       "      <th>price</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>AMCPacer</td>\n",
       "      <td>4749</td>\n",
       "      <td>3350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>AMCSpirit</td>\n",
       "      <td>3799</td>\n",
       "      <td>2640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickCentury</td>\n",
       "      <td>4816</td>\n",
       "      <td>3250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickOpel</td>\n",
       "      <td>4453</td>\n",
       "      <td>2230</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Domestic</td>\n",
       "      <td>BuickRegal</td>\n",
       "      <td>5189</td>\n",
       "      <td>3280</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "\n",
       "    foreign          make  price  weight\n",
       "1  Domestic      AMCPacer   4749    3350\n",
       "2  Domestic     AMCSpirit   3799    2640\n",
       "3  Domestic  BuickCentury   4816    3250\n",
       "6  Domestic     BuickOpel   4453    2230\n",
       "7  Domestic    BuickRegal   5189    3280"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_data = pd.DataFrame(raw_data)\n",
    "df_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the different files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_path = os.path.join(os.getcwd(), 'example_data')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(os.path.join(data_path, 'text_sample.txt'), 'w+') as file:\n",
    "    file.write('Learning Python is great. \\nGood luck!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_data.to_excel(os.path.join(data_path, 'excel_sample.xlsx'))\n",
    "df_data.to_csv(os.path.join(data_path, 'csv_sample.csv'))\n",
    "df_data.to_stata(os.path.join(data_path, 'stata_sample.dta'))\n",
    "df_data.to_hdf(os.path.join(data_path, 'hdf_sample.h5'), 'hdf_sample')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_data.to_json(os.path.join(data_path, 'json_sample.json'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** pandas does not have a `.to_sas()` function"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}