03_how-medgan-count-works.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <a name=\"top\"></a>\n",
    "# Understanding how medGAN works (with count features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: [Sylvain Combettes](https://github.com/sylvaincom).\n",
    "\n",
    "Edward Choi's original repository: [medgan](https://github.com/mp2893/medgan). <br/>\n",
    "My own medGAN repository (that is based on Edward Choi's work): [medgan](https://github.com/sylvaincom/medgan-tips).\n",
    "\n",
    "The final goal of my project is to use medGAN on my own dataset (electronic health records). For that, I first need to understand how medGAN works. In this notebook, I provide a few code cells and explanations that can help better understand and run medGAN. Because there are some confidentiality issues with the MIMIC-III dataset, I cleared the output of the cells.\n",
    "\n",
    "Before reading this notebook, be sure to have read [A few additional tips on how to run Edward Choi's medGAN](https://github.com/sylvaincom/medgan-tips/blob/master/01_tips-for-medgan.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Tables of Contents\n",
    "\n",
    "- [Loading the MIMIC-III dataset](#load-mimic)\n",
    "- [Using process_`mimic.py` and `medgan.py` to generate the fake realistic data](#run)\n",
    "- [How to interpret `gen-samples.npy`?](#gen-samples)\n",
    "- [Comparing the (fake) generated samples to the real-life original ones](#comparison)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import pickle\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Loading the MIMIC-III dataset <a name=\"load-mimic\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ADMISSIONS.csv` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_adm = pd.read_csv(\"ADMISSIONS.csv\")\n",
    "print(df_adm.shape)\n",
    "df_adm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do we have a lot of missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n,p = df_adm.shape\n",
    "for f in df_adm:\n",
    "    percentage = sum(df_adm[f].isna())*100/n\n",
    "    if percentage>0:\n",
    "        print('Missing values in {}: {}%'.format(f, percentage))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `DIAGNOSES_ICD.csv` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ICD = pd.read_csv('DIAGNOSES_ICD.csv')\n",
    "print(df_ICD.shape)\n",
    "df_ICD.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do we have a lot of missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n,p = df_ICD.shape\n",
    "for f in df_ICD:\n",
    "    percentage = sum(df_ICD[f].isna())*100/n\n",
    "    if percentage>0:\n",
    "        print('Missing values in {}: {}%'.format(f, percentage))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check if our dataset is balanced. Does one `ICD9_CODE` appear distinctly more than others in proportion?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ICD['ICD9_CODE'].value_counts(normalize=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Using `process_mimic.py` and `medgan.py` to generate the fake realistic data <a name=\"run\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step is detailed in [A few additional tips on how to run Edward Choi's medGAN\n",
    "](https://github.com/sylvaincom/medgan/blob/master/tips-for-medgan.md).\n",
    "\n",
    "In short, in the Anaconda prompt, we run:\n",
    "```\n",
    "cd C:\\Users\\<username>\\Documents\\mimic_count\n",
    "python process_mimic.py ADMISSIONS.csv DIAGNOSES_ICD.csv training-data \"count\"\n",
    "mkdir generated\n",
    "python medgan.py training-data.matrix ./generated/samples --data_type=\"count\"\n",
    "python medgan.py training-data.matrix gen-samples --model_file=./generated/samples-999 --generate_data=True --data_type=\"count\"\n",
    "```\n",
    "Some default values are `n_epoch=1000` and `n_pretrain_epoch=100`. The computing time took more than 5 hours for me."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From now on, whenever we refer to input or output, we refer to the input and output of `medgan.py` (unless specified otherwise)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# How to interpret `gen-samples.npy`? <a name=\"gen-samples\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load the `gen-samples.npy` file which is `medgan.py`'s output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output = np.load('gen-samples.npy')\n",
    "df_output = pd.DataFrame(output)\n",
    "print(df_output.shape)\n",
    "df_output.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the range of the values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(df_output.max(axis=0).tolist(), 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Maximum')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some questions about this data frame:\n",
    "* What do the columns correspond to? They do not look like `ADMISSIONS.csv` nor `DIAGNOSIS_ICD.csv`.\n",
    "* What do the rows correspond to?\n",
    "* Why are is the maximum value of each column inferior to 1?\n",
    "* Why are the values not count values?\n",
    "\n",
    "We can find some answers in an issue opened in Edward Choi's GitHub: [How to interpret the samples?](https://github.com/mp2893/medgan/issues/3). In order to understand the output `gen-samples.npy` of `medgan.py`, we are going to back to the input of `medgan.py`: the output of `process_mimic.py`.\n",
    "\n",
    "Actually, as in the `.matrix` file, each row corresponds to a single synthetic patient and each column corresponds to a specific ICD9 diagnosis code. We can use the `.types` file created by `process_mimic.py` to map each column to a specific ICD9 diagnosis code. Read the beginning part of the source code of `process_mimic.py` for more information about `.types` file:\n",
    "```python\n",
    "# Output files\n",
    "# <output file>.pids: cPickled Python list of unique Patient IDs. Used for intermediate processing\n",
    "# <output file>.matrix: Numpy float32 matrix. Each row corresponds to a patient. Each column corresponds to a ICD9 diagnosis code.\n",
    "# <output file>.types: cPickled Python dictionary that maps string diagnosis codes to integer diagnosis codes.\n",
    "```\n",
    "\n",
    "What is ICD-9? See [ICD-9](https://en.wikipedia.org/wiki/International_Statistical_Classification_of_Diseases_and_Related_Health_Problems#ICD-9) and [List of ICD-9 codes](https://en.wikipedia.org/wiki/List_of_ICD-9_codes)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to round the values ourselves:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_output = df_output.round(0)\n",
    "df_output.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We claim that we should delete the rows with missing values (if there are any):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_output = pd.DataFrame.dropna(df_output)\n",
    "print(df_output.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, in line 406 of `medgan.py`, it is written `nSamples=10000`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the `.types` file (an ouput of `process_mimic.py`)\n",
    "\n",
    "_cPickled Python dictionary that maps string diagnosis codes to integer diagnosis codes._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_dict = pickle.load(open('training-data.types', 'rb'))\n",
    "print(type(map_dict))\n",
    "print('An excerpt is:', dict(list(map_dict.items())[0:5]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, as its name suggests, `process_mimic.py` is really dependent on the MIMIC-III dataset. We probably will not use `process_mimic.py` on our own dataset and only run `medgan.py`. Out of `process_mimic.py`, we only need to understand how the generated `.matrix` file is constructed (lines 109 to 119)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the `.pids` file (an ouput of `process_mimic.py`)\n",
    "\n",
    "_cPickled Python list of unique Patient IDs. Used for intermediate processing_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_list = pickle.load(open('training-data.pids', 'rb'))\n",
    "print(type(id_list))\n",
    "print('An excerpt is:', id_list[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the `.matrix` file (an ouput of `process_mimic.py` and the input of `medgan.py`)\n",
    "\n",
    "_Numpy float32 matrix. Each row corresponds to a patient. Each column corresponds to a ICD9 diagnosis code._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_data_array = pickle.load(open('training-data.matrix', 'rb'))\n",
    "print(type(input_data_array))\n",
    "input_data_array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_input = pd.DataFrame(input_data_array)\n",
    "print(df_input.shape)\n",
    "df_input.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the range of the values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(df_input.max(axis=0).tolist(), 'o')\n",
    "plt.xlabel('Index of feature')\n",
    "plt.ylabel('Maximum')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we chose, the input data has count values. However, contrary to the input data, the maximum of each variable is not 1. \n",
    "\n",
    "We can note that the input of `medgan` and the [output](#gen-samples) of `medgan` have the same number of columns. Thus, `gen-samples.npy` is a (fake) realistic generated dataset corresponding to the `.matrix` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Comparing the (fake) generated samples to the real-life original ones  <a name=\"comparison\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we wish to compare the accuracy of the (fake) generated dataset considering the original one. Contrary to binary data, we can no longer use dimension-wise probability.\n",
    "\n",
    "Taking the boxplot of each variable is not efficient for visualisation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With statistical indicators: mean, median, variance, quantile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_mean = df_input.mean().tolist()\n",
    "output_mean = df_output.mean().tolist()\n",
    "\n",
    "start = min(np.min(input_mean), np.min(output_mean))\n",
    "stop = max(np.max(input_mean), np.max(output_mean))\n",
    "p = df_input.shape[1]\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(input_mean, output_mean, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['mean for each feature', 'ideal mean for each feature'])\n",
    "plt.title('Performance of medGAN on count features')\n",
    "plt.xlabel('For the real dataset')\n",
    "plt.ylabel('For the (fake) generated dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of the x-axis and the y-axis are ordered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_var = df_input.var().tolist()\n",
    "output_var = df_output.var().tolist()\n",
    "\n",
    "start = min(np.min(input_var), np.min(output_var))\n",
    "stop = max(np.max(input_var), np.max(output_var))\n",
    "p = df_input.shape[1]\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(input_var, output_var, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['variance for each feature', 'ideal variance for each feature'])\n",
    "plt.title('Performance of medGAN on count features')\n",
    "plt.xlabel('For the real dataset')\n",
    "plt.ylabel('For the (fake) generated dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qt = 0.5 # choose quantile between 0 and 1\n",
    "input_qt = df_input.quantile(qt).tolist()\n",
    "output_qt = df_output.quantile(qt).tolist()\n",
    "\n",
    "start = min(np.min(input_qt), np.min(output_qt))\n",
    "stop = max(np.max(input_qt), np.max(output_qt))\n",
    "p = df_input.shape[1]\n",
    "X = np.linspace(start, stop, num=p+1)\n",
    "\n",
    "plt.plot(input_qt, output_qt, 'ok', X, X, '-g');\n",
    "\n",
    "plt.legend(['%s quantile for each feature' %(qt), 'ideal %s quantile for each feature' %(qt)])\n",
    "plt.title('Performance of medGAN on count features')\n",
    "plt.xlabel('For the real dataset')\n",
    "plt.ylabel('For the (fake) generated dataset')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## With correlation matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# corr_mat = df_input.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The computing time is too long."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Back to [top](#top)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}