Market_Basket_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Full credit to Jason Le / Alex Ellman \n",
    "\n",
    "I originally credited Alex Ellman from his blog at https://towardsdatascience.com/creating-a-grocery-product-recommender-for-instacart-c1b6bdf5ae13 for the use of his code.  \n",
    "\n",
    "Then I discovered another blog by Jason Le at https://towardsdatascience.com/instacart-market-basket-analysis-part-3-which-sets-of-products-should-be-recommended-to-shoppers-9651751d3cd3.  I believe he was the original creator the functions for creating the Association Rules.\n",
    "\n",
    "I have used these functions with very little changes except to some of the variable names, print statements, and comments.  We do have differing output because I had clustered my users slightly differently and created 20 clusters instead of 6."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Dataset information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This data was retrieved from Kaggle and was provided by Instacart for a market basket analysis competition in 2018.  \n",
    "\n",
    "The data is divided into 6 files:\n",
    "\n",
    "- **_Aisles.csv_**: 134 Unique aisle numbers and descriptions\n",
    "- **_Departments.csv_**: 21 Unique department numbers and descriptions\n",
    "- **_Products.csv_**: 49,688 Unique product ids, with description, aisle id, and department id\n",
    "- **_Orders.csv_**: 3,421,083 Unique order id, with user id, order number, order_dow, order_hour_of_day, days_since_prior_order, and eval_set indicating if the order is in train, prior, or test\n",
    "- **_Order_products_train.csv_**: Order id, product id, add to cart order, and reorder indicator\n",
    "- **_Order_products_prior.csv_**: Order id, product id, add to cart order, and reorder indicator\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import and basic info\n",
    "\n",
    "**Output** - aisles, orders, products, departments, order_products_prior, order_products_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import itertools\n",
    "import pickle\n",
    "from collections import Counter\n",
    "from itertools import combinations, groupby, chain\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare Data for Market Basket Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# order_products_prior = pd.read_csv(\"Data/order_products__prior.csv\")\n",
    "# order_products_train = pd.read_csv(\"Data/order_products__train.csv\")\n",
    "# orders = pd.read_csv(\"Data/orders.csv\")\n",
    "aisles = pd.read_csv(\"Data/aisles.csv\")\n",
    "# grouped_users = pickle.load(open(\"Pickle/grouped_users.p\", \"rb\"))\n",
    "products_desc = pickle.load(open(\"Pickle/products_desc.p\", \"rb\"))\n",
    "merged_orders = pickle.load(open(\"Pickle/merged_orders.p\", \"rb\"))\n",
    "# orders_test = orders[orders['eval_set']=='test']\n",
    "clustered_users = pickle.load(open(\"Pickle/clustered_users.p\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>order_id</th>\n",
       "      <th>user_id</th>\n",
       "      <th>eval_set</th>\n",
       "      <th>order_number</th>\n",
       "      <th>order_dow</th>\n",
       "      <th>order_hour_of_day</th>\n",
       "      <th>days_since_prior_order</th>\n",
       "      <th>product_id</th>\n",
       "      <th>add_to_cart_order</th>\n",
       "      <th>reordered</th>\n",
       "      <th>product_name</th>\n",
       "      <th>aisle_id</th>\n",
       "      <th>department_id</th>\n",
       "      <th>aisle</th>\n",
       "      <th>department</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>18486555</th>\n",
       "      <td>1</td>\n",
       "      <td>112108</td>\n",
       "      <td>train</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10246</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>Organic Celery Hearts</td>\n",
       "      <td>83</td>\n",
       "      <td>4</td>\n",
       "      <td>fresh vegetables</td>\n",
       "      <td>produce</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18486554</th>\n",
       "      <td>1</td>\n",
       "      <td>112108</td>\n",
       "      <td>train</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>11109</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Organic 4% Milk Fat Whole Milk Cottage Cheese</td>\n",
       "      <td>108</td>\n",
       "      <td>16</td>\n",
       "      <td>other creams cheeses</td>\n",
       "      <td>dairy eggs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18486558</th>\n",
       "      <td>1</td>\n",
       "      <td>112108</td>\n",
       "      <td>train</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>13176</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>24</td>\n",
       "      <td>4</td>\n",
       "      <td>fresh fruits</td>\n",
       "      <td>produce</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18486560</th>\n",
       "      <td>1</td>\n",
       "      <td>112108</td>\n",
       "      <td>train</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>22035</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>Organic Whole String Cheese</td>\n",
       "      <td>21</td>\n",
       "      <td>16</td>\n",
       "      <td>packaged cheese</td>\n",
       "      <td>dairy eggs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18486557</th>\n",
       "      <td>1</td>\n",
       "      <td>112108</td>\n",
       "      <td>train</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>9.0</td>\n",
       "      <td>43633</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>Lightly Smoked Sardines in Olive Oil</td>\n",
       "      <td>95</td>\n",
       "      <td>15</td>\n",
       "      <td>canned meat seafood</td>\n",
       "      <td>canned goods</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          order_id  user_id eval_set  order_number  order_dow  \\\n",
       "18486555         1   112108    train             4          4   \n",
       "18486554         1   112108    train             4          4   \n",
       "18486558         1   112108    train             4          4   \n",
       "18486560         1   112108    train             4          4   \n",
       "18486557         1   112108    train             4          4   \n",
       "\n",
       "          order_hour_of_day  days_since_prior_order  product_id  \\\n",
       "18486555                 10                     9.0       10246   \n",
       "18486554                 10                     9.0       11109   \n",
       "18486558                 10                     9.0       13176   \n",
       "18486560                 10                     9.0       22035   \n",
       "18486557                 10                     9.0       43633   \n",
       "\n",
       "          add_to_cart_order  reordered  \\\n",
       "18486555                  3          0   \n",
       "18486554                  2          1   \n",
       "18486558                  6          0   \n",
       "18486560                  8          1   \n",
       "18486557                  5          1   \n",
       "\n",
       "                                           product_name  aisle_id  \\\n",
       "18486555                          Organic Celery Hearts        83   \n",
       "18486554  Organic 4% Milk Fat Whole Milk Cottage Cheese       108   \n",
       "18486558                         Bag of Organic Bananas        24   \n",
       "18486560                    Organic Whole String Cheese        21   \n",
       "18486557           Lightly Smoked Sardines in Olive Oil        95   \n",
       "\n",
       "          department_id                 aisle    department  \n",
       "18486555              4      fresh vegetables       produce  \n",
       "18486554             16  other creams cheeses    dairy eggs  \n",
       "18486558              4          fresh fruits       produce  \n",
       "18486560             16       packaged cheese    dairy eggs  \n",
       "18486557             15   canned meat seafood  canned goods  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I want to calculate the association rules by each cluster.  It will make the data smaller.\n",
    "merged_orders = merged_orders.sort_values(['order_id','product_id'])\n",
    "merged_orders.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "clustered_users.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Merging these to add the cluster to each order\n",
    "clustered_orders = merged_orders.merge(clustered_users[['user_id','cluster']], on = 'user_id')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Market Basket Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns frequency counts for items and item pairs\n",
    "def freq(iterable):\n",
    "    if type(iterable) == pd.core.series.Series:\n",
    "        return iterable.value_counts().rename(\"freq\")\n",
    "    else: \n",
    "        return pd.Series(Counter(iterable)).rename(\"freq\")\n",
    "\n",
    "    \n",
    "# Returns number of unique orders\n",
    "def order_count(order_item):\n",
    "    return len(set(order_item.index))\n",
    "\n",
    "\n",
    "# Returns generator that yields item pairs, one at a time\n",
    "'''Python Generators - A generator is a special type of function that returns an iterable sequence of items.\n",
    "However, unlike regular functions that return all the values at once, a generator yields one value at a time. To get the\n",
    "next value in the set, we must ask for it — either by explicitly calling the generator’s built-in “next” method, or\n",
    "implicitly via a for loop.  This is a great property of generators because it means that we don’t have to store all of\n",
    "the values in memory at once. We can load and process one value at a time, discard that value when we finished, and move\n",
    "on to process the next value. This feature makes generators perfect for creating item pairs and counting their\n",
    "frequency of co-occurrence.'''\n",
    "def get_item_pairs(order_item):\n",
    "    order_item = order_item.reset_index().to_numpy()\n",
    "    for order_id, order_object in groupby(order_item, lambda x: x[0]):\n",
    "        item_list = [item[1] for item in order_object]\n",
    "              \n",
    "        for item_pair in combinations(item_list, 2):\n",
    "            yield item_pair\n",
    "            \n",
    "\n",
    "# Returns frequency and support associated with item\n",
    "def merge_item_stats(item_pairs, item_stats):\n",
    "    return (item_pairs\n",
    "                .merge(item_stats.rename(columns={'freq': 'freqA', 'support': 'supportA'}), \n",
    "                       left_on='item_A', right_index=True)\n",
    "                .merge(item_stats.rename(columns={'freq': 'freqB', 'support': 'supportB'}), \n",
    "                       left_on='item_B', right_index=True))\n",
    "\n",
    "\n",
    "# Returns name associated with item\n",
    "def merge_item_name(rules, item_name):\n",
    "    columns = ['itemA','itemB','freqAB','supportAB','freqA','supportA','freqB','supportB', \n",
    "               'confidenceAtoB','confidenceBtoA','lift']\n",
    "    rules = (rules\n",
    "                .merge(item_name.rename(columns={'item_name': 'itemA'}), left_on='item_A', right_on='item_id')\n",
    "                .merge(item_name.rename(columns={'item_name': 'itemB'}), left_on='item_B', right_on='item_id'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def association_rules(order_item, min_support):\n",
    "\n",
    "#     print(\"Starting number of order_item pairs: {:22d}\".format(len(order_item)))\n",
    "    print(\"Starting number of orders, items, order_item pairs: {}, {}, {}\".format(order_item.index.nunique(),\n",
    "                                                                          order_item.nunique(),len(order_item))) \n",
    "\n",
    "    # Calculate item frequency and support\n",
    "    item_stats             = freq(order_item).to_frame(\"freq\")\n",
    "    item_stats['support']  = item_stats['freq'] / order_count(order_item) \n",
    "\n",
    "\n",
    "    # Filter from order_item items below min support \n",
    "    qualifying_items       = item_stats[item_stats['support'] >= min_support].index\n",
    "    order_item             = order_item[order_item.isin(qualifying_items)]\n",
    "\n",
    "    print(\"Items with support >= {}: {:15d}\".format(min_support, len(qualifying_items)))\n",
    "#     print(\"Remaining order_item: {:21d}\".format(len(order_item)))\n",
    "    print(\"Remaning number of orders, items, order_item pairs: {}, {}, {}\".format(order_item.index.nunique(),\n",
    "                                                                          order_item.nunique(),len(order_item)))\n",
    "\n",
    "\n",
    "    # Filter from order_item orders with less than 2 items\n",
    "    order_size             = freq(order_item.index)\n",
    "    qualifying_orders      = order_size[order_size >= 2].index\n",
    "    order_item             = order_item[order_item.index.isin(qualifying_orders)]\n",
    "\n",
    "    print(\"Remaining orders with 2+ items: {:11d}\".format(len(qualifying_orders)))\n",
    "#     print(\"Remaining order_item: {:21d}\".format(len(order_item)))\n",
    "    print(\"Remaining number of orders, items, order_item pairs: {}, {}, {}\".format(order_item.index.nunique(),\n",
    "                                                                          order_item.nunique(),len(order_item)))\n",
    "\n",
    "\n",
    "    # Recalculate item frequency and support\n",
    "    item_stats             = freq(order_item).to_frame(\"freq\")\n",
    "    item_stats['support']  = item_stats['freq'] / order_count(order_item) \n",
    "\n",
    "\n",
    "    # Get item pairs generator\n",
    "    item_pair_gen          = get_item_pairs(order_item)\n",
    "\n",
    "\n",
    "    # Calculate item pair frequency and support\n",
    "    item_pairs              = freq(item_pair_gen).to_frame(\"freqAB\")\n",
    "    item_pairs['supportAB'] = item_pairs['freqAB'] / len(qualifying_orders)\n",
    "\n",
    "    print(\"Item pairs: {:31d}\".format(len(item_pairs)))\n",
    "\n",
    "\n",
    "    # Filter from item_pairs those below min support\n",
    "    item_pairs              = item_pairs[item_pairs['supportAB'] >= min_support]\n",
    "\n",
    "    print(\"Item pairs with support >= {}: {:10d}\\n\".format(min_support, len(item_pairs)))\n",
    "\n",
    "\n",
    "    # Create table of association rules and compute relevant metrics\n",
    "    item_pairs = item_pairs.reset_index().rename(columns={'level_0': 'item_A', 'level_1': 'item_B'})\n",
    "    item_pairs = merge_item_stats(item_pairs, item_stats)\n",
    "    \n",
    "    item_pairs['confAtoB'] = item_pairs['supportAB'] / item_pairs['supportA']\n",
    "    item_pairs['confBtoA'] = item_pairs['supportAB'] / item_pairs['supportB']\n",
    "    item_pairs['lift']           = item_pairs['supportAB'] / (item_pairs['supportA'] * item_pairs['supportB'])\n",
    "    \n",
    "    \n",
    "    # Return association rules sorted by lift in descending order\n",
    "    return item_pairs.sort_values('lift', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate order data by cluster and transform into expected format of group association function\n",
    "\n",
    "cluster_orders = {}\n",
    "for x in range(0,20):\n",
    "    i = clustered_orders[clustered_orders['cluster'] == x]\n",
    "    i = i[['order_id', 'product_id']]\n",
    "    i = i.set_index('order_id')['product_id'].rename('item_id')\n",
    "    cluster_orders[x] = i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster:  0\n",
      "Starting number of orders, items, order_item pairs: 22099, 14626, 264115\n",
      "Items with support >= 0.0001:            8717\n",
      "Remaning number of orders, items, order_item pairs: 22078, 8717, 256351\n",
      "Remaining orders with 2+ items:       21637\n",
      "Remaining number of orders, items, order_item pairs: 21637, 8717, 255910\n",
      "Item pairs:                          808257\n",
      "Item pairs with support >= 0.0001:     178869\n",
      "\n",
      "Cluster:  1\n",
      "Starting number of orders, items, order_item pairs: 785071, 46426, 7788280\n",
      "Items with support >= 0.0001:           10650\n",
      "Remaning number of orders, items, order_item pairs: 781265, 10650, 7187429\n",
      "Remaining orders with 2+ items:      746198\n",
      "Remaining number of orders, items, order_item pairs: 746198, 10650, 7152362\n",
      "Item pairs:                        10064495\n",
      "Item pairs with support >= 0.0001:      54507\n",
      "\n",
      "Cluster:  2\n",
      "Starting number of orders, items, order_item pairs: 53338, 16897, 754548\n",
      "Items with support >= 0.0001:            8319\n",
      "Remaning number of orders, items, order_item pairs: 53223, 8319, 736505\n",
      "Remaining orders with 2+ items:       52262\n",
      "Remaining number of orders, items, order_item pairs: 52262, 8319, 735544\n",
      "Item pairs:                         1756266\n",
      "Item pairs with support >= 0.0001:     276018\n",
      "\n",
      "Cluster:  3\n",
      "Starting number of orders, items, order_item pairs: 146839, 37993, 1862032\n",
      "Items with support >= 0.0001:           15065\n",
      "Remaning number of orders, items, order_item pairs: 146568, 15065, 1754943\n",
      "Remaining orders with 2+ items:      143766\n",
      "Remaining number of orders, items, order_item pairs: 143766, 15065, 1752141\n",
      "Item pairs:                         6383258\n",
      "Item pairs with support >= 0.0001:      72327\n",
      "\n",
      "Cluster:  4\n",
      "Starting number of orders, items, order_item pairs: 542910, 36201, 5951742\n",
      "Items with support >= 0.0001:            8735\n",
      "Remaning number of orders, items, order_item pairs: 541504, 8735, 5646989\n",
      "Remaining orders with 2+ items:      526738\n",
      "Remaining number of orders, items, order_item pairs: 526738, 8735, 5632223\n",
      "Item pairs:                         6933457\n",
      "Item pairs with support >= 0.0001:      87117\n",
      "\n",
      "Cluster:  5\n",
      "Starting number of orders, items, order_item pairs: 34338, 21392, 347793\n",
      "Items with support >= 0.0001:           12165\n",
      "Remaning number of orders, items, order_item pairs: 34277, 12165, 332363\n",
      "Remaining orders with 2+ items:       32990\n",
      "Remaining number of orders, items, order_item pairs: 32990, 12165, 331076\n",
      "Item pairs:                         1368605\n",
      "Item pairs with support >= 0.0001:     101697\n",
      "\n",
      "Cluster:  6\n",
      "Starting number of orders, items, order_item pairs: 201588, 27591, 2797103\n",
      "Items with support >= 0.0001:            9451\n",
      "Remaning number of orders, items, order_item pairs: 201393, 9451, 2699856\n",
      "Remaining orders with 2+ items:      198973\n",
      "Remaining number of orders, items, order_item pairs: 198973, 9451, 2697436\n",
      "Item pairs:                         4833446\n",
      "Item pairs with support >= 0.0001:     174063\n",
      "\n",
      "Cluster:  7\n",
      "Starting number of orders, items, order_item pairs: 8934, 9021, 122688\n",
      "Items with support >= 0.0001:            9021\n",
      "Remaning number of orders, items, order_item pairs: 8934, 9021, 122688\n",
      "Remaining orders with 2+ items:        8798\n",
      "Remaining number of orders, items, order_item pairs: 8798, 9012, 122552\n",
      "Item pairs:                          521231\n",
      "Item pairs with support >= 0.0001:     521231\n",
      "\n",
      "Cluster:  8\n",
      "Starting number of orders, items, order_item pairs: 90579, 26917, 1653446\n",
      "Items with support >= 0.0001:           11881\n",
      "Remaning number of orders, items, order_item pairs: 90523, 11881, 1606804\n",
      "Remaining orders with 2+ items:       89932\n",
      "Remaining number of orders, items, order_item pairs: 89932, 11881, 1606213\n",
      "Item pairs:                         4496908\n",
      "Item pairs with support >= 0.0001:     383194\n",
      "\n",
      "Cluster:  9\n",
      "Starting number of orders, items, order_item pairs: 28956, 14337, 237431\n",
      "Items with support >= 0.0001:            8642\n",
      "Remaning number of orders, items, order_item pairs: 28907, 8642, 229862\n",
      "Remaining orders with 2+ items:       27331\n",
      "Remaining number of orders, items, order_item pairs: 27331, 8641, 228286\n",
      "Item pairs:                          740441\n",
      "Item pairs with support >= 0.0001:     109262\n",
      "\n",
      "Cluster:  10\n",
      "Starting number of orders, items, order_item pairs: 81074, 6052, 455761\n",
      "Items with support >= 0.0001:            1982\n",
      "Remaning number of orders, items, order_item pairs: 80944, 1982, 445255\n",
      "Remaining orders with 2+ items:       73151\n",
      "Remaining number of orders, items, order_item pairs: 73151, 1982, 437462\n",
      "Item pairs:                          172361\n",
      "Item pairs with support >= 0.0001:      33197\n",
      "\n",
      "Cluster:  11\n",
      "Starting number of orders, items, order_item pairs: 27243, 15085, 315124\n",
      "Items with support >= 0.0001:            9743\n",
      "Remaning number of orders, items, order_item pairs: 27224, 9743, 307972\n",
      "Remaining orders with 2+ items:       26602\n",
      "Remaining number of orders, items, order_item pairs: 26602, 9743, 307350\n",
      "Item pairs:                         1124388\n",
      "Item pairs with support >= 0.0001:     196489\n",
      "\n",
      "Cluster:  12\n",
      "Starting number of orders, items, order_item pairs: 841322, 46782, 5804594\n",
      "Items with support >= 0.0001:            8594\n",
      "Remaning number of orders, items, order_item pairs: 830547, 8594, 5172644\n",
      "Remaining orders with 2+ items:      729445\n",
      "Remaining number of orders, items, order_item pairs: 729445, 8594, 5071542\n",
      "Item pairs:                         6168742\n",
      "Item pairs with support >= 0.0001:      26943\n",
      "\n",
      "Cluster:  13\n",
      "Starting number of orders, items, order_item pairs: 40293, 24738, 563544\n",
      "Items with support >= 0.0001:           13234\n",
      "Remaning number of orders, items, order_item pairs: 40256, 13234, 541195\n",
      "Remaining orders with 2+ items:       39559\n",
      "Remaining number of orders, items, order_item pairs: 39559, 13234, 540498\n",
      "Item pairs:                         2309232\n",
      "Item pairs with support >= 0.0001:     309892\n",
      "\n",
      "Cluster:  14\n",
      "Starting number of orders, items, order_item pairs: 192427, 24899, 2374227\n",
      "Items with support >= 0.0001:            8237\n",
      "Remaning number of orders, items, order_item pairs: 192140, 8237, 2288654\n",
      "Remaining orders with 2+ items:      188859\n",
      "Remaining number of orders, items, order_item pairs: 188859, 8237, 2285373\n",
      "Item pairs:                         3636379\n",
      "Item pairs with support >= 0.0001:     138346\n",
      "\n",
      "Cluster:  15\n",
      "Starting number of orders, items, order_item pairs: 7051, 7499, 72868\n",
      "Items with support >= 0.0001:            7499\n",
      "Remaning number of orders, items, order_item pairs: 7051, 7499, 72868\n",
      "Remaining orders with 2+ items:        6701\n",
      "Remaining number of orders, items, order_item pairs: 6701, 7487, 72518\n",
      "Item pairs:                          274107\n",
      "Item pairs with support >= 0.0001:     274107\n",
      "\n",
      "Cluster:  16\n",
      "Starting number of orders, items, order_item pairs: 35667, 16841, 507608\n",
      "Items with support >= 0.0001:            9570\n",
      "Remaning number of orders, items, order_item pairs: 35646, 9570, 495824\n",
      "Remaining orders with 2+ items:       35209\n",
      "Remaining number of orders, items, order_item pairs: 35209, 9570, 495387\n",
      "Item pairs:                         1684437\n",
      "Item pairs with support >= 0.0001:     267992\n",
      "\n",
      "Cluster:  17\n",
      "Starting number of orders, items, order_item pairs: 11543, 7112, 124542\n",
      "Items with support >= 0.0001:            5124\n",
      "Remaning number of orders, items, order_item pairs: 11532, 5124, 122554\n",
      "Remaining orders with 2+ items:       10987\n",
      "Remaining number of orders, items, order_item pairs: 10987, 5123, 122009\n",
      "Item pairs:                          348589\n",
      "Item pairs with support >= 0.0001:     137967\n",
      "\n",
      "Cluster:  18\n",
      "Starting number of orders, items, order_item pairs: 169327, 39047, 1553965\n",
      "Items with support >= 0.0001:           13520\n",
      "Remaning number of orders, items, order_item pairs: 168488, 13520, 1421256\n",
      "Remaining orders with 2+ items:      159974\n",
      "Remaining number of orders, items, order_item pairs: 159974, 13520, 1412742\n",
      "Item pairs:                         4469825\n",
      "Item pairs with support >= 0.0001:      26478\n",
      "\n",
      "Cluster:  19\n",
      "Starting number of orders, items, order_item pairs: 25484, 16350, 267695\n",
      "Items with support >= 0.0001:            9644\n",
      "Remaning number of orders, items, order_item pairs: 25457, 9644, 258928\n",
      "Remaining orders with 2+ items:       24568\n",
      "Remaining number of orders, items, order_item pairs: 24568, 9644, 258039\n",
      "Item pairs:                          854966\n",
      "Item pairs with support >= 0.0001:     160001\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# run each cluster series through association rule function and obtain association rules at item level\n",
    "\n",
    "group_association_rules_dic = {}\n",
    "for x in range(0,20):\n",
    "    print('Cluster: ', x)\n",
    "    group_association_rules_dic[x] = association_rules(cluster_orders[x],.0001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# format item association rule dataframes and merge product names\n",
    "\n",
    "for x in group_association_rules_dic:\n",
    "    group_association_rules_dic[x] = group_association_rules_dic[x].merge(products_desc[['product_id','product_name']]\n",
    "               .rename(columns = {'product_id':'item_A','product_name':'product_name_A'}), on = 'item_A')\n",
    "    group_association_rules_dic[x] = group_association_rules_dic[x].merge(products_desc[['product_id','product_name']]\n",
    "               .rename(columns = {'product_id':'item_B','product_name':'product_name_B'}), on = 'item_B')\n",
    "    group_association_rules_dic[x] = group_association_rules_dic[x][['item_A','item_B','product_name_A',\n",
    "                                                                    'product_name_B','freqAB','supportAB','freqA',\n",
    "                                                                    'supportA','freqB','supportB','confAtoB',\n",
    "                                                                    'confBtoA','lift']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(group_association_rules_dic, open(\"Pickle/group_association_rules_dic.p\", \"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# group_association_rules_dic = pickle.load(open(\"Pickle/group_association_rules_dic.p\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_A</th>\n",
       "      <th>item_B</th>\n",
       "      <th>product_name_A</th>\n",
       "      <th>product_name_B</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confAtoB</th>\n",
       "      <th>confBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>12191</td>\n",
       "      <td>29169</td>\n",
       "      <td>Kettle Cooked Original Potato Chips</td>\n",
       "      <td>Sea Salt &amp; Cracked Pepper Potato Chips</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8189.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2202</td>\n",
       "      <td>47716</td>\n",
       "      <td>98% Fat Free Condensed Soup Cream of Chicken</td>\n",
       "      <td>98% Fat Free Condensed Soup Cream Of Celery</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8189.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>850</th>\n",
       "      <td>2753</td>\n",
       "      <td>21985</td>\n",
       "      <td>Blueberry Drinkable Whole Milk Yogurt</td>\n",
       "      <td>Vanilla Whole Milk Drinkable Yogurt</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8189.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>823</th>\n",
       "      <td>6907</td>\n",
       "      <td>42569</td>\n",
       "      <td>Chips Ahoy! White Fudge Chunky Chocolate Chunk...</td>\n",
       "      <td>Cinnamon Bun Sandwich Cookies</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8189.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>5909</td>\n",
       "      <td>42436</td>\n",
       "      <td>Lemon Verbena Hand Wash</td>\n",
       "      <td>Hand Wash, Lavender Fields</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8189.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40530</th>\n",
       "      <td>13176</td>\n",
       "      <td>43965</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Glazed Buttermilk Doughnuts</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000122</td>\n",
       "      <td>3716</td>\n",
       "      <td>0.151254</td>\n",
       "      <td>117</td>\n",
       "      <td>0.004762</td>\n",
       "      <td>0.000807</td>\n",
       "      <td>0.025641</td>\n",
       "      <td>0.169523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29662</th>\n",
       "      <td>5450</td>\n",
       "      <td>47209</td>\n",
       "      <td>Small Hass Avocado</td>\n",
       "      <td>Organic Hass Avocado</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000204</td>\n",
       "      <td>387</td>\n",
       "      <td>0.015752</td>\n",
       "      <td>1994</td>\n",
       "      <td>0.081162</td>\n",
       "      <td>0.012920</td>\n",
       "      <td>0.002508</td>\n",
       "      <td>0.159186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9782</th>\n",
       "      <td>6729</td>\n",
       "      <td>21137</td>\n",
       "      <td>Cookie Tray</td>\n",
       "      <td>Organic Strawberries</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>302</td>\n",
       "      <td>0.012292</td>\n",
       "      <td>2496</td>\n",
       "      <td>0.101596</td>\n",
       "      <td>0.013245</td>\n",
       "      <td>0.001603</td>\n",
       "      <td>0.130370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9847</th>\n",
       "      <td>16797</td>\n",
       "      <td>21137</td>\n",
       "      <td>Strawberries</td>\n",
       "      <td>Organic Strawberries</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000448</td>\n",
       "      <td>1069</td>\n",
       "      <td>0.043512</td>\n",
       "      <td>2496</td>\n",
       "      <td>0.101596</td>\n",
       "      <td>0.010290</td>\n",
       "      <td>0.004407</td>\n",
       "      <td>0.101284</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2428</th>\n",
       "      <td>13176</td>\n",
       "      <td>24852</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Banana</td>\n",
       "      <td>21</td>\n",
       "      <td>0.000855</td>\n",
       "      <td>3716</td>\n",
       "      <td>0.151254</td>\n",
       "      <td>3236</td>\n",
       "      <td>0.131716</td>\n",
       "      <td>0.005651</td>\n",
       "      <td>0.006489</td>\n",
       "      <td>0.042905</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>160001 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       item_A  item_B                                     product_name_A  \\\n",
       "0       12191   29169                Kettle Cooked Original Potato Chips   \n",
       "1        2202   47716       98% Fat Free Condensed Soup Cream of Chicken   \n",
       "850      2753   21985              Blueberry Drinkable Whole Milk Yogurt   \n",
       "823      6907   42569  Chips Ahoy! White Fudge Chunky Chocolate Chunk...   \n",
       "10       5909   42436                            Lemon Verbena Hand Wash   \n",
       "...       ...     ...                                                ...   \n",
       "40530   13176   43965                             Bag of Organic Bananas   \n",
       "29662    5450   47209                                 Small Hass Avocado   \n",
       "9782     6729   21137                                        Cookie Tray   \n",
       "9847    16797   21137                                       Strawberries   \n",
       "2428    13176   24852                             Bag of Organic Bananas   \n",
       "\n",
       "                                    product_name_B  freqAB  supportAB  freqA  \\\n",
       "0           Sea Salt & Cracked Pepper Potato Chips       3   0.000122      3   \n",
       "1      98% Fat Free Condensed Soup Cream Of Celery       3   0.000122      3   \n",
       "850            Vanilla Whole Milk Drinkable Yogurt       3   0.000122      3   \n",
       "823                  Cinnamon Bun Sandwich Cookies       3   0.000122      3   \n",
       "10                      Hand Wash, Lavender Fields       3   0.000122      3   \n",
       "...                                            ...     ...        ...    ...   \n",
       "40530                  Glazed Buttermilk Doughnuts       3   0.000122   3716   \n",
       "29662                         Organic Hass Avocado       5   0.000204    387   \n",
       "9782                          Organic Strawberries       4   0.000163    302   \n",
       "9847                          Organic Strawberries      11   0.000448   1069   \n",
       "2428                                        Banana      21   0.000855   3716   \n",
       "\n",
       "       supportA  freqB  supportB  confAtoB  confBtoA         lift  \n",
       "0      0.000122      3  0.000122  1.000000  1.000000  8189.333333  \n",
       "1      0.000122      3  0.000122  1.000000  1.000000  8189.333333  \n",
       "850    0.000122      3  0.000122  1.000000  1.000000  8189.333333  \n",
       "823    0.000122      3  0.000122  1.000000  1.000000  8189.333333  \n",
       "10     0.000122      3  0.000122  1.000000  1.000000  8189.333333  \n",
       "...         ...    ...       ...       ...       ...          ...  \n",
       "40530  0.151254    117  0.004762  0.000807  0.025641     0.169523  \n",
       "29662  0.015752   1994  0.081162  0.012920  0.002508     0.159186  \n",
       "9782   0.012292   2496  0.101596  0.013245  0.001603     0.130370  \n",
       "9847   0.043512   2496  0.101596  0.010290  0.004407     0.101284  \n",
       "2428   0.151254   3236  0.131716  0.005651  0.006489     0.042905  \n",
       "\n",
       "[160001 rows x 13 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# example of association rule dataframe\n",
    "\n",
    "x = group_association_rules_dic[19]\n",
    "x.sort_values('lift', ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Market Basket Analysis at Aisle level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate order data by cluster and transform into expected format of group association function at aisle level\n",
    "\n",
    "cluster_aisle_orders = {}\n",
    "for x in range(0,20):\n",
    "    i = clustered_orders[clustered_orders['cluster'] == x]\n",
    "    i = i[['order_id', 'product_id', 'aisle_id']]\n",
    "    i = i.groupby(['order_id','aisle_id'])['product_id'].count().reset_index().set_index('order_id')['aisle_id'].rename('item_id')\n",
    "    cluster_aisle_orders[x] = i"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster:  0\n",
      "Starting number of orders, items, order_item pairs: 22099, 134, 190556\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 22099, 134, 190556\n",
      "Remaining orders with 2+ items:       21561\n",
      "Remaining number of orders, items, order_item pairs: 21561, 134, 190018\n",
      "Item pairs:                            8092\n",
      "Item pairs with support >= 0.0001:       7159\n",
      "\n",
      "Cluster:  1\n",
      "Starting number of orders, items, order_item pairs: 785071, 134, 5650431\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 785071, 134, 5650431\n",
      "Remaining orders with 2+ items:      744957\n",
      "Remaining number of orders, items, order_item pairs: 744957, 134, 5610317\n",
      "Item pairs:                            8906\n",
      "Item pairs with support >= 0.0001:       7324\n",
      "\n",
      "Cluster:  2\n",
      "Starting number of orders, items, order_item pairs: 53338, 134, 410177\n",
      "Items with support >= 0.0001:             133\n",
      "Remaning number of orders, items, order_item pairs: 53338, 133, 410174\n",
      "Remaining orders with 2+ items:       51295\n",
      "Remaining number of orders, items, order_item pairs: 51295, 133, 408131\n",
      "Item pairs:                            7223\n",
      "Item pairs with support >= 0.0001:       5201\n",
      "\n",
      "Cluster:  3\n",
      "Starting number of orders, items, order_item pairs: 146839, 134, 1418848\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 146839, 134, 1418848\n",
      "Remaining orders with 2+ items:      143604\n",
      "Remaining number of orders, items, order_item pairs: 143604, 134, 1415613\n",
      "Item pairs:                            8847\n",
      "Item pairs with support >= 0.0001:       8147\n",
      "\n",
      "Cluster:  4\n",
      "Starting number of orders, items, order_item pairs: 542910, 134, 4177720\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 542910, 134, 4177720\n",
      "Remaining orders with 2+ items:      526247\n",
      "Remaining number of orders, items, order_item pairs: 526247, 134, 4161057\n",
      "Item pairs:                            8837\n",
      "Item pairs with support >= 0.0001:       6657\n",
      "\n",
      "Cluster:  5\n",
      "Starting number of orders, items, order_item pairs: 34338, 134, 273476\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 34338, 134, 273476\n",
      "Remaining orders with 2+ items:       32833\n",
      "Remaining number of orders, items, order_item pairs: 32833, 134, 271971\n",
      "Item pairs:                            8521\n",
      "Item pairs with support >= 0.0001:       7938\n",
      "\n",
      "Cluster:  6\n",
      "Starting number of orders, items, order_item pairs: 201588, 134, 1967768\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 201588, 134, 1967768\n",
      "Remaining orders with 2+ items:      198837\n",
      "Remaining number of orders, items, order_item pairs: 198837, 134, 1965017\n",
      "Item pairs:                            8663\n",
      "Item pairs with support >= 0.0001:       6860\n",
      "\n",
      "Cluster:  7\n",
      "Starting number of orders, items, order_item pairs: 8934, 134, 88466\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 8934, 134, 88466\n",
      "Remaining orders with 2+ items:        8760\n",
      "Remaining number of orders, items, order_item pairs: 8760, 134, 88292\n",
      "Item pairs:                            7569\n",
      "Item pairs with support >= 0.0001:       7569\n",
      "\n",
      "Cluster:  8\n",
      "Starting number of orders, items, order_item pairs: 90579, 134, 1129572\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 90579, 134, 1129572\n",
      "Remaining orders with 2+ items:       89898\n",
      "Remaining number of orders, items, order_item pairs: 89898, 134, 1128891\n",
      "Item pairs:                            8757\n",
      "Item pairs with support >= 0.0001:       7900\n",
      "\n",
      "Cluster:  9\n",
      "Starting number of orders, items, order_item pairs: 28956, 134, 179363\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 28956, 134, 179363\n",
      "Remaining orders with 2+ items:       26351\n",
      "Remaining number of orders, items, order_item pairs: 26351, 134, 176758\n",
      "Item pairs:                            8221\n",
      "Item pairs with support >= 0.0001:       7406\n",
      "\n",
      "Cluster:  10\n",
      "Starting number of orders, items, order_item pairs: 81074, 134, 342174\n",
      "Items with support >= 0.0001:             123\n",
      "Remaning number of orders, items, order_item pairs: 81074, 123, 342125\n",
      "Remaining orders with 2+ items:       71526\n",
      "Remaining number of orders, items, order_item pairs: 71526, 123, 332577\n",
      "Item pairs:                            5853\n",
      "Item pairs with support >= 0.0001:       3478\n",
      "\n",
      "Cluster:  11\n",
      "Starting number of orders, items, order_item pairs: 27243, 134, 230075\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 27243, 134, 230075\n",
      "Remaining orders with 2+ items:       26484\n",
      "Remaining number of orders, items, order_item pairs: 26484, 134, 229316\n",
      "Item pairs:                            8317\n",
      "Item pairs with support >= 0.0001:       7672\n",
      "\n",
      "Cluster:  12\n",
      "Starting number of orders, items, order_item pairs: 841322, 134, 4367609\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 841322, 134, 4367609\n",
      "Remaining orders with 2+ items:      729657\n",
      "Remaining number of orders, items, order_item pairs: 729657, 134, 4255944\n",
      "Item pairs:                            8901\n",
      "Item pairs with support >= 0.0001:       7031\n",
      "\n",
      "Cluster:  13\n",
      "Starting number of orders, items, order_item pairs: 40293, 134, 415669\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 40293, 134, 415669\n",
      "Remaining orders with 2+ items:       39462\n",
      "Remaining number of orders, items, order_item pairs: 39462, 134, 414838\n",
      "Item pairs:                            8760\n",
      "Item pairs with support >= 0.0001:       8382\n",
      "\n",
      "Cluster:  14\n",
      "Starting number of orders, items, order_item pairs: 192427, 134, 1599614\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 192427, 134, 1599614\n",
      "Remaining orders with 2+ items:      188763\n",
      "Remaining number of orders, items, order_item pairs: 188763, 134, 1595950\n",
      "Item pairs:                            8552\n",
      "Item pairs with support >= 0.0001:       6526\n",
      "\n",
      "Cluster:  15\n",
      "Starting number of orders, items, order_item pairs: 7051, 134, 49185\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 7051, 134, 49185\n",
      "Remaining orders with 2+ items:        6511\n",
      "Remaining number of orders, items, order_item pairs: 6511, 134, 48645\n",
      "Item pairs:                            6567\n",
      "Item pairs with support >= 0.0001:       6567\n",
      "\n",
      "Cluster:  16\n",
      "Starting number of orders, items, order_item pairs: 35667, 134, 349190\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 35667, 134, 349190\n",
      "Remaining orders with 2+ items:       35121\n",
      "Remaining number of orders, items, order_item pairs: 35121, 134, 348644\n",
      "Item pairs:                            8126\n",
      "Item pairs with support >= 0.0001:       7071\n",
      "\n",
      "Cluster:  17\n",
      "Starting number of orders, items, order_item pairs: 11543, 129, 86072\n",
      "Items with support >= 0.0001:             129\n",
      "Remaning number of orders, items, order_item pairs: 11543, 129, 86072\n",
      "Remaining orders with 2+ items:       10900\n",
      "Remaining number of orders, items, order_item pairs: 10900, 129, 85429\n",
      "Item pairs:                            6423\n",
      "Item pairs with support >= 0.0001:       5625\n",
      "\n",
      "Cluster:  18\n",
      "Starting number of orders, items, order_item pairs: 169327, 134, 1202860\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 169327, 134, 1202860\n",
      "Remaining orders with 2+ items:      159580\n",
      "Remaining number of orders, items, order_item pairs: 159580, 134, 1193113\n",
      "Item pairs:                            8860\n",
      "Item pairs with support >= 0.0001:       8178\n",
      "\n",
      "Cluster:  19\n",
      "Starting number of orders, items, order_item pairs: 25484, 134, 200397\n",
      "Items with support >= 0.0001:             134\n",
      "Remaning number of orders, items, order_item pairs: 25484, 134, 200397\n",
      "Remaining orders with 2+ items:       24389\n",
      "Remaining number of orders, items, order_item pairs: 24389, 134, 199302\n",
      "Item pairs:                            8108\n",
      "Item pairs with support >= 0.0001:       7107\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# run association function at aisle level\n",
    "\n",
    "group_aisle_association_rules_dic = {}\n",
    "for x in range(0,20):\n",
    "    print(\"Cluster: \", x)\n",
    "    group_aisle_association_rules_dic[x] = association_rules(cluster_aisle_orders[x],.0001)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(group_aisle_association_rules_dic, open(\"Pickle/group_aisle_association_rules_dic.p\", \"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# group_aisle_association_rules_dic = pickle.load(open(\"Pickle/group_aisle_association_rules_dic.p\", \"rb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reformat cluster aisle assocation dataframes\n",
    "\n",
    "for x in group_aisle_association_rules_dic:\n",
    "    group_aisle_association_rules_dic[x] = group_aisle_association_rules_dic[x].merge(aisles\n",
    "               .rename(columns = {'aisle_id':'item_A','aisle':'aisle_name_A'}), on = 'item_A')\n",
    "    group_aisle_association_rules_dic[x] = group_aisle_association_rules_dic[x].merge(aisles\n",
    "               .rename(columns = {'aisle_id':'item_B','aisle':'aisle_name_B'}), on = 'item_B')\n",
    "    group_aisle_association_rules_dic[x] = group_aisle_association_rules_dic[x][['item_A','item_B','aisle_name_A',\n",
    "                                                                    'aisle_name_B','freqAB','supportAB','freqA',\n",
    "                                                                    'supportA','freqB','supportB','confAtoB',\n",
    "                                                                    'confBtoA','lift']]\n",
    "    group_aisle_association_rules_dic[x] = group_aisle_association_rules_dic[x].rename(columns = {'item_A':'aisle_A','item_B':'aisle_B'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aisle_A</th>\n",
       "      <th>aisle_B</th>\n",
       "      <th>aisle_name_A</th>\n",
       "      <th>aisle_name_B</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confAtoB</th>\n",
       "      <th>confBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28</td>\n",
       "      <td>62</td>\n",
       "      <td>red wines</td>\n",
       "      <td>white wines</td>\n",
       "      <td>1022</td>\n",
       "      <td>0.001372</td>\n",
       "      <td>4408</td>\n",
       "      <td>0.005917</td>\n",
       "      <td>3995</td>\n",
       "      <td>0.005363</td>\n",
       "      <td>0.231851</td>\n",
       "      <td>0.255820</td>\n",
       "      <td>43.233832</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>62</td>\n",
       "      <td>134</td>\n",
       "      <td>white wines</td>\n",
       "      <td>specialty wines champagnes</td>\n",
       "      <td>236</td>\n",
       "      <td>0.000317</td>\n",
       "      <td>3995</td>\n",
       "      <td>0.005363</td>\n",
       "      <td>1297</td>\n",
       "      <td>0.001741</td>\n",
       "      <td>0.059074</td>\n",
       "      <td>0.181958</td>\n",
       "      <td>33.930202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>28</td>\n",
       "      <td>134</td>\n",
       "      <td>red wines</td>\n",
       "      <td>specialty wines champagnes</td>\n",
       "      <td>221</td>\n",
       "      <td>0.000297</td>\n",
       "      <td>4408</td>\n",
       "      <td>0.005917</td>\n",
       "      <td>1297</td>\n",
       "      <td>0.001741</td>\n",
       "      <td>0.050136</td>\n",
       "      <td>0.170393</td>\n",
       "      <td>28.796647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>124</td>\n",
       "      <td>134</td>\n",
       "      <td>spirits</td>\n",
       "      <td>specialty wines champagnes</td>\n",
       "      <td>155</td>\n",
       "      <td>0.000208</td>\n",
       "      <td>3349</td>\n",
       "      <td>0.004496</td>\n",
       "      <td>1297</td>\n",
       "      <td>0.001741</td>\n",
       "      <td>0.046282</td>\n",
       "      <td>0.119507</td>\n",
       "      <td>26.583232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5415</th>\n",
       "      <td>27</td>\n",
       "      <td>28</td>\n",
       "      <td>beers coolers</td>\n",
       "      <td>red wines</td>\n",
       "      <td>808</td>\n",
       "      <td>0.001085</td>\n",
       "      <td>5438</td>\n",
       "      <td>0.007300</td>\n",
       "      <td>4408</td>\n",
       "      <td>0.005917</td>\n",
       "      <td>0.148584</td>\n",
       "      <td>0.183303</td>\n",
       "      <td>25.110871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>157</th>\n",
       "      <td>91</td>\n",
       "      <td>124</td>\n",
       "      <td>soy lactosefree</td>\n",
       "      <td>spirits</td>\n",
       "      <td>264</td>\n",
       "      <td>0.000354</td>\n",
       "      <td>138303</td>\n",
       "      <td>0.185652</td>\n",
       "      <td>3349</td>\n",
       "      <td>0.004496</td>\n",
       "      <td>0.001909</td>\n",
       "      <td>0.078830</td>\n",
       "      <td>0.424608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5368</th>\n",
       "      <td>28</td>\n",
       "      <td>50</td>\n",
       "      <td>red wines</td>\n",
       "      <td>fruit vegetable snacks</td>\n",
       "      <td>83</td>\n",
       "      <td>0.000111</td>\n",
       "      <td>4408</td>\n",
       "      <td>0.005917</td>\n",
       "      <td>33070</td>\n",
       "      <td>0.044392</td>\n",
       "      <td>0.018829</td>\n",
       "      <td>0.002510</td>\n",
       "      <td>0.424164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>120</td>\n",
       "      <td>124</td>\n",
       "      <td>yogurt</td>\n",
       "      <td>spirits</td>\n",
       "      <td>398</td>\n",
       "      <td>0.000534</td>\n",
       "      <td>211846</td>\n",
       "      <td>0.284373</td>\n",
       "      <td>3349</td>\n",
       "      <td>0.004496</td>\n",
       "      <td>0.001879</td>\n",
       "      <td>0.118841</td>\n",
       "      <td>0.417906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>729</th>\n",
       "      <td>68</td>\n",
       "      <td>77</td>\n",
       "      <td>bulk grains rice dried goods</td>\n",
       "      <td>soft drinks</td>\n",
       "      <td>116</td>\n",
       "      <td>0.000156</td>\n",
       "      <td>3839</td>\n",
       "      <td>0.005153</td>\n",
       "      <td>60813</td>\n",
       "      <td>0.081633</td>\n",
       "      <td>0.030216</td>\n",
       "      <td>0.001907</td>\n",
       "      <td>0.370147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>3</td>\n",
       "      <td>124</td>\n",
       "      <td>energy granola bars</td>\n",
       "      <td>spirits</td>\n",
       "      <td>111</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>67726</td>\n",
       "      <td>0.090913</td>\n",
       "      <td>3349</td>\n",
       "      <td>0.004496</td>\n",
       "      <td>0.001639</td>\n",
       "      <td>0.033144</td>\n",
       "      <td>0.364572</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7324 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      aisle_A  aisle_B                  aisle_name_A  \\\n",
       "0          28       62                     red wines   \n",
       "40         62      134                   white wines   \n",
       "39         28      134                     red wines   \n",
       "41        124      134                       spirits   \n",
       "5415       27       28                 beers coolers   \n",
       "...       ...      ...                           ...   \n",
       "157        91      124               soy lactosefree   \n",
       "5368       28       50                     red wines   \n",
       "158       120      124                        yogurt   \n",
       "729        68       77  bulk grains rice dried goods   \n",
       "114         3      124           energy granola bars   \n",
       "\n",
       "                    aisle_name_B  freqAB  supportAB   freqA  supportA  freqB  \\\n",
       "0                    white wines    1022   0.001372    4408  0.005917   3995   \n",
       "40    specialty wines champagnes     236   0.000317    3995  0.005363   1297   \n",
       "39    specialty wines champagnes     221   0.000297    4408  0.005917   1297   \n",
       "41    specialty wines champagnes     155   0.000208    3349  0.004496   1297   \n",
       "5415                   red wines     808   0.001085    5438  0.007300   4408   \n",
       "...                          ...     ...        ...     ...       ...    ...   \n",
       "157                      spirits     264   0.000354  138303  0.185652   3349   \n",
       "5368      fruit vegetable snacks      83   0.000111    4408  0.005917  33070   \n",
       "158                      spirits     398   0.000534  211846  0.284373   3349   \n",
       "729                  soft drinks     116   0.000156    3839  0.005153  60813   \n",
       "114                      spirits     111   0.000149   67726  0.090913   3349   \n",
       "\n",
       "      supportB  confAtoB  confBtoA       lift  \n",
       "0     0.005363  0.231851  0.255820  43.233832  \n",
       "40    0.001741  0.059074  0.181958  33.930202  \n",
       "39    0.001741  0.050136  0.170393  28.796647  \n",
       "41    0.001741  0.046282  0.119507  26.583232  \n",
       "5415  0.005917  0.148584  0.183303  25.110871  \n",
       "...        ...       ...       ...        ...  \n",
       "157   0.004496  0.001909  0.078830   0.424608  \n",
       "5368  0.044392  0.018829  0.002510   0.424164  \n",
       "158   0.004496  0.001879  0.118841   0.417906  \n",
       "729   0.081633  0.030216  0.001907   0.370147  \n",
       "114   0.004496  0.001639  0.033144   0.364572  \n",
       "\n",
       "[7324 rows x 13 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = group_aisle_association_rules_dic[1].sort_values('lift',ascending = False)\n",
    "x.sort_values('lift',ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item_A</th>\n",
       "      <th>item_B</th>\n",
       "      <th>product_name_A</th>\n",
       "      <th>product_name_B</th>\n",
       "      <th>freqAB</th>\n",
       "      <th>supportAB</th>\n",
       "      <th>freqA</th>\n",
       "      <th>supportA</th>\n",
       "      <th>freqB</th>\n",
       "      <th>supportB</th>\n",
       "      <th>confAtoB</th>\n",
       "      <th>confBtoA</th>\n",
       "      <th>lift</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>21206</td>\n",
       "      <td>39365</td>\n",
       "      <td>English Breakfast Black Tea</td>\n",
       "      <td>French Vanilla Syrup</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>7212.333333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>35749</td>\n",
       "      <td>39365</td>\n",
       "      <td>Ginger Tea</td>\n",
       "      <td>French Vanilla Syrup</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>4327.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>35102</td>\n",
       "      <td>39365</td>\n",
       "      <td>The Original Liquid Coffee Creamer</td>\n",
       "      <td>French Vanilla Syrup</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>6</td>\n",
       "      <td>0.000277</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3606.166667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10312</td>\n",
       "      <td>39365</td>\n",
       "      <td>Zen Tea</td>\n",
       "      <td>French Vanilla Syrup</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>14</td>\n",
       "      <td>0.000647</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>0.214286</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1545.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>21206</td>\n",
       "      <td>35749</td>\n",
       "      <td>English Breakfast Black Tea</td>\n",
       "      <td>Ginger Tea</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>4327.400000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178864</th>\n",
       "      <td>13176</td>\n",
       "      <td>17619</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Premium Fish Sauce</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000231</td>\n",
       "      <td>3987</td>\n",
       "      <td>0.184268</td>\n",
       "      <td>16</td>\n",
       "      <td>0.000739</td>\n",
       "      <td>0.001254</td>\n",
       "      <td>0.312500</td>\n",
       "      <td>1.695902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178865</th>\n",
       "      <td>13176</td>\n",
       "      <td>45664</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Onion Powder</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3987</td>\n",
       "      <td>0.184268</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000462</td>\n",
       "      <td>0.000752</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>1.628066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178866</th>\n",
       "      <td>13176</td>\n",
       "      <td>17592</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Chinese Eggplant</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3987</td>\n",
       "      <td>0.184268</td>\n",
       "      <td>11</td>\n",
       "      <td>0.000508</td>\n",
       "      <td>0.000752</td>\n",
       "      <td>0.272727</td>\n",
       "      <td>1.480060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178867</th>\n",
       "      <td>13176</td>\n",
       "      <td>18272</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Slices Cheddar Cheese</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3987</td>\n",
       "      <td>0.184268</td>\n",
       "      <td>12</td>\n",
       "      <td>0.000555</td>\n",
       "      <td>0.000752</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>1.356722</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178868</th>\n",
       "      <td>13176</td>\n",
       "      <td>18660</td>\n",
       "      <td>Bag of Organic Bananas</td>\n",
       "      <td>Orange Marmalade</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000139</td>\n",
       "      <td>3987</td>\n",
       "      <td>0.184268</td>\n",
       "      <td>13</td>\n",
       "      <td>0.000601</td>\n",
       "      <td>0.000752</td>\n",
       "      <td>0.230769</td>\n",
       "      <td>1.252359</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>178869 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        item_A  item_B                      product_name_A  \\\n",
       "0        21206   39365         English Breakfast Black Tea   \n",
       "1        35749   39365                          Ginger Tea   \n",
       "2        35102   39365  The Original Liquid Coffee Creamer   \n",
       "3        10312   39365                             Zen Tea   \n",
       "4        21206   35749         English Breakfast Black Tea   \n",
       "...        ...     ...                                 ...   \n",
       "178864   13176   17619              Bag of Organic Bananas   \n",
       "178865   13176   45664              Bag of Organic Bananas   \n",
       "178866   13176   17592              Bag of Organic Bananas   \n",
       "178867   13176   18272              Bag of Organic Bananas   \n",
       "178868   13176   18660              Bag of Organic Bananas   \n",
       "\n",
       "               product_name_B  freqAB  supportAB  freqA  supportA  freqB  \\\n",
       "0        French Vanilla Syrup       3   0.000139      3  0.000139      3   \n",
       "1        French Vanilla Syrup       3   0.000139      5  0.000231      3   \n",
       "2        French Vanilla Syrup       3   0.000139      6  0.000277      3   \n",
       "3        French Vanilla Syrup       3   0.000139     14  0.000647      3   \n",
       "4                  Ginger Tea       3   0.000139      3  0.000139      5   \n",
       "...                       ...     ...        ...    ...       ...    ...   \n",
       "178864     Premium Fish Sauce       5   0.000231   3987  0.184268     16   \n",
       "178865           Onion Powder       3   0.000139   3987  0.184268     10   \n",
       "178866       Chinese Eggplant       3   0.000139   3987  0.184268     11   \n",
       "178867  Slices Cheddar Cheese       3   0.000139   3987  0.184268     12   \n",
       "178868       Orange Marmalade       3   0.000139   3987  0.184268     13   \n",
       "\n",
       "        supportB  confAtoB  confBtoA         lift  \n",
       "0       0.000139  1.000000  1.000000  7212.333333  \n",
       "1       0.000139  0.600000  1.000000  4327.400000  \n",
       "2       0.000139  0.500000  1.000000  3606.166667  \n",
       "3       0.000139  0.214286  1.000000  1545.500000  \n",
       "4       0.000231  1.000000  0.600000  4327.400000  \n",
       "...          ...       ...       ...          ...  \n",
       "178864  0.000739  0.001254  0.312500     1.695902  \n",
       "178865  0.000462  0.000752  0.300000     1.628066  \n",
       "178866  0.000508  0.000752  0.272727     1.480060  \n",
       "178867  0.000555  0.000752  0.250000     1.356722  \n",
       "178868  0.000601  0.000752  0.230769     1.252359  \n",
       "\n",
       "[178869 rows x 13 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "group_association_rules_dic[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "products_desc.set_index('product_id', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns top x items associated based on lift\n",
    "\n",
    "def product_to_product(cluster, product_id, item_lift, product_name, num_products):\n",
    "    df = group_association_rules_dic[cluster]\n",
    "    df = df[(df['item_A'] == product_id) | (df['item_B'] == product_id)][['product_name_A','item_A','product_name_B','item_B','confAtoB','lift']].sort_values('lift', ascending = False)\n",
    "    df = df[df['lift'] > item_lift]\n",
    "    df = df.sort_values('lift', ascending = False)\n",
    "    df = df.head(n = num_products)\n",
    "    product_to_product_associations = df['product_name_A'].values.tolist()\n",
    "    for x in df['product_name_B'].values.tolist():\n",
    "        product_to_product_associations.append(x)\n",
    "    product_to_product_associations = [x for x in product_to_product_associations if x != product_name]\n",
    "    return product_to_product_associations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns recommended products given inputs\n",
    "\n",
    "def pdp_recommender(user_id, product_id, item_lift, num_products):\n",
    "    product_name = products_desc.at[product_id,'product_name']\n",
    "    aisle_id = products_desc.at[product_id,'aisle_id']\n",
    "    aisle_name = products_desc.at[product_id,'aisle']\n",
    "    cluster = clustered_users.at[user_id, 'cluster']\n",
    "    return product_to_product(cluster = cluster, product_id = product_id, item_lift = item_lift, \n",
    "                              product_name = product_name, num_products = num_products)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "cluster 0 : user  148488\n",
      "Coffee House Blend Deep Roast Coffee K\n",
      "Ham, Parma, Prosciutto di Parma\n",
      "Birthday Cake\n",
      "Organic Frosted Wildberry Acai Toaster Pastries\n",
      "Salsa, Verde, Roasted\n",
      "\n",
      "cluster 1 : user  46545\n",
      "Thin & Light Tortilla Chips\n",
      "Original Hummus\n",
      "Limes\n",
      "Thick & Crispy Tortilla Chips\n",
      "Organic Hass Avocado\n",
      "\n",
      "cluster 2 : user  54727\n",
      "Roast Beef\n",
      "Clear Plastic Wrap\n",
      "Super Soft Taco Flour Tortillas\n",
      "Via Instant Dark French Roast Coffee\n",
      "Fit Popcorn Sea Salt\n",
      "\n",
      "cluster 3 : user  102113\n",
      "Real Guacamole\n",
      "Pure & Natural Sour Cream\n",
      "Thick & Crispy Tortilla Chips\n",
      "Original Taco Seasoning Mix\n",
      "Organic Avocado\n",
      "\n",
      "cluster 4 : user  180140\n",
      "Thin & Light Tortilla Chips\n",
      "Organic Large Brown Grade AA Cage Free Eggs\n",
      "Organic Reduced Fat 2% Milk\n",
      "Organic Whole String Cheese\n",
      "Thick & Crispy Tortilla Chips\n",
      "\n",
      "cluster 5 : user  149174\n",
      "Guacamole Seasoning Mix\n",
      "Thin & Light Tortilla Chips\n",
      "Diet Green Tea with Ginseng\n",
      "\n",
      "cluster 6 : user  51125\n",
      "Real Guacamole\n",
      "Chocolate Mint Builder's Bar\n",
      "Mild Guacamole\n",
      "Mango Flavor Probiotic Juice Drink\n",
      "Thick & Crispy Tortilla Chips\n",
      "\n",
      "cluster 7 : user  153476\n",
      "1% Low Fat Chocolate Milk\n",
      "Deli Style Mustard\n",
      "7 Ancient Grains with Hint of Sea Salt Crackers\n",
      "Organic Gluten-Free Quinoa Spaghetti\n",
      "Challah Twist\n",
      "\n",
      "cluster 8 : user  119472\n",
      "Totopos Chips\n",
      "Dark & Mint Filled Chocolate Squares\n",
      "The Peruvian Burrito\n",
      "Clementina Sparkling Beverage\n",
      "Organic Raspberry Mate Energy Drink\n",
      "\n",
      "cluster 9 : user  188714\n",
      "Organic Worcestershire\n",
      "Baked Sea Salt & Vinegar Potato Chips\n",
      "Real Guacamole\n",
      "Tortilla Triangles Restaurant Style Chips\n",
      "Summertime Blues Multigrain Tortilla Chips\n",
      "\n",
      "cluster 10 : user  12639\n",
      "\n",
      "\n",
      "cluster 11 : user  102397\n",
      "Baked Crackers, Cheddar Cheese\n",
      "Umcka ColdCare Soothing Syrup Cherry\n",
      "California White Balsamic Vinegar\n",
      "Peas and Potatoe Roti Paratha\n",
      "Apricot Mango Juice\n",
      "\n",
      "cluster 12 : user  18484\n",
      "Thin & Light Tortilla Chips\n",
      "Banana\n",
      "Thick & Crispy Tortilla Chips\n",
      "Organic Hass Avocado\n",
      "\n",
      "cluster 13 : user  125463\n",
      "Crumbled Cheese, Traditional, Feta, Fat Free\n",
      "Whole Wheat Pre-Sliced Mini Bagels\n",
      "Fruit Squish'ems! Squeezable Fruit Pouch Apple\n",
      "45 Calories & Delight Wheat Bread\n",
      "Dark Chocolate, Intense Mint\n",
      "\n",
      "cluster 14 : user  191053\n",
      "Thin & Light Tortilla Chips\n",
      "Boneless Skinless Chicken Breast\n",
      "Organic Large Brown Grade AA Cage Free Eggs\n",
      "Thick & Crispy Tortilla Chips\n",
      "Organic Sour Cream\n",
      "\n",
      "cluster 15 : user  133360\n",
      "Wonderful 100% Pomegranate Juice\n",
      "Frozen Limeade Concentrate\n",
      "Triple Sec\n",
      "Tortilla Chips Classic Yellow & Blue Family Value Pack\n",
      "D'Noir Prunes\n",
      "\n",
      "cluster 16 : user  9640\n",
      "Thins Light & Tasty Snack Crackers Lightly Salted\n",
      "Vegan Mac N Cheese\n",
      "Lavender Harvest Aromatherapy Mist\n",
      "Spinach And Cheese Ravioli\n",
      "No Salt Added Organic Mixed Vegetables\n",
      "\n",
      "cluster 17 : user  93685\n",
      "Gluten Free Spaghetti\n",
      "Pet Waste Bags\n",
      "Black Bean Bonanza Tamales\n",
      "Chicken-Free Strips Seasoned\n",
      "Gluten Free Dream Cookies Chocolate Vanilla Creme\n",
      "\n",
      "cluster 18 : user  180138\n",
      "Mango Tango Fruit Smoothie\n",
      "Pomelo Grapefruit & Aloe Appeal\n",
      "Pineapple\n",
      "Thin & Light Tortilla Chips\n",
      "Red Mango\n",
      "\n",
      "cluster 19 : user  88570\n",
      "Hearty Nine Grain Bread\n",
      "Munchable Falafel With Lemon Hummus\n",
      "Organic Unrefined Sesame Oil\n",
      "Very Berry Flavor Sparking Mineral Water\n",
      "Veggie Tray\n"
     ]
    }
   ],
   "source": [
    "# results for users in the 20 different clusters on prodcut 39055 Mild Roja Salsa, lift >1, 5 products\n",
    "\n",
    "for i in range(0,20):\n",
    "    user = clustered_users[clustered_users['cluster']==i].sample().index[0]\n",
    "    print('\\ncluster', i, ': user ', user)\n",
    "    print('\\n'.join([x for x in (pdp_recommender(user, 39055, 1, 5))]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}