6-cow-bloom-index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "44288d4b-1602-44ad-9881-f0e510a30d76",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Apache Hudi Core Conceptions (6) - COW: Bloom Index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d6261e3-fa15-410d-8472-29dd02f86b10",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## 1. Configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "57b9edaf-90a7-4b7c-b072-bc8c50c2e045",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:24:34.044582Z",
     "iopub.status.busy": "2023-03-04T08:24:34.044255Z",
     "iopub.status.idle": "2023-03-04T08:24:34.072165Z",
     "shell.execute_reply": "2023-03-04T08:24:34.071469Z",
     "shell.execute_reply.started": "2023-03-04T08:24:34.044553Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "Current session configs: <tt>{'conf': {'spark.jars': 'hdfs:///tmp/hudi-spark-bundle.jar', 'spark.serializer': 'org.apache.spark.serializer.KryoSerializer', 'spark.sql.extensions': 'org.apache.spark.sql.hudi.HoodieSparkSessionExtension', 'spark.sql.catalog.spark_catalog': 'org.apache.spark.sql.hudi.catalog.HoodieCatalog'}, 'proxyUser': 'user_bluishglc', 'kind': 'spark'}</tt><br>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody><tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>102</td><td>application_1677377031637_0141</td><td>spark</td><td>idle</td><td><a target=\"_blank\" href=\"http://ip-10-0-143-215.ec2.internal:20888/proxy/application_1677377031637_0141/\" class=\"emr-proxy-link j-2CS22PN3RJ0XE application_1677377031637_0141\" emr-resource=\"j-2CS22PN3RJ0XE\n",
       "\" application-id=\"application_1677377031637_0141\">Link</a></td><td><a target=\"_blank\" href=\"http://ip-10-0-134-70.ec2.internal:8042/node/containerlogs/container_1677377031637_0141_01_000001/livy\">Link</a></td><td>None</td><td></td></tr></tbody></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%configure -f\n",
    "{\n",
    "    \"conf\" : {\n",
    "        \"spark.jars\":\"hdfs:///tmp/hudi-spark-bundle.jar\",            \n",
    "        \"spark.serializer\":\"org.apache.spark.serializer.KryoSerializer\",\n",
    "        \"spark.sql.extensions\":\"org.apache.spark.sql.hudi.HoodieSparkSessionExtension\",\n",
    "        \"spark.sql.catalog.spark_catalog\":\"org.apache.spark.sql.hudi.catalog.HoodieCatalog\"\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "35143e14-7b24-450e-b029-c6ae0829e98f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:24:34.388975Z",
     "iopub.status.busy": "2023-03-04T08:24:34.388649Z",
     "iopub.status.idle": "2023-03-04T08:24:40.798006Z",
     "shell.execute_reply": "2023-03-04T08:24:40.796920Z",
     "shell.execute_reply.started": "2023-03-04T08:24:34.388947Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-r--r--   1 emr-notebook hdfsadmingroup   61421977 2023-03-04 08:24 /tmp/hudi-spark-bundle.jar\n",
      "/home/emr-notebook/hudi-stat.sh\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "# deploy hudi bundle jar\n",
    "hdfs dfs -copyFromLocal -f /usr/lib/hudi/hudi-spark-bundle.jar /tmp/hudi-spark-bundle.jar\n",
    "hdfs dfs -ls /tmp/hudi-spark-bundle.jar\n",
    "# deploy hudi-stat.sh - a utility shell script \n",
    "wget https://github.com/bluishglc/hudi-core-conceptions/releases/download/v1.0/hudi-stat.sh -O ~/hudi-stat.sh &>/dev/null\n",
    "chmod a+x ~/hudi-stat.sh\n",
    "ls ~/hudi-stat.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6bb44084-8afe-47fe-9454-b5a496275808",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:24:40.800126Z",
     "iopub.status.busy": "2023-03-04T08:24:40.799724Z",
     "iopub.status.idle": "2023-03-04T08:24:40.807013Z",
     "shell.execute_reply": "2023-03-04T08:24:40.805880Z",
     "shell.execute_reply.started": "2023-03-04T08:24:40.800083Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "table {float:left}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    "table {float:left}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00bdc485-3043-484c-8603-0e630184d445",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2. Test Case 1 - Gradually Increase"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d5d687b-51e2-4ed0-95e5-c70a1f8e8755",
   "metadata": {},
   "source": [
    "如果要观察到符合预期的文件创建和split行为，需保证如下两点：\n",
    "1. 每条记录的大小应基本相同 => 否则Hudi估算出的单条记录平均大小会很不准确，进而会影响到Hudi估算的记录数，输入数据的，以便进行split\n",
    "2. 手动设置记录的平均大小 => 便于Hudi准确估算记录数，已便进行符合预期的文件split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4901824d-58be-41ab-bd7a-0a394dfd0ca3",
   "metadata": {
    "tags": []
   },
   "source": [
    "[0MB, 100MB) -> [100MB, 120MB) -> [120MB, )\n",
    "\n",
    "KEY|VALUE\n",
    ":---|:---\n",
    "hoodie.index.type|SIMPLE\n",
    "hoodie.copyonwrite.record.size.estimate|175"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992230f3-5f68-4a33-954c-3e1e6e120f06",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-25T02:57:45.037101Z",
     "iopub.status.busy": "2023-02-25T02:57:45.036681Z",
     "iopub.status.idle": "2023-02-25T02:58:07.353493Z",
     "shell.execute_reply": "2023-02-25T02:58:07.352596Z",
     "shell.execute_reply.started": "2023-02-25T02:57:45.037060Z"
    },
    "tags": []
   },
   "source": [
    "### 2.1. Set Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "63bf2e47-d172-4f81-949c-2b46c408e99e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:24:40.809536Z",
     "iopub.status.busy": "2023-03-04T08:24:40.809150Z",
     "iopub.status.idle": "2023-03-04T08:25:12.616618Z",
     "shell.execute_reply": "2023-03-04T08:25:12.615748Z",
     "shell.execute_reply.started": "2023-03-04T08:24:40.809436Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting Spark application\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<tbody><tr><th>ID</th><th>YARN Application ID</th><th>Kind</th><th>State</th><th>Spark UI</th><th>Driver log</th><th>User</th><th>Current session?</th></tr><tr><td>103</td><td>application_1677377031637_0142</td><td>spark</td><td>idle</td><td><a target=\"_blank\" href=\"http://ip-10-0-143-215.ec2.internal:20888/proxy/application_1677377031637_0142/\" class=\"emr-proxy-link j-2CS22PN3RJ0XE application_1677377031637_0142\" emr-resource=\"j-2CS22PN3RJ0XE\n",
       "\" application-id=\"application_1677377031637_0142\">Link</a></td><td><a target=\"_blank\" href=\"http://ip-10-0-134-70.ec2.internal:8042/node/containerlogs/container_1677377031637_0142_01_000001/livy\">Link</a></td><td>None</td><td>✔</td></tr></tbody></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparkSession available as 'spark'.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c9d4cd4ed23c4b85a109ab28aa77a4c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b15b289eb6064e29af9d03d9d75aeff9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_NAME=reviews_cow_bloom_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3cb3dd19-a55e-4ebd-ab6d-cb1c2cdef2b3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:12.619097Z",
     "iopub.status.busy": "2023-03-04T08:25:12.618442Z",
     "iopub.status.idle": "2023-03-04T08:25:12.624615Z",
     "shell.execute_reply": "2023-03-04T08:25:12.623575Z",
     "shell.execute_reply.started": "2023-03-04T08:25:12.619059Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_NAME=reviews_cow_bloom_1\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_NAME=reviews_cow_bloom_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "950433ab-5ccd-400c-936b-77eaa8476dea",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:12.626564Z",
     "iopub.status.busy": "2023-03-04T08:25:12.625997Z",
     "iopub.status.idle": "2023-03-04T08:25:13.630009Z",
     "shell.execute_reply": "2023-03-04T08:25:13.628940Z",
     "shell.execute_reply.started": "2023-03-04T08:25:12.626528Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3c1b4337418e44f3b2d38e01578ad5f1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c082f61d9834886b99e114ce5c8869e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7247b331-25f9-4900-87a1-6e7c3a4b149a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:13.631742Z",
     "iopub.status.busy": "2023-03-04T08:25:13.631425Z",
     "iopub.status.idle": "2023-03-04T08:25:13.638682Z",
     "shell.execute_reply": "2023-03-04T08:25:13.636619Z",
     "shell.execute_reply.started": "2023-03-04T08:25:13.631711Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_1\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ba3031f-f92c-486d-a0de-3f141410f2bf",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2.2 Create Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bdcd022c-8a54-463d-9f3e-fbfb92969328",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:13.640392Z",
     "iopub.status.busy": "2023-03-04T08:25:13.640083Z",
     "iopub.status.idle": "2023-03-04T08:25:19.425663Z",
     "shell.execute_reply": "2023-03-04T08:25:19.424748Z",
     "shell.execute_reply.started": "2023-03-04T08:25:13.640361Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reviews_cow_bloom_1\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "echo $(basename $TABLE_PATH)\n",
    "aws s3 rm $TABLE_PATH --recursive &>/dev/null\n",
    "rm -rf ~/${TABLE_NAME}\n",
    "sleep 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2dd45b7f-cce2-4aec-99ec-7359d6d060e7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:19.428504Z",
     "iopub.status.busy": "2023-03-04T08:25:19.428239Z",
     "iopub.status.idle": "2023-03-04T08:25:20.848223Z",
     "shell.execute_reply": "2023-03-04T08:25:20.847390Z",
     "shell.execute_reply.started": "2023-03-04T08:25:19.428475Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6092bdb71615495395febdf9da252d2c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5c2b8e44da2f47019a739abef40492c2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "drop table if exists ${TABLE_NAME}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f740f107-0ab8-46b7-bdd3-206571a40c09",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:20.850474Z",
     "iopub.status.busy": "2023-03-04T08:25:20.849882Z",
     "iopub.status.idle": "2023-03-04T08:25:26.208126Z",
     "shell.execute_reply": "2023-03-04T08:25:26.207199Z",
     "shell.execute_reply.started": "2023-03-04T08:25:20.850437Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0f5d707772f642e1b73f90a369ef4cde",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4fcea6e6942842d68c69381e71757d02",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "create table if not exists ${TABLE_NAME}(\n",
    "    review_id string, \n",
    "    star_rating int, \n",
    "    review_body string, \n",
    "    review_date date, \n",
    "    year long,\n",
    "    timestamp long,\n",
    "    parity int\n",
    ")\n",
    "using hudi\n",
    "location '${TABLE_PATH}'\n",
    "partitioned by (parity)\n",
    "options ( \n",
    "    type = 'cow',  \n",
    "    primaryKey = 'review_id', \n",
    "    preCombineField = 'timestamp',\n",
    "    hoodie.index.type = 'BLOOM',\n",
    "    hoodie.copyonwrite.record.size.estimate = '175'\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26d91fec-b0b0-4bc0-b295-71798ac42766",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.3. Batch 1 - Insert ( 0 -> 96MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "de1fac92-fdc7-403a-a491-e11510324f64",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:25:26.209853Z",
     "iopub.status.busy": "2023-03-04T08:25:26.209515Z",
     "iopub.status.idle": "2023-03-04T08:26:23.781593Z",
     "shell.execute_reply": "2023-03-04T08:26:23.780354Z",
     "shell.execute_reply.started": "2023-03-04T08:25:26.209818Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "87d414344a8c4905a0cb44d365a7d448",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8d93f40c309a4dc89e615cfc1f643db8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2003;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9adfa455-d9ea-4527-8d0e-942dc97a114e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:26:23.783348Z",
     "iopub.status.busy": "2023-03-04T08:26:23.782993Z",
     "iopub.status.idle": "2023-03-04T08:26:43.982337Z",
     "shell.execute_reply": "2023-03-04T08:26:43.981256Z",
     "shell.execute_reply.started": "2023-03-04T08:26:23.783310Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230304082527210 │ commit │ COMPLETED │ 03-04 08:25 │ 03-04 08:25 │ 03-04 08:26 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230304082527210 │ 191.6 MB            │ 2                 │ 0                   │ 2                        │ 1155127               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bloom_1\n",
      "├── [ 96M 08:26:43]  parity=0\n",
      "│   └── [ 96M 08:26:10]  907c39dc-258f-488d-9d94-7112c19bc4d5-0_0-35-297_20230304082527210.parquet\n",
      "└── [ 96M 08:26:43]  parity=1\n",
      "    └── [ 96M 08:26:10]  a2346c69-d592-425b-9f74-376eb839195c-0_1-35-298_20230304082527210.parquet\n",
      "\n",
      " 192M used in 2 directories, 2 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36dd0b25-0283-405f-b991-9e78dae4b353",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.4. Batch 2 - Insert ( 96MB -> 110MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "06b721c3-1364-4622-ae23-d590e55a8da4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:27:44.317439Z",
     "iopub.status.busy": "2023-03-04T08:27:44.317094Z",
     "iopub.status.idle": "2023-03-04T08:28:43.820160Z",
     "shell.execute_reply": "2023-03-04T08:28:43.819302Z",
     "shell.execute_reply.started": "2023-03-04T08:27:44.317404Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "991b83c36c1f4857aa866b0030a8c49c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e08ae0de6894481a91660ff4ac82e162",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 1998;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "314c257b-00e9-48ea-a5a6-50e89d871d8f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:28:43.822108Z",
     "iopub.status.busy": "2023-03-04T08:28:43.821774Z",
     "iopub.status.idle": "2023-03-04T08:29:04.464092Z",
     "shell.execute_reply": "2023-03-04T08:29:04.463205Z",
     "shell.execute_reply.started": "2023-03-04T08:28:43.822071Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230304082527210 │ commit │ COMPLETED │ 03-04 08:25 │ 03-04 08:25 │ 03-04 08:26 ║\n",
      "╟─────┼───────────────────┼────────┼───────────┼─────────────┼─────────────┼─────────────╢\n",
      "║ 1   │ 20230304082751852 │ commit │ COMPLETED │ 03-04 08:27 │ 03-04 08:28 │ 03-04 08:28 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230304082751852 │ 220.9 MB            │ 0                 │ 2                   │ 2                        │ 1333111               │ 0                            │ 0            ║\n",
      "╟───────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢\n",
      "║ 20230304082527210 │ 191.6 MB            │ 2                 │ 0                   │ 2                        │ 1155127               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bloom_1\n",
      "├── [206M 08:29:04]  parity=0\n",
      "│   ├── [ 96M 08:26:10]  907c39dc-258f-488d-9d94-7112c19bc4d5-0_0-35-297_20230304082527210.parquet\n",
      "│   └── [110M 08:28:32]  907c39dc-258f-488d-9d94-7112c19bc4d5-0_0-83-378_20230304082751852.parquet\n",
      "└── [207M 08:29:04]  parity=1\n",
      "    ├── [ 96M 08:26:10]  a2346c69-d592-425b-9f74-376eb839195c-0_1-35-298_20230304082527210.parquet\n",
      "    └── [111M 08:28:32]  a2346c69-d592-425b-9f74-376eb839195c-0_1-83-379_20230304082751852.parquet\n",
      "\n",
      " 413M used in 2 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e56f987-da37-4cc5-9aa4-bfbabc80c2dd",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.5. Batch 3 - Insert ( 110MB -> 113.7MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9acff543-7a63-4f5b-9d0b-ce1752a095c5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:29:04.466079Z",
     "iopub.status.busy": "2023-03-04T08:29:04.465835Z",
     "iopub.status.idle": "2023-03-04T08:29:25.925924Z",
     "shell.execute_reply": "2023-03-04T08:29:25.925063Z",
     "shell.execute_reply.started": "2023-03-04T08:29:04.466053Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "731d801e23e844dfb6523f2f7ce44a8e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "53eb4cfed13b4e848e2869ed8472af79",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 1997;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "9f8e8282-9766-4c23-a301-0a0143023368",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:29:25.927775Z",
     "iopub.status.busy": "2023-03-04T08:29:25.927419Z",
     "iopub.status.idle": "2023-03-04T08:29:43.116525Z",
     "shell.execute_reply": "2023-03-04T08:29:43.114766Z",
     "shell.execute_reply.started": "2023-03-04T08:29:25.927738Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230304082527210 │ commit │ COMPLETED │ 03-04 08:25 │ 03-04 08:25 │ 03-04 08:26 ║\n",
      "╟─────┼───────────────────┼────────┼───────────┼─────────────┼─────────────┼─────────────╢\n",
      "║ 1   │ 20230304082751852 │ commit │ COMPLETED │ 03-04 08:27 │ 03-04 08:28 │ 03-04 08:28 ║\n",
      "╟─────┼───────────────────┼────────┼───────────┼─────────────┼─────────────┼─────────────╢\n",
      "║ 2   │ 20230304082906058 │ commit │ COMPLETED │ 03-04 08:29 │ 03-04 08:29 │ 03-04 08:29 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230304082906058 │ 7.4 MB              │ 2                 │ 0                   │ 2                        │ 39868                 │ 0                            │ 0            ║\n",
      "╟───────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢\n",
      "║ 20230304082751852 │ 220.9 MB            │ 0                 │ 2                   │ 2                        │ 1333111               │ 0                            │ 0            ║\n",
      "╟───────────────────┼─────────────────────┼───────────────────┼─────────────────────┼──────────────────────────┼───────────────────────┼──────────────────────────────┼──────────────╢\n",
      "║ 20230304082527210 │ 191.6 MB            │ 2                 │ 0                   │ 2                        │ 1155127               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bloom_1\n",
      "├── [210M 08:29:42]  parity=0\n",
      "│   ├── [3.7M 08:29:17]  03943188-51da-4e59-951d-83c46828e2fc-0_0-134-496_20230304082906058.parquet\n",
      "│   ├── [ 96M 08:26:10]  907c39dc-258f-488d-9d94-7112c19bc4d5-0_0-35-297_20230304082527210.parquet\n",
      "│   └── [110M 08:28:32]  907c39dc-258f-488d-9d94-7112c19bc4d5-0_0-83-378_20230304082751852.parquet\n",
      "└── [210M 08:29:42]  parity=1\n",
      "    ├── [3.7M 08:29:17]  4fe2b50f-9ad5-43f9-9899-caf84e2329f3-0_1-134-497_20230304082906058.parquet\n",
      "    ├── [ 96M 08:26:10]  a2346c69-d592-425b-9f74-376eb839195c-0_1-35-298_20230304082527210.parquet\n",
      "    └── [111M 08:28:32]  a2346c69-d592-425b-9f74-376eb839195c-0_1-83-379_20230304082751852.parquet\n",
      "\n",
      " 420M used in 2 directories, 6 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b832040c-6a81-4b9a-a6fa-965313940356",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3. Test Case 2 - Sharply Increasing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f19b71f-ebbc-4af5-a235-b5f111cf72f7",
   "metadata": {},
   "source": [
    "如果要观察到符合预期的文件创建和split行为，需保证如下两点：\n",
    "1. 每条记录的大小应基本相同 => 否则Hudi估算出的单条记录平均大小会很不准确，进而会影响到Hudi估算的记录数，输入数据的，以便进行split\n",
    "2. 手动设置记录的平均大小 => 便于Hudi准确估算记录数，已便进行符合预期的文件split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "259735eb-d471-4c31-b6e3-6102f7297a7f",
   "metadata": {
    "tags": []
   },
   "source": [
    "[0MB, 100MB) -> [100MB, 120MB) -> [120MB, )\n",
    "\n",
    "KEY|VALUE\n",
    ":---|:---\n",
    "hoodie.index.type|SIMPLE\n",
    "hoodie.copyonwrite.record.size.estimate|175"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "332765f5-2be6-49b0-b11f-6b4c3d9b5e9d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-25T02:57:45.037101Z",
     "iopub.status.busy": "2023-02-25T02:57:45.036681Z",
     "iopub.status.idle": "2023-02-25T02:58:07.353493Z",
     "shell.execute_reply": "2023-02-25T02:58:07.352596Z",
     "shell.execute_reply.started": "2023-02-25T02:57:45.037060Z"
    },
    "tags": []
   },
   "source": [
    "### 3.1. Set Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0be8a353-f169-4096-ad37-4bec08309af1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:12.620586Z",
     "iopub.status.busy": "2023-03-04T08:31:12.620251Z",
     "iopub.status.idle": "2023-03-04T08:31:20.129588Z",
     "shell.execute_reply": "2023-03-04T08:31:20.128838Z",
     "shell.execute_reply.started": "2023-03-04T08:31:12.620556Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7a7df6aadee04b75b7b52220899cea87",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a54b4729c8ae4ce4bd78309779c436d5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_NAME=reviews_cow_bloom_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "94797c17-719c-4133-953b-4f72c4f102f4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:20.130984Z",
     "iopub.status.busy": "2023-03-04T08:31:20.130745Z",
     "iopub.status.idle": "2023-03-04T08:31:20.135926Z",
     "shell.execute_reply": "2023-03-04T08:31:20.134894Z",
     "shell.execute_reply.started": "2023-03-04T08:31:20.130961Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_NAME=reviews_cow_bloom_2\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_NAME=reviews_cow_bloom_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d1d33218-e1fa-440a-a1c9-2edaa560aaaa",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:20.138323Z",
     "iopub.status.busy": "2023-03-04T08:31:20.137918Z",
     "iopub.status.idle": "2023-03-04T08:31:21.054477Z",
     "shell.execute_reply": "2023-03-04T08:31:21.053690Z",
     "shell.execute_reply.started": "2023-03-04T08:31:20.138292Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6891f7cefeb541ea88b594799eecd913",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(children=(HTML(value='Type:'), Button(description='Table', layout=Layout(width='70px'), st…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "baba5b4c489d47cf929b35f81d62c511",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "set TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "93fdac3a-12fd-4a47-9bc3-5d6d109f6460",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:21.056427Z",
     "iopub.status.busy": "2023-03-04T08:31:21.056106Z",
     "iopub.status.idle": "2023-03-04T08:31:21.061163Z",
     "shell.execute_reply": "2023-03-04T08:31:21.060413Z",
     "shell.execute_reply.started": "2023-03-04T08:31:21.056396Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_2\n"
     ]
    }
   ],
   "source": [
    "%env TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45083c14-437f-4d78-9f1b-d58d11a85323",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3.2 Create Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "78638153-aada-4e19-94a3-645eda5c806d",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:21.062907Z",
     "iopub.status.busy": "2023-03-04T08:31:21.062454Z",
     "iopub.status.idle": "2023-03-04T08:31:26.577687Z",
     "shell.execute_reply": "2023-03-04T08:31:26.576466Z",
     "shell.execute_reply.started": "2023-03-04T08:31:21.062873Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "reviews_cow_bloom_2\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "echo $(basename $TABLE_PATH)\n",
    "aws s3 rm $TABLE_PATH --recursive &>/dev/null\n",
    "rm -rf ~/${TABLE_NAME}\n",
    "sleep 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "da793b97-2434-4fc5-9aeb-60447ff9b120",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:26.579340Z",
     "iopub.status.busy": "2023-03-04T08:31:26.579019Z",
     "iopub.status.idle": "2023-03-04T08:31:26.978200Z",
     "shell.execute_reply": "2023-03-04T08:31:26.977294Z",
     "shell.execute_reply.started": "2023-03-04T08:31:26.579295Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c910c4141c7a448f9a26fba9e3287ab5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c6407f2f0e374c6496e9b06aa417acd2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "drop table if exists ${TABLE_NAME}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "40c8db16-f433-4838-b75b-04e4f986aef9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:26.979652Z",
     "iopub.status.busy": "2023-03-04T08:31:26.979404Z",
     "iopub.status.idle": "2023-03-04T08:31:30.321723Z",
     "shell.execute_reply": "2023-03-04T08:31:30.320817Z",
     "shell.execute_reply.started": "2023-03-04T08:31:26.979628Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1ab1d321d794bca94584c6484247cfc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ad51fd93b324bc990c0ecab0dc3709c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "create table if not exists ${TABLE_NAME}(\n",
    "    review_id string, \n",
    "    star_rating int, \n",
    "    review_body string, \n",
    "    review_date date, \n",
    "    year long,\n",
    "    timestamp long,\n",
    "    parity int\n",
    ")\n",
    "using hudi\n",
    "location '${TABLE_PATH}'\n",
    "partitioned by (parity)\n",
    "options ( \n",
    "    type = 'cow',  \n",
    "    primaryKey = 'review_id', \n",
    "    preCombineField = 'timestamp',\n",
    "    hoodie.index.type = 'BLOOM',\n",
    "    hoodie.copyonwrite.record.size.estimate = '175'\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bad4bcc-7fd4-408b-b10d-78def052bee9",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3.3. Batch 1 - Insert ( 0 -> 341MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7e51bc42-66e2-422a-bee8-5e6dbf7c7ae1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:31:30.323849Z",
     "iopub.status.busy": "2023-03-04T08:31:30.323611Z",
     "iopub.status.idle": "2023-03-04T08:32:45.927187Z",
     "shell.execute_reply": "2023-03-04T08:32:45.926273Z",
     "shell.execute_reply.started": "2023-03-04T08:31:30.323826Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FloatProgress(value=0.0, bar_style='info', description='Progress:', layout=Layout(height='25px', width='50%'),…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>.jp-OutputArea-child .jupyter-widgets.jp-OutputArea-output { overflow: auto; }</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "72d2e9d056964a71bdb5ed4dbf7930ae",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(HBox(), EncodingWidget(children=(VBox(children=(HTML(value='Encoding:'), Dropdown(description='…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bef9bc9b3c654c55bd4925d70956a812",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Output()"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2010;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "56b5c7a0-78e5-4df8-9e01-571418bc4dcf",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-03-04T08:32:45.929213Z",
     "iopub.status.busy": "2023-03-04T08:32:45.928682Z",
     "iopub.status.idle": "2023-03-04T08:33:13.259631Z",
     "shell.execute_reply": "2023-03-04T08:33:13.258481Z",
     "shell.execute_reply.started": "2023-03-04T08:32:45.929175Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[ TIMELINE ]\n",
      "\n",
      "╔═════╤═══════════════════╤════════╤═══════════╤═════════════╤═════════════╤═════════════╗\n",
      "║ No. │ Instant           │ Action │ State     │ Requested   │ Inflight    │ Completed   ║\n",
      "║     │                   │        │           │ Time        │ Time        │ Time        ║\n",
      "╠═════╪═══════════════════╪════════╪═══════════╪═════════════╪═════════════╪═════════════╣\n",
      "║ 0   │ 20230304083130910 │ commit │ COMPLETED │ 03-04 08:31 │ 03-04 08:32 │ 03-04 08:32 ║\n",
      "╚═════╧═══════════════════╧════════╧═══════════╧═════════════╧═════════════╧═════════════╝\n",
      "\n",
      "[ COMMITS ]\n",
      "\n",
      "╔═══════════════════╤═════════════════════╤═══════════════════╤═════════════════════╤══════════════════════════╤═══════════════════════╤══════════════════════════════╤══════════════╗\n",
      "║ CommitTime        │ Total Bytes Written │ Total Files Added │ Total Files Updated │ Total Partitions Written │ Total Records Written │ Total Update Records Written │ Total Errors ║\n",
      "╠═══════════════════╪═════════════════════╪═══════════════════╪═════════════════════╪══════════════════════════╪═══════════════════════╪══════════════════════════════╪══════════════╣\n",
      "║ 20230304083130910 │ 682.6 MB            │ 6                 │ 0                   │ 2                        │ 4119699               │ 0                            │ 0            ║\n",
      "╚═══════════════════╧═════════════════════╧═══════════════════╧═════════════════════╧══════════════════════════╧═══════════════════════╧══════════════════════════════╧══════════════╝\n",
      "\n",
      "[ STORAGE ]\n",
      "\n",
      "/home/emr-notebook/reviews_cow_bloom_2\n",
      "├── [341M 08:33:12]  parity=0\n",
      "│   ├── [103M 08:32:29]  74d3fe01-5885-4a9b-9575-79aae94a379e-0_2-194-672_20230304083130910.parquet\n",
      "│   ├── [119M 08:32:32]  a22b8f61-1e48-4c26-a011-bc016b426816-0_0-194-670_20230304083130910.parquet\n",
      "│   └── [119M 08:32:32]  e3de4018-225d-478d-94df-69a603558f2a-0_1-194-671_20230304083130910.parquet\n",
      "└── [341M 08:33:13]  parity=1\n",
      "    ├── [103M 08:32:29]  0649537d-5f14-49ee-a1ef-fbaf95a72120-0_5-194-675_20230304083130910.parquet\n",
      "    ├── [119M 08:32:32]  6259366b-2be6-4d06-9c31-59da0f62c144-0_3-194-673_20230304083130910.parquet\n",
      "    └── [119M 08:32:33]  9c33f6e0-6309-4b70-bf52-d10138f771df-0_4-194-674_20230304083130910.parquet\n",
      "\n",
      " 683M used in 2 directories, 6 files\n"
     ]
    }
   ],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c383c0c9-9598-4bf3-af10-1953d9589fe9",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 4. Test Case 3 - Default Settings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d89b866-ae98-4e5d-af2a-8b54c91fb377",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-02-25T02:57:45.037101Z",
     "iopub.status.busy": "2023-02-25T02:57:45.036681Z",
     "iopub.status.idle": "2023-02-25T02:58:07.353493Z",
     "shell.execute_reply": "2023-02-25T02:58:07.352596Z",
     "shell.execute_reply.started": "2023-02-25T02:57:45.037060Z"
    },
    "tags": []
   },
   "source": [
    "### 4.1. Set Variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "245f76a0-54af-41ea-b6ce-34189dfda1da",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "set TABLE_NAME=reviews_cow_bloom_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0dd248d6-df36-4846-b0f2-f53a1cc891e6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env TABLE_NAME=reviews_cow_bloom_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4735cded-4783-4726-809a-3ad4b3a65624",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "set TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97bd92c7-c04d-4e39-ab00-76979219303d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%env TABLE_PATH=s3://glc-examples/hudi-core-conceptions/reviews_cow_bloom_3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "873fbb59-c0db-4a57-925c-e91702062fa5",
   "metadata": {},
   "source": [
    "## 4.2 Create Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6985b21-f9d9-4fec-9d6b-cd7c3ff346d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "echo $(basename $TABLE_PATH)\n",
    "aws s3 rm $TABLE_PATH --recursive &>/dev/null\n",
    "rm -rf ~/${TABLE_NAME}\n",
    "sleep 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86661be6-2a20-427d-bf33-246567affd7d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "drop table if exists ${TABLE_NAME}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8c6e3bf-e01e-428d-8d26-bffbc4313f68",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "create table if not exists ${TABLE_NAME}(\n",
    "    review_id string, \n",
    "    star_rating int, \n",
    "    review_body string, \n",
    "    review_date date, \n",
    "    year long,\n",
    "    timestamp long,\n",
    "    parity int\n",
    ")\n",
    "using hudi\n",
    "location '${TABLE_PATH}'\n",
    "partitioned by (parity)\n",
    "options ( \n",
    "    type = 'cow',  \n",
    "    primaryKey = 'review_id', \n",
    "    preCombineField = 'timestamp',\n",
    "    hoodie.index.type = 'BLOOM'\n",
    "    -- hoodie.copyonwrite.record.size.estimate = '175'\n",
    ");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a3bd4c0-8cbb-44ef-aacf-880e54c7638c",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.3. Batch 1 - Insert ( 0 -> 96MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f8d4370-23da-4c05-b708-ffdb96147712",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2003;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45164a8c-726a-4949-b09e-e8f7c384c014",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37546d50-d336-408c-82b4-c7d33c5b3d89",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4.3. Batch 1 - Insert ( 96MB -> 417MB / Partition )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7b92fe2-0f7f-46d9-aead-6345b2a6b571",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sql\n",
    "insert into \n",
    "    ${TABLE_NAME}\n",
    "select \n",
    "    review_id, \n",
    "    star_rating, \n",
    "    review_body, \n",
    "    review_date, \n",
    "    year,\n",
    "    unix_timestamp(current_timestamp()) as timestamp,\n",
    "    mod(crc32(review_id), 2) as parity\n",
    "from\n",
    "    reviews\n",
    "where\n",
    "    year = 2010;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9cc764a-2a72-4364-910b-2c6af64e6d1c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%sh\n",
    "~/hudi-stat.sh $TABLE_PATH timeline commits storage"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Spark",
   "language": "scala",
   "name": "sparkkernel"
  },
  "language_info": {
   "codemirror_mode": "text/x-scala",
   "file_extension": ".sc",
   "mimetype": "text/x-scala",
   "name": "scala",
   "pygments_lexer": "scala"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}