tensorflow_programming.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Programming\n",
    "\n",
    "**TF** offers different features, e.g.,\n",
    "- ***Estimators API*** (less favorable alternative to `tf.keras`),\n",
    "- data preprocessing and loading (e.g., `tf.data`, `tf.io`),\n",
    "- image processing ops (`tf.image`),\n",
    "- signal processing ops (`tf.signal`), etc.\n",
    "\n",
    "**TensorFlow Lite** enables mobile device implementation:\n",
    " - ***TensorFlow.js*** for the JavaScript implementation in browsers,\n",
    " - ***TensorFlow Hub*** for downloading of pretrained NN, and\n",
    " - ***TensorFlow Extended (TFX)*** for machine learning production, e.g,\n",
    "   - data validation,\n",
    "   - preprocessing,\n",
    "   - model analysis, and\n",
    "   - serving (using ***TF Serving***).\n",
    "\n",
    "Each TF operation (i.e., `op`), at the lowest level, is developed by using highly efficient C++ code that can be customized. Each op has its own implementation (i.e., **kernel**) for each device, e.g., ***TPUs*** (tensor processing units; especially designed ASIC chips for Deep Learning)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Python ≥3.5 is required\n",
    "import sys\n",
    "assert sys.version_info >= (3, 5)\n",
    "\n",
    "# Scikit-Learn ≥0.20 is required\n",
    "import sklearn\n",
    "assert sklearn.__version__ >= \"0.20\"\n",
    "\n",
    "try:\n",
    "    # %tensorflow_version only exists in Colab.\n",
    "    %tensorflow_version 2.x\n",
    "except Exception:\n",
    "    pass\n",
    "\n",
    "# TensorFlow ≥2.4 is required in this notebook\n",
    "# Earlier 2.x versions will mostly work the same, but with a few bugs\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "# assert tf.__version__ >= \"2.4\"\n",
    "\n",
    "# Common imports\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "# to make this notebook's output stable across runs\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "# To plot pretty figures\n",
    "%matplotlib inline\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "mpl.rc('axes', labelsize=14)\n",
    "mpl.rc('xtick', labelsize=12)\n",
    "mpl.rc('ytick', labelsize=12)\n",
    "\n",
    "# Where to save the figures\n",
    "PROJECT_ROOT_DIR = \".\"\n",
    "CHAPTER_ID = \"deep\"\n",
    "IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, \"images\", CHAPTER_ID)\n",
    "os.makedirs(IMAGES_PATH, exist_ok=True)\n",
    "\n",
    "def save_fig(fig_id, tight_layout=True, fig_extension=\"png\", resolution=300):\n",
    "    path = os.path.join(IMAGES_PATH, fig_id + \".\" + fig_extension)\n",
    "    print(\"Saving figure\", fig_id)\n",
    "    if tight_layout:\n",
    "        plt.tight_layout()\n",
    "    plt.savefig(path, format=fig_extension, dpi=resolution)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data structures\n",
    "### Immutable tensors\n",
    "- Tensors are similar to the `ndarray` data type but\n",
    "  - it can be also scalar, and\n",
    "  - it uses the 32-bit floats by default (as 64-bit is not typically neccessary).\n",
    "- They can be created by `tf.constant()` and support indexing, and other operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=42>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# scalar tensor creation\n",
    "\n",
    "tf.constant(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([2, 3])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# matrix tensor creation\n",
    "\n",
    "t = tf.constant([[1., 2., 3.], [4., 5., 6.]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([2, 3]), tf.float32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ops similar to numpy ndarray\n",
    "\n",
    "t.shape, t.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[2., 3.],\n",
       "       [5., 6.]], dtype=float32)>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# indexing\n",
    "\n",
    "t[:, 1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 1), dtype=float32, numpy=\n",
       "array([[2.],\n",
       "       [5.]], dtype=float32)>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# adding an extra dimension\n",
    "\n",
    "t[..., 1, tf.newaxis]\n",
    "\n",
    "# in Python \"...\" means \"all dimensions behind the next\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n",
       "array([[14., 32.],\n",
       "       [32., 77.]], dtype=float32)>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# other ops\n",
    "\n",
    "t + 10\n",
    "tf.square(t)\n",
    "t @ tf.transpose(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For portability you may use `keras.backend` instead, but it is not as complete as TF."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[11., 26.],\n",
       "       [14., 35.],\n",
       "       [19., 46.]], dtype=float32)>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tensorflow import keras\n",
    "\n",
    "K = keras.backend\n",
    "K.square(K.transpose(t)) + 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- TF does not handle type conversion automarically (for performance), which can be enforced by `tf.cast()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cannot compute AddV2 as input #1(zero-based) was expected to be a float tensor but is a int32 tensor [Op:AddV2]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.constant(2.0) + tf.constant(40)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=42.0>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = tf.constant(40., dtype=tf.float64)\n",
    "\n",
    "tf.constant(2.0) + tf.cast(t, tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Comparing to np, some functions may have different names mostly, e.g., `tf.reduce_mean()`, `tf.reduce_sum()`, `tf.reduce_max()`, as they use a reduce for the GPU kernel that can affect the results by changing the order, due to the limitation of 32-bit floats. Always pay attention to the dtypes.\n",
    "- Use `.numpy()` to convert it into numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2., 4., 5.])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = tf.constant(np.array([2., 4., 5.]))\n",
    "\n",
    "t.numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Strings\n",
    "- `tf.string` (byte string tensors) are manageable by the `tf.strings` package.\n",
    "- In python 3 each str is unicode (not byte); which might be enforced with \"b\" (but not always;  e.g., with é)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'hello world'>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(b\"hello world\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- TF in python 3 can automatically converte each str to UTF-8, which are atomic, i.e., its length can be appeared if converted to Unicode tensors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'caf\\xc3\\xa9'>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.constant(\"café\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233])>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For clarification\n",
    "\n",
    "u = tf.constant([ord(c) for c in \"café\"]) # ord() returns the Unicode characters.\n",
    "\n",
    "u"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=string, numpy=b'caf\\xc3\\xa9'>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = tf.strings.unicode_encode(u, \"UTF-8\")\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=4>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.length(b, unit=\"UTF8_CHAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([ 99,  97, 102, 233])>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.unicode_decode(b, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### String arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=int32, numpy=array([4, 6, 5, 2])>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = tf.constant([\"Café\", \"Coffee\", \"caffè\", \"咖啡\"])\n",
    "\n",
    "tf.strings.length(p, unit=\"UTF8_CHAR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.RaggedTensor [[67, 97, 102, 233], [67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232], [21654, 21857]]>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = tf.strings.unicode_decode(p, \"UTF8\")\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ragged tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A `tf.RaggedTensor` is a tensor, managed by the `tf.ragged` package, with at least one *ragged dimension* (i.e., a dimension with slices of different lengths.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([ 67 111 102 102 101 101], shape=(6,), dtype=int32)\n",
      "<tf.RaggedTensor [[67, 111, 102, 102, 101, 101], [99, 97, 102, 102, 232]]>\n"
     ]
    }
   ],
   "source": [
    "print(r[1])\n",
    "print(r[1:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<tf.RaggedTensor [[67, 97, 102, 233, 68, 69, 70], [67, 111, 102, 102, 101, 101, 71], [99, 97, 102, 102, 232], [21654, 21857, 72, 73]]>\n"
     ]
    }
   ],
   "source": [
    "r3 = tf.ragged.constant([[68, 69, 70], [71], [], [72, 73]])\n",
    "print(tf.concat([r, r3], axis=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4,), dtype=string, numpy=array([b'DEF', b'G', b'', b'HI'], dtype=object)>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.strings.unicode_encode(r3, \"UTF-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(4, 6), dtype=int32, numpy=\n",
       "array([[   67,    97,   102,   233,     0,     0],\n",
       "       [   67,   111,   102,   102,   101,   101],\n",
       "       [   99,    97,   102,   102,   232,     0],\n",
       "       [21654, 21857,     0,     0,     0,     0]])>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.to_tensor()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sparse tensors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `tf.SparseTensor` is a mostly zero-valued tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 1]\n",
      " [1 0]\n",
      " [2 3]], shape=(3, 2), dtype=int64), values=tf.Tensor([1. 2. 3.], shape=(3,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "s = tf.SparseTensor(indices=[[0, 1], [1, 0], [2, 3]],\n",
    "                    values=[1., 2., 3.],\n",
    "                    dense_shape=[3, 4])\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 1., 0., 0.],\n",
       "       [2., 0., 0., 0.],\n",
       "       [0., 0., 0., 3.]], dtype=float32)>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sparse.to_dense(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "s2 = s * 2.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "unsupported operand type(s) for +: 'SparseTensor' and 'float'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    s3 = s + 1.\n",
    "except TypeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[ 30.,  40.],\n",
       "       [ 20.,  40.],\n",
       "       [210., 240.]], dtype=float32)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s4 = tf.constant([[10., 20.], [30., 40.], [50., 60.], [70., 80.]])\n",
    "tf.sparse.sparse_dense_matmul(s, s4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SparseTensor(indices=tf.Tensor(\n",
      "[[0 2]\n",
      " [0 1]], shape=(2, 2), dtype=int64), values=tf.Tensor([1. 2.], shape=(2,), dtype=float32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))\n"
     ]
    }
   ],
   "source": [
    "s5 = tf.SparseTensor(indices=[[0, 2], [0, 1]],\n",
    "                     values=[1., 2.],\n",
    "                     dense_shape=[3, 4])\n",
    "print(s5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "indices[1] = [0,1] is out of order. Many sparse ops require sorted indices.\n",
      "    Use `tf.sparse.reorder` to create a correctly ordered copy.\n",
      "\n",
      " [Op:SparseToDense]\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    tf.sparse.to_dense(s5)\n",
    "except tf.errors.InvalidArgumentError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 4), dtype=float32, numpy=\n",
       "array([[0., 2., 1., 0.],\n",
       "       [0., 0., 0., 0.],\n",
       "       [0., 0., 0., 0.]], dtype=float32)>"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s6 = tf.sparse.reorder(s5)\n",
    "tf.sparse.to_dense(s6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sets\n",
    "- `tf.sets` is a package to work with sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 6), dtype=int32, numpy=\n",
       "array([[ 2,  3,  4,  5,  6,  7],\n",
       "       [ 0,  7,  9, 10,  0,  0]])>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set1 = tf.constant([[2, 3, 5, 7], [7, 9, 0, 0]])\n",
    "set2 = tf.constant([[4, 5, 6], [9, 10, 0]])\n",
    "tf.sparse.to_dense(tf.sets.union(set1, set2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.sparse_tensor.SparseTensor at 0x1bd3bf3ee48>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sets.difference(set1, set2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n",
       "array([[5, 0],\n",
       "       [0, 9]])>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.sparse.to_dense(tf.sets.intersection(set1, set2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variables\n",
    "\n",
    "- `tf.Variable` is similar to `tf.Tensor` but can also update its value by `assign()`, or relatively by `assign_add()` or `assign_sub()`.\n",
    "- Individual cells and slices can also be changed by assigning to them directly or by `scatter_nd_update()`.\n",
    "- In practice, with Keras we hardly need them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = tf.Variable([[1., 2., 3.], [4., 5., 6.]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2.,  4.,  6.],\n",
       "       [ 8., 10., 12.]], dtype=float32)>"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.assign(2 * v)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2., 42.,  6.],\n",
       "       [ 8., 10., 12.]], dtype=float32)>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[0, 1].assign(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[ 2., 42.,  0.],\n",
       "       [ 8., 10.,  1.]], dtype=float32)>"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v[:, 2].assign([0., 1.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[100.,  42.,   0.],\n",
       "       [  8.,  10., 200.]], dtype=float32)>"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.scatter_nd_update(indices=[[0, 0], [1, 2]],\n",
    "                    updates=[100., 200.])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For indexing, we may also use `.scatter_update` based on the `tf.IndexedSlices` to define the new values and their indices (by a sparse representaion that imporves perfomance for big data). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Variable 'UnreadVariable' shape=(2, 3) dtype=float32, numpy=\n",
       "array([[  1.,   2.,   3.],\n",
       "       [  8.,  10., 200.]], dtype=float32)>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sparse_delta = tf.IndexedSlices(values=[[1., 2., 3.]],\n",
    "                                indices=[0])\n",
    "v.scatter_update(sparse_delta)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensor Arrays\n",
    "\n",
    "They are per-time-step and write-once tensor arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "array = tf.TensorArray(dtype=tf.float32, size=3)\n",
    "array = array.write(0, tf.constant([1., 2.]))\n",
    "array = array.write(1, tf.constant([3., 10.]))\n",
    "array = array.write(2, tf.constant([5., 7.]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([ 3., 10.], dtype=float32)>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array.read(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3, 2), dtype=float32, numpy=\n",
       "array([[1., 2.],\n",
       "       [0., 0.],\n",
       "       [5., 7.]], dtype=float32)>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "array.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(2,), dtype=float32, numpy=array([2., 3.], dtype=float32)>,\n",
       " <tf.Tensor: shape=(2,), dtype=float32, numpy=array([4.6666665, 8.666667 ], dtype=float32)>)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean, variance = tf.nn.moments(array.stack(), axes=0)\n",
    "mean, variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Queues\n",
    "**Queuing** is the TF method of threading, to handle different tasks asynchronously. `tf.queue` is a package dealing with storing tensors in multiple steps mostly for high performance customized parallel code, e.g. using,\n",
    "- First In, First Out (FIFO) queues (`FIFOQueue`), \n",
    "- queues that can prioritize some (`PriorityQueue`),\n",
    "- or shuffle them (`RandomShuffleQueue`),\n",
    "- and batch items with different shapes using padding (`PaddingFIFOQueue`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TF Functions and Graphs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `tf.function()` or `@tf.function` will make an optimized computation graph (from python functions). However, when using Keras models the graph is generated automatically (and we do not need these methods). To prohibit Keras from this:\n",
    "  - use `dynamic=True` in the custom layers and methods, or\n",
    "  - alternatively use `run_eagerly=True` when compiling.\n",
    "- Use TF functions instead of other functions if possible as calling `tf.py_function()` for many times, may hinder the performance and portability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def cube(x):\n",
    "    return x ** 3\n",
    "\n",
    "cube(2) # the output is not TF tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cube(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.eager.def_function.Function at 0x2875e40b848>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube = tf.function(cube)\n",
    "tf_cube"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# alternatively\n",
    "\n",
    "@tf.function\n",
    "def tf_cube(x):\n",
    "    return x ** 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=8>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube(2) # note the output is of int32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_cube.python_function(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AutoGraph and Tracing\n",
    "\n",
    "- TF first in the **AutoGraph** process analyzes the Python function’s source code to capture control flow statements, e.g., for loops. Then, an upgraded TF version of the code is generated (see, e.g., figure below).\n",
    "- During **tracing**, TF calls this function, passes a symbolic tensor to it, and make a node for it without computation (i.e., running in **graph mode** in contrary to the regular or **eager mode**).\n",
    " \n",
    "![alt text for screen readers](\\img\\graphtracing.jpg)\n",
    "Figure 1. Example of AutoGraph and Tracing.\n",
    "\n",
    "- Once a function is traced, TF creates a new ***concrete function***, can be accessed by `.get_concrete_function()`, which can also be used to explore its graph using `.graph.get_operations()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.func_graph.FuncGraph at 0x1bd5f573208>"
      ]
     },
     "execution_count": 212,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function = tf_cube.get_concrete_function(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.framework.func_graph.FuncGraph at 0x1bd5f573208>"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'pow/y' type=Const>,\n",
       " <tf.Operation 'pow' type=Pow>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ops = concrete_function.graph.get_operations()\n",
    "ops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'x:0' shape=() dtype=float32>,\n",
       " <tf.Tensor 'pow/y:0' shape=() dtype=float32>]"
      ]
     },
     "execution_count": 222,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pow_op = ops[2]\n",
    "list(pow_op.inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'pow:0' shape=() dtype=float32>]"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pow_op.outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Operation 'x' type=Placeholder>"
      ]
     },
     "execution_count": 223,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph.get_operation_by_name('x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor 'Identity:0' shape=() dtype=float32>"
      ]
     },
     "execution_count": 217,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.graph.get_tensor_by_name('Identity:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"__inference_cube_1908399\"\n",
       "input_arg {\n",
       "  name: \"x\"\n",
       "  type: DT_FLOAT\n",
       "}\n",
       "output_arg {\n",
       "  name: \"identity\"\n",
       "  type: DT_FLOAT\n",
       "}"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concrete_function.function_def.signature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- TF Functions can handle ***polymorphism*** (i.e., varying argument types and shapes) only by updating the graphs (happening only if the tensor size or shape, or python values are chagned). Therefore,\n",
    "  - delete redundant TF Functions to release RAM space, and\n",
    "  - use python values only for unique values, e.g., hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def tf_cube(x):\n",
    "    print(\"print:\", x)\n",
    "    return x ** 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print: Tensor(\"x:0\", shape=(), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "result = tf_cube(tf.constant(2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=8.0>"
      ]
     },
     "execution_count": 227,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "print: 2\n",
      "print: 3\n",
      "print: Tensor(\"x:0\", shape=(1, 2), dtype=float32)\n",
      "print: Tensor(\"x:0\", shape=(2, 2), dtype=float32)\n",
      "WARNING:tensorflow:5 out of the last 5 calls to <function tf_cube at 0x000001BD63032A68> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n",
      "print: Tensor(\"x:0\", shape=(3, 2), dtype=float32)\n",
      "WARNING:tensorflow:6 out of the last 6 calls to <function tf_cube at 0x000001BD63032A68> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings is likely due to passing python objects instead of tensors. Also, tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. Please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details.\n"
     ]
    }
   ],
   "source": [
    "result = tf_cube(2)\n",
    "result = tf_cube(3)\n",
    "result = tf_cube(tf.constant([[1., 2.]])) # New shape: trace!\n",
    "result = tf_cube(tf.constant([[3., 4.], [5., 6.]])) # New shape: trace!\n",
    "result = tf_cube(tf.constant([[7., 8.], [9., 10.], [11., 12.]])) # New shape: trace!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We may also define `input_signature` for the TF function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function(input_signature=[tf.TensorSpec([None, 28, 28], tf.float32)])\n",
    "def shrink(images):\n",
    "    print(\"Tracing\", images)\n",
    "    return images[:, ::2, ::2] # drop half the rows and columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tracing Tensor(\"images:0\", shape=(None, 28, 28), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "img_batch_1 = tf.random.uniform(shape=[100, 28, 28])\n",
    "img_batch_2 = tf.random.uniform(shape=[50, 28, 28])\n",
    "preprocessed_images = shrink(img_batch_1) # Traces the function.\n",
    "preprocessed_images = shrink(img_batch_2) # Reuses the same concrete function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 232,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python inputs incompatible with input_signature:\n",
      "  inputs: (\n",
      "    tf.Tensor(\n",
      "[[[0.7413678  0.62854624]\n",
      "  [0.01738465 0.3431449 ]]\n",
      "\n",
      " [[0.51063764 0.3777541 ]\n",
      "  [0.07321596 0.02137029]]], shape=(2, 2, 2), dtype=float32))\n",
      "  input_signature: (\n",
      "    TensorSpec(shape=(None, 28, 28), dtype=tf.float32, name=None))\n"
     ]
    }
   ],
   "source": [
    "img_batch_3 = tf.random.uniform(shape=[2, 2, 2])\n",
    "try:\n",
    "    preprocessed_images = shrink(img_batch_3)  # rejects unexpected types or shapes\n",
    "except ValueError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- If in a loop `range()` is used instead of `tf.range()`, the loop cannot capture the graph (not a *dymaic* for loop) and runs only during tracing. It is not generally fine unless you prefer to make a new graph for each loop (e.g., when making a new layer in a network)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'add/y' type=Const>,\n",
       " <tf.Operation 'add' type=AddV2>,\n",
       " <tf.Operation 'add_1/y' type=Const>,\n",
       " <tf.Operation 'add_1' type=AddV2>,\n",
       " <tf.Operation 'add_2/y' type=Const>,\n",
       " <tf.Operation 'add_2' type=AddV2>,\n",
       " <tf.Operation 'add_3/y' type=Const>,\n",
       " <tf.Operation 'add_3' type=AddV2>,\n",
       " <tf.Operation 'add_4/y' type=Const>,\n",
       " <tf.Operation 'add_4' type=AddV2>,\n",
       " <tf.Operation 'add_5/y' type=Const>,\n",
       " <tf.Operation 'add_5' type=AddV2>,\n",
       " <tf.Operation 'add_6/y' type=Const>,\n",
       " <tf.Operation 'add_6' type=AddV2>,\n",
       " <tf.Operation 'add_7/y' type=Const>,\n",
       " <tf.Operation 'add_7' type=AddV2>,\n",
       " <tf.Operation 'add_8/y' type=Const>,\n",
       " <tf.Operation 'add_8' type=AddV2>,\n",
       " <tf.Operation 'add_9/y' type=Const>,\n",
       " <tf.Operation 'add_9' type=AddV2>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A \"static\" for loop using range():\n",
    "\n",
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in range(10):\n",
    "        x += 1\n",
    "    return x\n",
    "\n",
    "add_10.get_concrete_function(tf.constant(5)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 236,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'Const' type=Const>,\n",
       " <tf.Operation 'while/maximum_iterations' type=Const>,\n",
       " <tf.Operation 'while/loop_counter' type=Const>,\n",
       " <tf.Operation 'while' type=StatelessWhile>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 236,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A \"dynamic\" loop using tf.while_loop():\n",
    "\n",
    "@tf.function\n",
    "def add_10(x):\n",
    "    condition = lambda i, x: tf.less(i, 10)\n",
    "    body = lambda i, x: (tf.add(i, 1), tf.add(x, 1))\n",
    "    final_i, final_x = tf.while_loop(condition, body, [tf.constant(0), x])\n",
    "    return final_x\n",
    "\n",
    "add_10.get_concrete_function(tf.constant(5)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 238,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Operation 'x' type=Placeholder>,\n",
       " <tf.Operation 'range/start' type=Const>,\n",
       " <tf.Operation 'range/limit' type=Const>,\n",
       " <tf.Operation 'range/delta' type=Const>,\n",
       " <tf.Operation 'range' type=Range>,\n",
       " <tf.Operation 'while/maximum_iterations' type=Const>,\n",
       " <tf.Operation 'while/loop_counter' type=Const>,\n",
       " <tf.Operation 'while' type=StatelessWhile>,\n",
       " <tf.Operation 'Identity' type=Identity>]"
      ]
     },
     "execution_count": 238,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# A \"dynamic\" for loop using tf.range() (captured by autograph):\n",
    "\n",
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in tf.range(10):\n",
    "        x = x + 1\n",
    "    return x\n",
    "\n",
    "add_10.get_concrete_function(tf.constant(0)).graph.get_operations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Calling functions from other libraries is not part of a graph, and therefore, it happens only for tracing. Some implications are:\n",
    "  - If your non-TF code has a side effects (e.g., updating a python counter), they only occurs during tracing (and not for the other TF function calls).\n",
    "  - E.g., if for `np.random.rand()` and `tf.random.rand()`, we pass constants of the same shape and types but different values, the NumPy random numbers are the same while the TF ones are different."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"counter\"\n",
       "type: DT_RESOURCE"
      ]
     },
     "execution_count": 248,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# E.g.,\n",
    "\n",
    "counter = tf.Variable(0)\n",
    "\n",
    "@tf.function\n",
    "def increment(counter, c=1):\n",
    "    return counter.assign_add(c)\n",
    "\n",
    "increment(counter)\n",
    "increment(counter)\n",
    "\n",
    "function_def = increment.get_concrete_function(counter).function_def\n",
    "function_def.signature.input_arg[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name: \"assignaddvariableop_resource\"\n",
       "type: DT_RESOURCE"
      ]
     },
     "execution_count": 246,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "counter = tf.Variable(0)\n",
    "\n",
    "@tf.function\n",
    "def increment(c=1):\n",
    "    return counter.assign_add(c)\n",
    "\n",
    "increment()\n",
    "increment()\n",
    "\n",
    "function_def = increment.get_concrete_function().function_def\n",
    "function_def.signature.input_arg[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=int32, numpy=2>"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class Counter:\n",
    "    def __init__(self):\n",
    "        self.counter = tf.Variable(0)\n",
    "\n",
    "    @tf.function\n",
    "    def increment(self, c=1):\n",
    "        return self.counter.assign_add(c)\n",
    "\n",
    "c = Counter()\n",
    "c.increment()\n",
    "c.increment()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You may show the TF code by `tf.autograph.to_code()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "def tf__add_10(x):\n",
      "  do_return = False\n",
      "  retval_ = ag__.UndefinedReturnValue()\n",
      "  with ag__.FunctionScope('add_10', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:\n",
      "\n",
      "    def get_state():\n",
      "      return ()\n",
      "\n",
      "    def set_state(_):\n",
      "      pass\n",
      "\n",
      "    def loop_body(iterates, x):\n",
      "      i = iterates\n",
      "      x += 1\n",
      "      return x,\n",
      "    x, = ag__.for_stmt(ag__.converted_call(tf.range, (10,), None, fscope), None, loop_body, get_state, set_state, (x,), ('x',), ())\n",
      "    do_return = True\n",
      "    retval_ = fscope.mark_return_value(x)\n",
      "  do_return,\n",
      "  return ag__.retval(retval_)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "@tf.function\n",
    "def add_10(x):\n",
    "    for i in tf.range(10):\n",
    "        x += 1\n",
    "    return x\n",
    "\n",
    "print(tf.autograph.to_code(add_10.python_function))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "```python\n",
       "def tf__add_10(x):\n",
       "  do_return = False\n",
       "  retval_ = ag__.UndefinedReturnValue()\n",
       "  with ag__.FunctionScope('add_10', 'fscope', ag__.ConversionOptions(recursive=True, user_requested=True, optional_features=(), internal_convert_user_code=True)) as fscope:\n",
       "\n",
       "    def get_state():\n",
       "      return ()\n",
       "\n",
       "    def set_state(_):\n",
       "      pass\n",
       "\n",
       "    def loop_body(iterates, x):\n",
       "      i = iterates\n",
       "      x += 1\n",
       "      return x,\n",
       "    x, = ag__.for_stmt(ag__.converted_call(tf.range, (10,), None, fscope), None, loop_body, get_state, set_state, (x,), ('x',), ())\n",
       "    do_return = True\n",
       "    retval_ = fscope.mark_return_value(x)\n",
       "  do_return,\n",
       "  return ag__.retval(retval_)\n",
       "\n",
       "```"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# alternatively\n",
    "\n",
    "def display_tf_code(func):\n",
    "    from IPython.display import display, Markdown\n",
    "    if hasattr(func, \"python_function\"):\n",
    "        func = func.python_function\n",
    "    code = tf.autograph.to_code(func)\n",
    "    display(Markdown('```python\\n{}\\n```'.format(code)))\n",
    "\n",
    "display_tf_code(add_10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other points:\n",
    "- Python source code should be accessible (e.g., in python shell or the compiled *.pyc files do not have such access).\n",
    "- Use the vectorized version when possible.\n",
    "- In a function, TF variables and stateful objects (e.g., TF datasets or queues) should be created upon the very first call. Generally, it is more preferred to make them outside functions (e.g., in the `build()` method of a custom layer).\n",
    "\n",
    "### Using TF Functions with tf.keras (or Not)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, tf.keras does not need tf.function():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom loss function\n",
    "def my_mse(y_true, y_pred):\n",
    "    print(\"Tracing loss my_mse()\")\n",
    "    return tf.reduce_mean(tf.square(y_pred - y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom metric function\n",
    "def my_mae(y_true, y_pred):\n",
    "    print(\"Tracing metric my_mae()\")\n",
    "    return tf.reduce_mean(tf.abs(y_pred - y_true))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom layer\n",
    "class MyDense(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=None, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.activation = keras.activations.get(activation)\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.kernel = self.add_weight(name='kernel', \n",
    "                                      shape=(input_shape[1], self.units),\n",
    "                                      initializer='uniform',\n",
    "                                      trainable=True)\n",
    "        self.biases = self.add_weight(name='bias', \n",
    "                                      shape=(self.units,),\n",
    "                                      initializer='zeros',\n",
    "                                      trainable=True)\n",
    "        super().build(input_shape)\n",
    "\n",
    "    def call(self, X):\n",
    "        print(\"Tracing MyDense.call()\")\n",
    "        return self.activation(X @ self.kernel + self.biases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing metric my_mae()\n",
      "Tracing loss my_mse()\n",
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "10496/11610 [==========================>...] - ETA: 0s - loss: 1.3640 - my_mae: 0.8114Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "11610/11610 [==============================] - 1s 94us/sample - loss: 1.2838 - my_mae: 0.7826 - val_loss: 0.4503 - val_my_mae: 0.4879\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.4418 - my_mae: 0.4782 - val_loss: 0.7718 - val_my_mae: 0.4583\n",
      "5160/5160 [==============================] - 0s 22us/sample - loss: 0.4174 - my_mae: 0.4584\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4173873734566592, 0.45841503]"
      ]
     },
     "execution_count": 258,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Custom model\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "class MyModel(keras.models.Model):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden1 = MyDense(30, activation=\"relu\")\n",
    "        self.hidden2 = MyDense(30, activation=\"relu\")\n",
    "        self.output_ = MyDense(1)\n",
    "\n",
    "    def call(self, input):\n",
    "        print(\"Tracing MyModel.call()\")\n",
    "        hidden1 = self.hidden1(input)\n",
    "        hidden2 = self.hidden2(hidden1)\n",
    "        concat = keras.layers.concatenate([input, hidden2])\n",
    "        output = self.output_(concat)\n",
    "        return output\n",
    "\n",
    "model = MyModel()\n",
    "model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae])\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n",
      "Tracing MyModel.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing MyDense.call()\n",
      "Tracing loss my_mse()\n",
      "Tracing metric my_mae()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[5.50464653968811, 2.056248]"
      ]
     },
     "execution_count": 260,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can turn this off by creating the model with dynamic=True\n",
    "# (or calling super().__init__(dynamic=True, **kwargs) in the model's constructor):\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = MyModel(dynamic=True)\n",
    "model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae])\n",
    "\n",
    "# alternatively use run_eagerly=True\n",
    "# model = MyModel()\n",
    "# model.compile(loss=my_mse, optimizer=\"nadam\", metrics=[my_mae], run_eagerly=True)\n",
    "\n",
    "model.fit(X_train_scaled[:64], y_train[:64], epochs=1,\n",
    "          validation_data=(X_valid_scaled[:64], y_valid[:64]), verbose=0)\n",
    "model.evaluate(X_test_scaled[:64], y_test[:64], verbose=0)\n",
    "\n",
    "# Note the custom code will be called at each iteration.\n",
    "# Let's use a tiny datasets to avoid getting too much output:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Gradients with Autodiff\n",
    "- `tf.GradientTape()` and its `gradient()` method can efficiently find the derivatives (comparing to the numerical methods) with respect to variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + 2 * w1 * w2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36.000003007075065"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# manually approximation of gradients\n",
    "\n",
    "w1, w2 = 5, 3\n",
    "eps = 1e-6\n",
    "(f(w1 + eps, w2) - f(w1, w2)) / eps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using autodiff\n",
    "\n",
    "w1, w2 = tf.Variable(5.), tf.Variable(3.)\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "gradients = tape.gradient(z, [w1, w2])\n",
    "\n",
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For vectors, TF finds the gradient of the vector sum (e.g., referring to the sum of the losses). To find specific component (i.e., partial derivate), we may use `persistent=True` finding one set of gradients (i.e., ***Jacobeans***)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GradientTape.gradient can only be called once on non-persistent tapes.\n"
     ]
    }
   ],
   "source": [
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "dz_dw1 = tape.gradient(z, w1)\n",
    "\n",
    "try:\n",
    "    dz_dw2 = tape.gradient(z, w2)\n",
    "except RuntimeError as ex:\n",
    "    print(ex)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "dz_dw1 = tape.gradient(z, w1)\n",
    "dz_dw2 = tape.gradient(z, w2) # works now!\n",
    "\n",
    "del tape # delete it to make the tape persistent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=136.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=30.0>]"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Another e.g.,\n",
    "\n",
    "with tf.GradientTape() as tape:\n",
    "    z1 = f(w1, w2 + 2.)\n",
    "    z2 = f(w1, w2 + 5.)\n",
    "    z3 = f(w1, w2 + 7.)\n",
    "\n",
    "tape.gradient([z1, z2, z3], [w1, w2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(2,), dtype=float32, numpy=array([136.,  30.], dtype=float32)>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# alternatively\n",
    "\n",
    "with tf.GradientTape(persistent=True) as tape:\n",
    "    z1 = f(w1, w2 + 2.)\n",
    "    z2 = f(w1, w2 + 5.)\n",
    "    z3 = f(w1, w2 + 7.)\n",
    "\n",
    "tf.reduce_sum(tf.stack([tape.gradient(z, [w1, w2]) for z in (z1, z2, z3)]), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "del tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2nd order derivate e.g.,\n",
    "\n",
    "with tf.GradientTape(persistent=True) as hessian_tape: # hessian is the second order derivative of a vector\n",
    "    with tf.GradientTape() as jacobian_tape:\n",
    "        z = f(w1, w2)\n",
    "    jacobians = jacobian_tape.gradient(z, [w1, w2])\n",
    "hessians = [hessian_tape.gradient(jacobian, [w1, w2])\n",
    "            for jacobian in jacobians]\n",
    "del hessian_tape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "jacobians"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[<tf.Tensor: shape=(), dtype=float32, numpy=6.0>,\n",
       "  <tf.Tensor: shape=(), dtype=float32, numpy=2.0>],\n",
       " [<tf.Tensor: shape=(), dtype=float32, numpy=2.0>, None]]"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hessians"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `tf.stop_gradient()` can exclude some parts from backprop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=30.0>, None]"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(w1, w2):\n",
    "    return 3 * w1 ** 2 + tf.stop_gradient(2 * w1 * w2)\n",
    "\n",
    "with tf.GradientTape() as tape:\n",
    "    z = f(w1, w2)\n",
    "\n",
    "tape.gradient(z, [w1, w2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We can force the constants to be derivable by `watch()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(), dtype=float32, numpy=36.0>,\n",
       " <tf.Tensor: shape=(), dtype=float32, numpy=10.0>]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c1, c2 = tf.constant(5.), tf.constant(3.)\n",
    "\n",
    "with tf.GradientTape() as tape:\n",
    "    tape.watch(c1) # tf.constant does not have gradients unless we force them by tape.watch()\n",
    "    tape.watch(c2) # .watch is useful when\n",
    "    z = f(c1, c2)\n",
    "\n",
    "gradients = tape.gradient(z, [c1, c2])\n",
    "gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- With numerical issues (e.g., zero division) find the *analytical derivative* while decorating with `@tf.custom_gradient` and returning the increment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor: shape=(1,), dtype=float32, numpy=array([nan], dtype=float32)>]"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# e.g., zero devision:\n",
    "\n",
    "x = tf.Variable([100.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_softplus(x)\n",
    "\n",
    "tape.gradient(z, [x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.custom_gradient\n",
    "def my_better_softplus(z):\n",
    "    exp = tf.exp(z)\n",
    "    def my_softplus_gradients(grad):\n",
    "        return grad / (1 + 1 / exp)\n",
    "    return tf.math.log(exp + 1), my_softplus_gradients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(1,), dtype=float32, numpy=array([inf], dtype=float32)>,\n",
       " [<tf.Tensor: shape=(1,), dtype=float32, numpy=array([1.], dtype=float32)>])"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = tf.Variable([1000.])\n",
    "with tf.GradientTape() as tape:\n",
    "    z = my_better_softplus(x)\n",
    "\n",
    "z, tape.gradient(z, [x])\n",
    "\n",
    "# to avoid [inf] softplus output, you may define it by tf.where(z > 30., z, tf.math.log(tf.exp(z) + 1.))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customization\n",
    "Some of the follwing customizations may not be portable for some Keras versions.\n",
    "\n",
    "### Custom loss function\n",
    "E.g., if the data was too noisy, MSE and MAE may penalize respectively too much or too few, then `keras.losses.Huber()` may be used. We can also make our own, for this:\n",
    "- make the code vectorized, if possible;\n",
    "- use only TF objects;\n",
    "- try to return a tensor having one loss per instance (instead of the mean) so that keras can add sample (or class) weights later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# preprocessing\n",
    "\n",
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "housing = fetch_california_housing()\n",
    "X_train_full, X_test, y_train_full, y_test = train_test_split(\n",
    "    housing.data, housing.target.reshape(-1, 1), random_state=42)\n",
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train_full, y_train_full, random_state=42)\n",
    "\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)\n",
    "X_valid_scaled = scaler.transform(X_valid)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def huber_fn(y_true, y_pred):\n",
    "    error = y_true - y_pred\n",
    "    is_small_error = tf.abs(error) < 1\n",
    "    squared_loss = tf.square(error) / 2\n",
    "    linear_loss  = tf.abs(error) - 0.5\n",
    "    return tf.where(is_small_error, squared_loss, linear_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAd8AAAEDCAYAAAB0/A4MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8li6FKAAAgAElEQVR4nO3deZzN9ffA8dd7xsQMY8seIhmyS5GQJaKS9r1QiVS/LGlR2lGEUIg2pCyVULJkjbKU4kuK7PtuzIxZzPL+/XFmGMuYOzP33s9dzvPxuI+ZufOZ+zmfuTP33M/nfd7nbay1KKWUUsp7QpwOQCmllAo2mnyVUkopL9Pkq5RSSnmZJl+llFLKyzT5KqWUUl6myVcppZTyMk2+SvkQY0wLY4w1xpTw0v46G2PivLEvpdQZmnyVyiNjzHhjzI8XuP+a9ERayftRKaV8mSZfpYKAMeYSp2NQSp2hyVcpL7nQJWVjTKX0+645Z/PrjDFrjTGJxpg1xpgG5zzW9caYpcaYeGPMXmPMGGNM4UzfX5J+3xBjzGHg1xzE2c0Ys8UYcyr945MX+P7m9NgOG2PmGWPypX+vtjFmoTEmxhgTa4xZZ4xpmZPfk1LBQJOvUr5pCPAScA2wDZhtjIkASXDAfGAWUBe4C6gHfH7OYzwCGKAZ0NGVnRpj7gQ+AoYDtYARwGhjzG3p378GGAW8BVQDWgNzMz3E18B+oCFQH3gTSHT5qJUKEvmcDkCpANHuAoVLeXlz+461dh6AMeYxYA/wEPAp8AIw1Vo7NGNjY0x34C9jTClr7aH0u7dba5/P4X77AF9aaz9K/3pz+ln3S8APQEXgJDDLWhsL7ATWZfr5y4Eh1tp/07/eksP9KxUU9MxXKff4BTn7zHx7KA+PtyLjE2ttHLAeqJF+VwPgEWNMXMaNM5eVq2R6jDW52O9VnH+Jenmmff+MJNztxpivjDGdjDGRmbYdBnxqjFlkjHnVGFM9FzEoFfA0+SrlHvHW2i2Zb8jZamZp6R9NpvvCcrGvEOQMOHOirwtUBdZm2u5kLh4b4EJLnVmA9LPdq4H7gF1AX+BfY0y59O+/iSTqGcD1wP+MMY/nMg6lApYmX6W853D6x7KZ7quXxbbXZXxijCmIjL/+k37Xn0DNc5N9+i0hjzH+AzQ9576mwMaML6y1KdbaRdbavkAdoCDQPtP3/7PWjrTW3gp8BnTJY0xKBRwd81XKe7YAu4E3jTEvA5WAflls2y+9Snkf8DpwCilmAhgErDTGfAyMBWKB6sBt1tpueYzxfeAbY8wapKirHfAwUtSFMaY9cmn7F+AY0BKIBP4xxoQjhWLfADuA0kjiXpXHmJQKOJp8lfISa22yMeYBYDRSpLQWeAU4r0EH8DIwFKko/htob609mf44/zPG3AD0B5YCoUhF9PduiHGGMeb/kMKr4cj47tPW2h/SN4kG7kDeEEQAW4Eu1tpl6XOJiwETgDLA0fRj65PXuJQKNMbaCw3vKKWUUspTdMxXKaWU8rIcJV9jTNX0rjaTPBWQUkopFehyeuY7CvjdE4EopZRSwcLl5JteKBINLPRcOEoppVTgcyn5pjdsfxvIaas6pZRSSp3D1alG7wCfWWt3G2Oy3MgY0xXoClCgQIEGFStWzHuEPiotLY2QkKzfu1gLR4/mp0SJJC9G5R7ZHZu/C+Tj2717N9Zagvl/z9/54/ElJoaSP38axlx89ow/HltObN68+Yi1tqQr22abfI0x9ZCVS+pnt621dhwwDqBatWp206ZNrsTgl5YsWUKLFi2y3S4+HiIiPB+PO7l6bP4qkI+vRYsWREdHs3bt2uw39lOB/PyB/x3fypVQsyZERma/rb8dW04ZY3a6uq0rb0FaIJ14dhljDiAT5u82xvyZq+iCyPHjUKcOpKQ4HYlSSnnGhAmwb5/TUfgfVy47jwOmZPq6D5KMu3sioEBSrBhs2AD5tI+YUipAjRnjdAT+KdszX2ttvLX2QMYNiAMSrbWHs/tZBcnJ8MorMgaslFKB5Pbb4e+/nY7CP+X4nCx9yTDlokKFoEIFufQclpvF45RSykd98AEEcG2fRwVu2ZmPMAa6d4cDB5yORCml3Oe776BoUR1Wyy1Nvl6QkAC33ioflVIqEKxapcNpeaHvWbwgPBzWrZOzYKWU8nfJyTB4sNNR+Dc98/WS1FS4/36Z96uUUv7KWqhfH/budToS/6Znvl6SLx906aLjI0op/2YM/PYbFC7sdCT+Tc98vah1a1i9WsdJlFL+69139TXMHTT5etkHH8BhnSGtlPJDqalwySVQsKDTkfg/vQjqRcZIeb5SSvmjI0fgeV3bzi30zNfLrIU2bWDXLqcjUUop1yUlQfPmEBfndCSBQc98vcwYGDUKypd3OhKllHJd/vywcSME8IqAXqW/RgdERcE33+i0I6WUfzh1Cjp1kvm9yj00+Tpk40Y4dMjpKJRSyjV33SVnv8o99LKzQ956SxZbsFY7XymlfNv69dChg9NRBBY983XQrbfCmjVOR6GUUlk7dkyWRU1LczqSwKJnvg6aOlVWBVFKKV9VvDjMm+d0FIFHz3wdVLQofPopbNrkdCRKKXW+3bvh5pu1o5UneCz5RkfryvGuKFZMS/eVUr6pXDkYMkTrUlzxxRc5295jL/uHDhWgRw9pR6aydvfdUKYMxMQ4HYlSSp1x4gTMmgU1azodiW9LS4O+feHxx3P2cx495xo5Em6/HWJjPbkX/9evH8yd63QUSil1xsGDMiVSZS0+XpaKfe89CA3N2c96LPlWqBBP8eIwezY0ayZjB+rChg+H++5zOgqllBKpqVClCrz6qtOR+K4DB6BlS/j2W1le8aefcvbzHku+4eGprFol3ZzWrYNGjXRaTVaMgYkT4auvnI5EKaVg/nzo2NHpKHzXhg2S01avhkqVZH3jm27K2WN49LLzlVfCihXQogXs3y9nwDNmeHKP/uvaa+H6652OQimlpMJ5zBino/BNc+fKa/WuXXDddbByZe7GxT1eZ5sxR6xzZ0hIkBZlQ4Zo6fq5rrrqTONypZRyyqpV0oOgcGGnI/E9o0dLc6TYWBkqXLQISpfO3WN5ZZLLJZfA55/DwIGSdF94Abp10ybd5/r1V1iyxOkolFLBLCJCpkCqM1JToVcveOYZqW7u1w8mT4bw8Nw/ptc6XBkj5dhXXiljCZ98Atu2yWC1dnkS994rH7Xfs1LKCUeOSKFV7dpOR+I74uLgwQfhxx8hLExyV6dOeX9cr7d3uPdeObsrVQoWLoTGjSUJK/HTT9C9u9NRKKWC0TffwAcfOB2F79izR2qVfvxRrgb8/LN7Ei841Nu5USMZV2jfHv7+W76eOVMLjkCe6OuuczoKpVQw6t5d63Ey/Pkn3HYb7NsHVatKAo6Kct/jO9bYsFIlGeNs21YudbRqJdfQg11kpFzm+O47pyNRSgWTkSPlJEiHvOT30KyZJN4bbpBZO+5MvODwwgpFisi7ie7dISkJHnoI3n5b33mlpsrlDqWU8pZ27aBePaejcJa1MGwY3HmndK/q2FHmPF96qfv35XhL/3z5YNQoGWcwBt54Qw44KcnpyJxTuTL06CG9VZVSytNWr5YxzcsvdzoS5yQny4ng889LEu7fH8aPlymgnuB48gVJuj17yql+wYIwaRK0bi2Xo4PV9u1yKT7YrwIopTxv3jz47z+no3DOiRNSgzR2rCTbKVOktaYnL8H7RPLNcNttsGwZXHYZLF8uhUfButZt5coyzqDjL0opT0pLg9deC96C1x075Njnz4eSJWHxYlkswdN8KvkC1K8vldD168PWrZKAFy92OipnnDoFTz6pyzIqpTznxhtl1kkwWrlSZtts3Ag1akjuadzYO/v2ueQLcua7bBl06ADR0dKwOqcLFQeCggWlCCItzelIlFKBaupUSTzBZto0WZXo0CFo00Zm31Su7L39+2TyBUk806fL4HdKiixU3LdvcCUiY6QX9po1OvarlHK/wYNljDOYhreslVbH998PiYnQtassfevtTos+m3xBFiceMgQ+/lg+f+89+YXFxzsdmXcNHixrRyqllLukpEgiKljQ6Ui859QpeOyxM8VUQ4dKfgkL834sPp18M3TrBnPmyCob334rlwqCJRkZI1cAypRxOhKlVCA5dAheekmmewaDY8dkCHPCBFk8Yvp06N3bubN+v0i+INfkV6yQzlirV8sg+fr1TkflPffdB3/84XQUSqlAcOyYrNmbkuJ0JN7x339SvLt0KZQtC7/8Anfc4WxMLiVfY8wkY8x+Y0yMMWazMaaLpwO7kBo1pDrtuutkIeMmTWRh42Dw/vtw9dVOR6GUCgTFi8PatcFx1vvLL5Iz/vsP6taVk7cGDZyOyvUz33eBStbawkAHoL8xxpHwS5eWBYzvv18WNL71VlngONBVqiQrHgXrvGellHts2wZdugRHkdXEidKw6dgxaaKxfDmUL+90VMKl5Gut/dtam9Hw0abfqngsqmyEh8PXX8uCxmlpssBxz56BPx/2xAl5w6GUUrlVpgw88YTTUXhWRuOQTp2kbWSPHjBjBhQq5HRkZxjr4hwWY8xooDMQDvwF3GCtjTtnm65AV4CSJUs2mDZtmluDvZB580ozZEg1UlJCaNz4CP36/UNEhOezcFxcHIUceCZTUw1JSSEePUanjs1bAvn4evbsSWpqKh9++KHToXhMID9/4NnjO348jP37w6lRI8Yjj58dbzx3p06F8N571Vm8uBQhIZZnn/2PO+/c59F9ZmjZsuUaa+01Lm1srXX5BoQCTYF+QNjFto2KirLesnSptcWLWwvW1q1r7e7dnt/n4sWLPb+TC+jf39qhQz27D6eOzVsC+fiaN29u69at63QYHhXIz5+1nj2+FSvkNcQpnn7uDh60tnFjyQWRkdbOmePR3Z0H+MO6mE9zVO1srU211i4HygPdc/SWwINuuEEKsapWhXXroGFDaUwRiF58UcrjlVIqJ1JSpPDo1VedjsQzNm6UWTArVkDFitKxql07p6PKWm6nGuXDwTHfC6laVRJw8+awf78k5BkznI7K/cLC5I9q0CCnI1FK+ZMRI6RRUSD6+WfpybxjB1x7rfRorl3b6aguLtvka4wpZYx5wBhTyBgTaoxpCzwILPJ8eDlTvLisTNGpk3TBuusu6WASaK0Zr7gCWrRwOgqllD/p0UPWqw00n3wic5ZjYuDuu2HJEv9oSuTKma9FLjHvAY4DQ4Ce1tqZngwsty65RBZhGDBAkm6fPvDUU1LxFijKlpV3dcuWOR2JUsofTJgA//sfFCnidCTuk5oKL7wgvZlTU+Hll2WxhIgIpyNzTbZTrK21h4HmXojFbYyBV16BK6+Us+Bx42Ru2zffeL95tqccPQqffw7NmjkdiVLK1xUvHliJ9+RJeOQRGVrMl0/6M/vb9Cm/aS+ZG/fdJ2sBlyoFCxbIgsnbtzsdlXtUqCBn+IE+t1kplTebNkkzoio+VaWTe/v2SW3PjBlyMjVvnv8lXgjw5AtS3bdqFdSsCf/8c6YaLhAkJECtWsG3ypNSynXPPw9btzodhXusWyev4WvWSO3LihXQqpXTUeVOwCdfkNaMv/4qK1ocPiyrIk2Z4nRUeRceLuO+/jLGoZTyvh9/lNkg/m72bOnnv2ePfFy1CqpXdzqq3AuK5Asy3jF7thRfJSXBgw9C//7+XwldooQcx7FjTkeilPIl8fHQtGlgXBkbORI6dJCx3ocekmHEEiWcjipvgib5ggzMjx4NH3wgRVkZvT+TkrL/WV9WsWLwLA2mlHJNRIQUm/rzlbGUFHj2WZkmlZYGb74JkyZBgQJOR5Z3QZV8QZJuz54yWF+wIHz5pawVfPSo05HlXseO8od54oTTkSilfMHJk/DZZ7IMq7+KiZGz3VGjZArppEnwxhuBsxpT0CXfDB06yHhpuXLy8brr/Hu5voED4bffnI5CKeULjh2DI0ecjiL3du2SS+Zz5sjl5YUL4eGHnY7KvYI2+QLUry8LK9evD1u2SHuyJUucjip3Ro6ULi9KqeCWmAiXXgovveR0JLnz++/Sn3/9eqhWTdoGN23qdFTuF9TJF+Cyy+CXX+RM+PhxqYj+4guno8qdiRNh8GCno1BKOWn+fBkn9UfffSdzeA8elFkpK1YEzvzkcwV98gVZYHn6dFktKDkZHn9cOmSlpTkdWc60bi2xK6WCV4cOMHas01HkjLVy4nDPPdK/4IknYO5cKFbM6cg8R5NvutBQWYRhzBj5/N134f775Q/BX5QrJ5Xb33zjdCRKKSeMHAmzZsnqZ/7i1Cl48skzl8kHDZLFEi65xNm4PC3b3s7B5qmnpHPKvffCt9/KwP+sWVC6tNORuSYtTZbVUkoFn7Zt/WsazvHjcra7aJHEPWmSrEwUDPTM9wJuukkqhy+/XAqyGjWCDRucjso1FSrISh8HDzodiVLKm+bNk5OEyy93OhLXbN0qRa6LFkncS5cGT+IFTb5ZqllT2pc1agQ7d8qiDPPmOR2Va+Li4MYbpepRKRUc5s+XubH+4Ndfz0zvrF1bTnIaNnQ6Ku/S5HsRpUvLqkj33QexsbIyyJgxTkeVvUKFZO1Of7r8pJTKvZMnpWalYkWnI8ne11/LYghHjkC7drB8uX/E7W6afLMRHg6TJ8Orr8ryfU8/Db16+f5SfiEh0LmzfzcOUUplLzYW6tXz/Std1sKECZfz8MNSZPXMM/DDD1C4sNOROUOTrwtCQmTxgvHjpYpw+HB4/fVaxMU5HdnFPfusrOiklApckZGy1J4vX+lKSoJHH4Xx4ysTEgIjRsBHH0m//WClyTcHOnWCn3+WuWe//VaCZs1keStfdc01su7lP/84HYlSyhM2b4aXX/btxROOHJEeBF99BQUKpDJzJjz3nNNROU+Tbw41by7tzsqXj2ftWinI+vNPp6PK2rZtsG+f01EopTyhRAmZneGr/v1XXiOXL4fy5eHDD/+ifXuno/INmnxzISoKPvroT264QRJbs2Ywc6bTUV3YI49IcYOueKRUYNmyRVZja9XK6UgubNEimUq0bRtcfbXMHrnySh8fq/MiTb65VKRICvPny6Xo+Hi4804YNkyKCnzNjBnw/PNOR6GUcqe1a313IZjPP5eGH9HRcPvt0j+/XDmno/ItQTzcnXf588siDFWrQr9+kuA2b4YPP/St9m4dOshNKRUYTp2SzlC+Ji1N+uIPGiRf9+kD770nLXvV2fTMN4+MkWlIU6dKMh47VuYD+9Jl3tBQKXp46CH/WyxCKXW+Bx/0vbPe+HjpiTBokLzmjB0L77+viTcrmnzd5L775J+hZEmpiL7+eti+3emozihVCrp0kTcLSin/NmGC1Jr4igMHoEULWRKwcGGYMwe6dnU6Kt+mydeNrrtOigpq1ICNG6XKb8UKp6MSxkhhxrRp/rVSk1LqjJQU6NZNPveVM8r16+W17vffpa/AihXQpo3TUfk+Tb5uVrmyLMrQpg0cPiwLQk+d6nRUZ2zYoIsuKOXP2rSBggWdjkLMnQtNmsjqb5lPPlT2NPl6QJEiMHu2vENNSoIHHpAOWb5QCf3OO1J1GBvrdCRKqZw4eVLe2N9zj28MH40eLfUtsbGy9vmiRTK8pVyjyddDwsJkEYahQ+Uf5bXXpNdyUpLTkUkCnjTJ6SiUUjmxbZv0QnZaair07Cm9mdPSZKbH119LH3zlOp1q5EHGQO/eUKWKVBpPnCgL3U+fDpde6lxcb7wR3D1VlfI3iYlQq5ZUDzspLk4qrX/8UU4wPv0UOnZ0NiZ/pWe+XnD77bBsmVzu/eUXGRvZvNm5ePLlk5aYzzzjXAxKKdcNGSILujhpzx6psP7xRyheHBYs0MSbF5p8vSSjvVq9etIWrnFjWLrUuXiqV5epR0op39e375kqZyesWSOL3a9dK02FVq6EG25wLp5AoMnXi8qXlzPg226DY8ekanHCBGdiiYiAmjXlspEvFIIppS7suedg507nVi6aOVMS7f798nHFCknAKm80+XpZoULw/ffQqxckJ0sRVr9+znSeCg2Vy9/x8d7ft1LKNbfcIm/cvc1aKRi98055jchYUtXJepVAosnXAaGhsgjD6NHy+YABUsTg7eYXoaEweLDs99Qp7+5bKXVxKSnSFKdtW7jkEu/uOzkZuneX3szWylTJL77wfhyBTJOvg7p3l/nAkZHyT9aypTMNMHr2hF9/9f5+lVJZO3xYxla97cQJmb87dqz0q586VfrX+8Lc4kCiyddhbdvKxPnLL5eCrEaN4O+/vRvDxImS+JVSviEuDooVkytk3kx627dLX/qff5Y+9UuWSN965X6afH1ArVpnEu/OnfLHP2+e9/YfEiJn4L17e2+fSqmsTZsm8/G9acUKeQ3auFFaRK5aJdMilWdkm3yNMfmNMZ8ZY3YaY2KNMX8ZY272RnDBpHRpWLwY7r0XYmLkss/HH3tv/02byviOUsp5jz8u46zeMnWqXP06fFhmYfz2m/SpV57jyplvPmA30BwoArwGTDPGVPJcWMEpPBymTJHFqFNTZUy4d2/53NOKFJGx54EDdeqRUk4aPLgaa9dKBylPs1YKPh94QFrfdusmV8GKFPH8voNdtsnXWnvSWvumtXaHtTbNWvsjsB1o4Pnwgk9IiPwzfPGF/PN98AHcdZeMAXlaRIQUWKSkaGWFUk65++49XlkZKCnpzFRHY2Ra0Zgx3kn6KhdjvsaY0kAU4OWyoODSuTPMny9FF7NmSVu3PXs8u8/QUHj+edi/P1zn/irlZUlJkgArVTrp8Sk9R4/CTTdJsWVEhPQe6N1bK5q9KUft9Y0xYcBXwARr7b8X+H5XoCtAyZIlWbJkiTti9ElxcXFeOb4RI8Lp27c2a9dGUL9+EgMHrqdqVc+eBk+eXJm4uDXUqBGY6w5667lzQnR0NKmpqQF7fBC4z19MTD62bi1DtWqePb49e+Q1Zc+eCEqUSGLAgPUUKRKHN36lgfrc5Yq11qUbcpY8BfgJCMtu+6ioKBvIFi9e7LV9HT5sbbNm1oK1ERHWzpzp2f1lHFtSkmf34xRvPnfe1rx5c1u3bl2nw/CoQHz+Dhyw9uBB+dyTx7dkibXFi8trSd261u7e7bFdXVAgPneZAX9YF3OqS5edjTEG+AwoDdxtrU320HsBdQElSsi8u0cflTZvd9whY8GeLIyaMQOeftpzj6+UOmPhQvjkE8/uY+JEqWQ+dgzat4fly51pW6mEq5edxwBXAa2ttV5ugqhACqEmTICoKHjtNRmf2bwZPvzQM2vz3nKLjAkppTwrJUXW+/aUtDSZM5wxdalnT1miMDTUc/tU2XNlnu/lQDegHnDAGBOXfnvY49GpsxgjlYmTJ0sy/vhjmQ984oT795VR8PHww97vOa1UMGnbFtav98xjJyRIYu/fX2ZSjBolV8008Tov23Mma+1OQGvgfMgDD0g7yttvl4roJk1kgetKldy7n4gIudStzdSV8pwpU2Royd0OHZLXiJUrz/SPb9fO/ftRuaPtJf1U48bS/u2qq6QXdKNGnmnC3q6d9Hfdts39j61UMNu9G555RhKvu6f4bNx45jWhYkVZOEUTr2/R5OvHKleWNnBt2si73JYt5d2tu+3Y4cxqS0oFsuLFZa1cdyfen3+WN+c7dsC118qb9Nq13bsPlXeafP1c0aLSDq5rV0hMhPvvd3+LyCeekHfR27e77zGVCmbLl8OuXdC6tXsfd9w4uPlm6Q9/zz1y1apMGffuQ7mHJt8AEBYmxVdDh8q76Fdfhcceg1On3LePtWt14QWl3GXXLti/332Pl5oq/5/dusnnL78siyVERLhvH8q9PDBJRTnBGJl+dMUVUqE8YYJcdvruO7j00rw//tVXw7ffyj+2VkoqlXvbt7t3atHJk/I/P3OmTDscO1ZWRVK+Tc98A8wdd8CyZVCuHCxdKmM///3nvsdv1UrmFyulcu7UKVmcPjraPY+3bx/ccIMk3qJFZfaDJl7/oMk3AF19tRRZ1K0rife66yQR55UxUtAVFZX3x1Iq2KSlyZnp6tWSKPNq7Vpo2BD+/BOqVIEVK6ToUvkHTb4Bqnx5Kepo317aybVpI+3l8qp0aZg7VybrK6VcN22arBrmjurmH3+Epk1h716Z579yJVSvnvfHVd6jyTeAFSokPZp79oTkZOjUSTpkpaXl7XGrV5d/fKWU6+69V4oh88JaGDFCmmdkjPUuXOiZJh3KszT5BrjQUGknN2qUfD5ggBR75KVlZKVK0tzj0089u7iDUoHAWmmmsX173pJkSgr83//Jm+m0NHjrLfjyS2k1q/yPJt8g8fTTcqkqMlKmILRqJY05ciskBLZskVWWlFJZM0aaaVSsmPvHiImB226TN9GXXAJffQWvv+7+Bh3KezT5BpF27aTNXMWKMkbUqJG0psyNfPngvfcgLg4OHHBvnEoFiqNHZdpf69a575G+c6eM686dK2fOixZ5dhUk5R2afINM7dpSCd2wocwDvv56aUeXWxMm5O3nlQpk0dFy1ppbv/8ub5I3bJBai5UrJREr/6fJNwiVKQOLF0v7uZgYaUc3dmzuHuvFF2Xlo5Mn3RujUv5u/XqZb/9//5e7n//uO2jeXPqqt2olfdyrVHFvjMo5mnyDVESEjP327Stdq556SqZBpKbm/LEOHZK5xCkp7o9TKX/1ySfw1185/zlrYdAgeXOckCC91efOhWLF3B+jco62lwxiISGyCEPVqrIww7BhsHUrdOuWs/dkpUrJ5bB8+tekFCCXm0eOzPnPnToFQ4ZU46ef5OtBg+CFF7SwKhDpma/iscekLV2xYtKmrkeP+uzdm7PHKFhQqi+nT/dMjEr5i02bpLlNTqfhHT8uRZE//VSW8HC57Pzii5p4A5UmXwVIW7oVK2RM6b//ImnUKOeXzDp1grZtPROfUv7AWqhWTRpf5CRpbt0qfdgXL4bixZNYuhTuustzcSrnafJVp1WrJpeP69SJZu9eaNYMfvjB9Z+vUkWmHr34ojbfUMHp6adh3rycNb5YvlwqmjdtktkIo0f/ybXXei5G5Rs0+aqzlCgB77+/jkcekQrm22+H4cNdT6bFi0O9epSae5EAAB2TSURBVJ6NUSlf9dprssqQq776Cm68UeYD33yzJOLSpZM8F6DyGZp81XkuucQycSK8/bYk3V69pD2eK9XMYWHSAGDBAumApVQw2LMHuneHsmUhPDz77a2FN9+ERx6RIqtnn4VZs6BwYY+HqnyEJl91QcbIu/ivv5ZLaGPGSBHJiROu/fyePXD4sGdjVMpXXHqptJB0ZZw3MVGS7ltvyYyDkSPhww91tkCw0eSrLurBB6WdXcmSMpbVpIl0xsrOY4/JONbvv3s8RKUc9fXXsHs33HRT9tsePiytJr/+WlYdmzUr9004lH9z7L1WTEwMhw4dIjk52akQ8qRIkSL8888/TofhEeceW6lSYSxbVoo77yzM339LUp01Sz5ezKFD0L+/TD8KDfVw0Eo5JDFRhluy8++/cOutsG2brLf9449Qt67n41O+yZHkGxMTw8GDB7nssssIDw/H+OFEttjYWCIjI50OwyMyH5u1loSEBPbu3cuCBdCpU2EWLIAWLWDiRFmjNCtlysi84bg4GeMK0F+XClKJibJQyeOPZ7/tokVw993SfKNBA5lFULas52NUvsuRy86HDh3isssuIyIiwi8TbzAxxhAREcFll11GfPwhfvoJnnxSXnjuuw/efTf7SuhBg2DaNO/Eq5S37NwpZ6/Z+ewzmf8eHQ133AFLl2riVQ4l3+TkZMJdKQlUPiM8PJzk5GTCwmQRhiFDpLjklVfknf+pU1n/7JtvSn/ai22jlD/ZvFnasn7wQdbbpKXByy9Dly4yU+CFF6RrVcGC3otT+S7HCq70jNe/ZH6+jJFFGKZPlwUaxo+XYpNjxy78s6Ghcum5fn1d/UgFhldflWX+shIfL1eGBg2Sv/9x42DwYKluVgq02lnlwR13wC+/yCW0pUulPd5//11420KFpIFAwYLa/Ur5r5QUWYZz2jSoU+fC2xw4IDUR330HRYrIikRPPunVMJUf0OSr8qRBA1i9Wqo2N2+WpQWXLbvwtsWKyRSLt9/2boxKuctPP0Hv3lnP512//swUu8qVZQ3e1q29G6PyD5p8VZ6VLy8J99Zb5dLzjTfCl19eeNt27WTtYKX8TUoKdOgAo0df+Ptz5sg8+F275CrQypVQo4Z3Y1T+Q5NvDrVo0YJnn33W8cfIzvHjxyldujRbt27Ndtt77rmHYcOG5Wl/kZEZyxFCcjJ07ChLDJ57ibl4cVn/97HHpFpUKX+QkgLXXgtHjsAll5z//VGjpANcbCw88IBMLSpVyvtxKv+hyTdADRw4kFtuuYUqVapku+0bb7xB//79OeFq78gshIbKIgwffSSFJe+8I32eExPP3s4YSb7lyuVpd0p5hbXS+nHePFl4JLPUVOjZU3ozp6VJS9avvoICBZyJVfkPTb4B5FT6XJ74+Hg+/fRTnnjiCZd+rnbt2lxxxRVMmjTJLXE884zMf4yMhClToFUr6XaV2Q03yJnvwIFu2aVSHjNggFT0n3smGxsrq36NGCEdrjIWI9GKZuUK/TPJhbS0NN566y1KlChBqVKl6NOnD2lpacCFLyl37tyZ9u3bn3VfSkoKPXr0oFixYhQrVowXXnjh9GOAdJYaPHgwVapUITw8nNq1a5+XHFu0aEH37t3p06cPJUuWpEmTJgD89NNPhISEnP4aYPDgwRhjzru9/vrrAHTo0IHJkye77Xd0883S/adiRVixQopQNm48e5uSJWWupFK+7KmnZKw3sz17ZL3r2bNlKGXBAnj0UWfiU/7JJ5KvMc7ccuurr74iNDSU3377jY8++ojhw4czderUHD9GWloaK1asYOzYsYwbN47hw4ef/n6/fv347LPPGDVqFBs3bqRv375069aN2bNnn/U4kyZNwlrLsmXLmDhxIgDLli2jQYMGZ83N7d69O/v37z99e/755ylTpgwdO3YEoGHDhqxevZqEhITc/lrOU7s2rFolY2U7dkgRys8/n/l+kSLSnnL6dFi71m27Vcottm2Dhx+WFYuKFz9z/5o10LAhrFsHUVFSWJWTNXyVAgcXVvBnNWrUoF+/fkRGRhIVFcUnn3zCwoULefDBB11+jLJlyzJy5EiMMVSvXp3NmzczbNgwevfuzcmTJxk2bBjz58+nWbNmAFSuXJnVq1czatQobr311tOPU7lyZYYOHXrWY+/cuZOy5/Svi4yMPN2vedCgQUyePJklS5Zw5ZVXAlCuXDmSk5PZt28fpdxYKVKmDCxZIgVY330nZ8SjR0PXrme2CQ3Vub/K91SsKAWEmd+oz5ghCTk+/sxc3syJWSlX+cSZr7XO3HKrzjmz68uVK8ehcwc1s3HdddeddWbauHFj9u7dS0xMDBs3biQxMZF27dpRqFCh07cxY8acV73coEGD8x47ISGBAllUfLz77ruMHDmSxYsXU61atdP3Z7T7dOeZb4aICGlK8PLLUqDSrRv06SOfg4yb1akDn3wiVaVKOclamcu7c6ec4WbcN3Qo3HWXJN5OnaQASxOvyi2XznyNMc8CnYHawGRrbWcPxuTzws5ZP8wYc3q8NiQkBHtOZs/psokZj/XDDz9QsWLFi+674AUaxZYoUYLjx4+fd/+AAQP4+OOPWbp06ekz3gzH0ntDlixZMkexuiokRBZhqFpVku/QobB1K0yadKbr1Y4d0n6ySBGPhKCUS4yBNm3gssvk6+RkqWYeN06+HjhQ3khqh1yVF66e+e4D+gOfezCWgFCyZEn2799/1n3r1q07b7tVq1adlaRXrlxJuXLlKFy4MDVq1CB//vzs3LmTK6+88qzb5Zdfnm0M9evXZ+M51U3vvPMOY8eOPetSc2YbNmygXLlylC5d2tVDzZXHH4f586FoUbmEd8MNsG+fTOUYMEDOfBcu9GgISmVpxgypQbj5ZpkuFB0Nt9wiibdAAbmC07evJl6Vdy4lX2vtdGvtDOCoh+Pxe61atWLOnDnMmjWLTZs20bt3b3bv3n3edvv27aNnz55s2rSJb7/9lvfff59evXoBMj7bp08f+vTpw+eff86WLVtYu3YtH3/8MeMy3n5fRNu2bfnnn384elSergEDBjBixAimTJlCwYIFOXDgAAcOHCAx0wTcZcuW0a5dOzf9Fi6uZUspUqlSBf78Uy7tZRRc7d0rPaCVcsIVV0ClSvL59u3SsWrBAplmtGTJxdevVion3FpwZYzpCnQFOQNcsmTJBbcrUqQIsbGx7ty116SmpnLq1ClSU1NPH0NycjIpKSnExsZy77338scff/DYY48B0KVLF9q3b8/Ro0dPb5+amsp9991HQkICjRo1whjDo48+SpcuXU5v8+KLL1KkSBEGDx5M9+7diYyMpE6dOvTo0eOsxzl16tR5v8tKlSrRoEEDxo8fz5NPPsngwYOJiYk5a+oRwKxZs2jRogWJiYl8//33TJ8+ndjY2LOOLbPExMQsn9PcGDo0jNdeq8n69UVp3DiV11/fSOPGR2neHD79tBCFCydTqlSS2/aXIS4uzq3H4Uuio6NJTU0N2OMDzzx/0dFhzJxZjo4dd2IMjBpVmH79ahEdfQmVKp3k3XfXk5CQiDd+rYH89xnIx5Zj1lqXb8il5/GubBsVFWWzsnHjxiy/5y9iYmKcDuGi5syZY6OiomxKSkq223700Ue2TZs2p7/O6tg88bwlJlr7yCNSAhcSYu3w4dampVk7cqS1c+a4fXfWWmsXL17smQf2Ac2bN7d169Z1OgyP8sTzd+yYtRMmyOdTplibP7/8Td50k7XR0W7f3UUF8t9nIB+btdYCf1gX86lPVDsr92vXrh3PPPMMe/bsyXbbsLAwPvzwQy9Edb78+aUz0FtvSXu+jFZ93bvLIgwLF56pilbK3ayVQkBrpUlG//7SmzkpSZprzJ6tBYDKM3SebwB77rnnXNqua+ZJtw4wRhZhqFoVOneWecDbtsHkyVINXb36mcpTpdzJWmmDaoz87U2cKJ8PG3b+HF+l3MnVqUb50rcNBUKNMQWAFGutzspUbvPgg9LY4I47ZAHyZs2kR3Tp0lLs0qKF0xGqQPLVV3DVVfJ3d8cd8MsvMid98uTz20kq5W6uXnbuByQALwOPpH/ez1NBqeDVpIm0pKxeHTZskJ7Qc+ZIY3vtgqXcqVAhWfDjuusk8ZYrJ+tSa+JV3uDqVKM3rbXmnNubHo5NBakrrpDFGG68EQ4ehPvuk7VSDx+G3393Ojrl7/73P7m8XLSotIrcsgXq14fVq+Hqq52OTgULLbhSPqloUTnj7dJF1gO+91549VVYvNjpyJS/K1BAFkdo0waOHYPbbpMzX60rUN6kyVf5rLAw6Sw0eLAUvnz6Kfz7rzThcKGIW6mzHDgAr7wCEybAyJHSNrJXL/j+e7kErZQ3afJVPs0YeOEFWT0mPBy++AIeewz++MPpyJS/yZdPrpwMHCgraY0eLVXNoaFOR6aCkSZf5RfuvFMuDZYpI2N0L70kL5wnTjgdmfJ1SUmypGW7dtLWNDJS5u927+50ZCqYafJVfuOaa6Qopk4d2LwZ+vWDn392Oirl6/77T6aurVkDl18Ov/0Gbds6HZUKdpp8lV+pUEHGfG+5BRIS4KGHpBraA8sQqwBw880ylejwYZm2tmoV1KrldFRKafLNsQ4dOlCsWDEeffRRp0MJWpGRMHMmPPecFM188420BdR5wCqDtTB2rCxfefKkVMsvXiwNW5TyBZp8c6hXr15MnDgxxz+3e/duWrRoQY0aNahbty7Tp0/3QHTBI18+GDECPvwQQkKkiKZ6dXmhVcEtNVXm6z71lPQLf+UVmDJFCvaU8hWafHOoZcuWREZG5vjn8uXLx/Dhw9m4cSM///wzPXr0ID4+3gMRBpdnn4UffoCCBWUc+Kab5BKjCk4nTsBdd8n60PnySXX8gAHyBk0pX6J/kl5StmxZ6tWrB0CpUqUoVqwYR44ccTiqwHDLLVJEU6GCfKxcGf75x+molLft3Su9mmfNgmLFpBivc2eno1LqwjT5OuCPP/4gOTmZChUqOB1KwKhTR4pprrlGLj03bgzz5jkdlfKWFSvkud+/H6pUkSlFuhCH8mWafL3s6NGjdOzYkc8++wyj65W5VdmysHQp3H23XH68+Wb45BOno1Ke9sMPkmgPHJCVsFauhKgop6NS6uI0+brR4MGDMcacd3v99dcBSEpK4s4776Rv375cf/31DkcbmCIiYNo0ePFFqXjt2hWefloKb1RgsVaq3Dt0gFOn4JFH5FJziRJOR6ZU9jT55lDr1q259957mT9/PuXLl2fFihWnv9e9e3f2799/+vb8889TpkwZOnbsiLWWzp0706pVK52m5GEhITBokPSCDgmBMWPkbFgroQNHSooU2732mnz9zjuyUlH+/M7GpZSr8jkdgL9ZsGABALGxsedVPUdGRp6+b9CgQUyePJklS5Zw5ZVXsnz5cqZOnUqdOnWYMWMGAF9++SW1a9f27gEEkSeekOKru+6CGTNkTHDhQqejUnl18mQoLVrAr79Ksv3iC3jwQaejUipnNPl6wLvvvstHH33E4sWLiUoffGratClpeu3T61q1kjHAVq1kRaRGjeCNNwpqMY6f2rkTnn22Pjt2QOHCsuykjuAof+Qzl53ffFNuIMUSmzdLL9YGDeS+55+HoUPl83LlYN8+WLLkTEVj166y/BxIB6TYWCnEuO02ue+hh+Drr+Xz3NY5ZR7HLVy48HljuwADBgxg9OjRLF269HTiVc6qXl0WUG/SRJYifPrpq5k92+moVE6tXg316sGOHYW46ir46y9NvMqPWWs9couKirJZ2bhxY5bf82W7du2yzZs3t1dddZWtVauW/e677876/ttvv20rVKhgt2zZ4lCE7hETE3PB+/31ecuQkGDtEzftsuXZZY2xdtgwpyNyv+bNm9u6des6HYbbjR9vbf781pZnl70xaoM9ftzpiDxn8eLFTofgMYF8bNZaC/xhXcyRPnPm6w8yd6maOXPmWV2qBgwYwIgRI5gyZQoFCxbkwIEDHDhwgMTERIejVhkKFIBP5lagdecUrIXevaVoJyXF6chUVlJS5Hnq3FmWBrylawVeHnWEokWdjkypvNHkmwOZu1SVLFnydJcqay2DBw/m6NGjNGnShLJly56+/frrrw5HrTIz06bSp8IXTJwIYWEwapSserN/v9ORqXMdOQJNm8IHH0iryFGjYGyrqZT9RavmlP/Tgqtc+vPPP093qTLGcEJXdfcPY8ZwWXQ0Nde+TZUqMkd0zRqoXx+mToXmzZ0OUAH8+adMD8sorJo9WxIxLeT54+23nQ5RqTzRM99cOHr0KN26ddMuVX7u+uvh77+hZUs4eFA+Dh6sSxM6yVpZreraayXxXnutPEdNmzodmVLupck3hzK6VPXu3Vu7VAWA0qVlzddeveSF/6WXpC3loUNORxZ8jh6F1q2hZ0/pSPbEE/DLL1C+vNORKeV+mnxzwGbqUvWgzuoPGPnywbBhMHOmXOKcNw9q1UKnI3nRsmUyjWjRIlke8rvvpENZgQJOR6aUZ2jyzYFff/2VqVOnMmPGDJo0aUK9evVYv36902EpN+nQAdavl+b8hw9D+/Zy9qXLLntOQoLM4b/hBpmD3bgxbNggXcmUCmRacJUDmbtUXai9pPID337L37/+SpMsvl2xojRvef99eOUV+PxzufT5xRc67uhuv/0GDz8sY7vGSBIeOFCq0LOUzfOnlL/QM18VXEqUILlIkYtuEhIiY79//gk1a8KWLXI2/NRTEB3tpTgDWHw89OkjHcd27IBq1WQt5vffzybxgkvPn1L+QJOvCi7jx1Nm7lyXNq1bF/74A/r2lXHhsWPhiitkPFIronPOWhlXr1pVWsUaI29y1q6VqmaX5OD5U8qXafJVwSWHL94FCsil0LVroU4dOH4c7rlHeor/84/nwgw0W7bIHOo77pC+7FdeKQtevPdeDouqNPmqAKHJVykX1KwpjfzHjJFq3F9+gdq1pT3lkSNOR+e7TpyAF1+EGjWkojk8HEaOlDcuDRs6HZ1SztHkq5SLQkJk3HfbNujWTeaijhoFVarIZdSEBKcj9B0JCfI7ueIKGctNToZOnWD7dvi//5PL+EoFM8eSr9VBM7+iz9cZpUrBxx/DunVSiBUTIwVE5cvD8OHBnYSTk+Gzz+QNSZ8+cOyYFFb99huMHy9NTZRSDiXfsLAwEoL5FcoPJSQkEJZtKWpwqV0bli6VZhxVq0qi6dULypSRph0nTzodoffExckbj8sugy5dZKGKqChZ7H7ZMpm/q5Q6w5HkW6pUKfbu3Ut8fLyeUfk4ay3x8fHs3buXUqVKOR1O3v30E/977z23PZwxcMstsGkTzJolY5sxMTJntUwZ6NFDptMEqoMHoV8/OaPt1Uuak1SoAJMny7huu3byO3IbNz9/SjnFkZGXwoULA7Bv3z6Sk5OdCCHPEhMTKRCgve/OPbawsDBKly59+nnzaxERpHngeTMGbrtNumLNng0DBkg178iR8OGH0KoVvPyyfAzx80qLtDRYsECObe5cSE2V+6+5Bl57TX4HHjtGDz1/SnmbY2UPhQsX9usX8yVLllC/fn2nw/CIQD42Ro+m3ObNMlfIA4yR5NO+PaxeLYl38mRYuFBuJUtCx46yOHytWh4JwWO2bIEpUyTpHj4s9xkDt98OL7wgY7se5+HnTylv0ZpDFVymTaOUl9pUNWwIX34p1b7jxkkh0q5dUgU8dKjMde3USS5b16/v5suzbrJxI3z7rbyB+PffM/eXKwfdu8Njj8k4r9d48flTypNcujhkjClujPneGHPSGLPTGPOQpwNTKlCUKQOvvy7TbJYtk2lKhQrJmeRrr0GDBlC2rCzi8M030oTCKQcOSKJ94gl5c1CzJrzxhiTeAgWkF/O8ebB7t4z1ejXxKhVAXD3zHQWcAkoD9YDZxph11tq/PRaZUgEmJEQWZ2jaVBaMnzcPfvxRziwPHpRFHD7/XLYtVUou4zZvLp21qleXJO7Os+OjR2W61F9/SQevxYth796zt4mMlLHshx+GG2+E/Pndt3+lglm2ydcYUxC4G6hlrY0DlhtjZgGPAi97OD6lAlL+/LKEYYcO0jP6f/+TaukFCyQRHjoE338vtwwFC0LlyjJWXLSozCsuXVrGkcPDpXHFiRNw8mQ+li+H2NgztxMnpDnInj2wcyds3Sr3nys8XN4ctGgBbdrI5XBtiKGU+5nspvoYY+oDv1lrwzPd1wdobq29Laufi4iIsA0DuH9cdHQ0RYsWdToMjwjkY2PtWlJSUsh3zTVOR5Ila2Xln5gYSZpxcZCUBCkprvz02vSP9bLdMiQEIiIkqRcuLJfCIyN9c+z5ND94/vIqkP//AvnYAJYuXbrGWuvSH6crybcZ8I21tkym+54EHrbWtjhn265A1/QvawEbchC3vykBBGpX30A+NtDj83d6fP4rkI8NoJq11qWF3l25oBQHnDsnqDBw3kUra+04YByAMeYPV98B+KNAPr5APjbQ4/N3enz+K5CPDeT4XN3WlWrnzUA+Y0zVTPfVBbTYSimllMqFbJOvtfYkMB142xhT0BjTBLgd+NLTwSmllFKByNUmcE8D4cAhYDLQ3YVpRuPyEpgfCOTjC+RjAz0+f6fH578C+dggB8eXbcGVUkoppdzLz1u8K6WUUv5Hk69SSinlZV5JvsaYqsaYRGPMJG/sz1uMMZOMMfuNMTHGmM3GmC5Ox+Quxpj8xpjP0nt5xxpj/jLG3Ox0XO5kjHnWGPOHMSbJGDPe6XjyKpB7sAfac3WuQP9/C+TXysxykuu81ThuFPC7l/blTe8CT1hrk4wx1YElxpi/rLVrnA7MDfIBu4HmwC7gFmCaMaa2tXaHk4G50T6gP9AWKSj0d4Hcgz3QnqtzBfr/WyC/Vmbmcq7z+JmvMeYBIBpY6Ol9eZu19m9rbVLGl+m3Kg6G5DbW2pPW2jettTustWnW2h+B7UADp2NzF2vtdGvtDOCo07HkVaYe7K9Za+OstcuBjB7sfi+QnqsLCfT/t0B+rcyQ01zn0eRrjCkMvA0878n9OMkYM9oYEw/8C+wHfnI4JI8wxpQGotDmKr4qCki11m7OdN86oKZD8ag8CMT/t0B+rcxNrvP0me87wGfW2t0e3o9jrLVPA5FAM6QZSdLFf8L/GGPCgK+ACdbaf7PbXjmiEHDinPtOIH+byo8E6v9bgL9W5jjX5Tr5GmOWGGNsFrflxph6QGvgg9zuw0nZHV/mba21qemX+coD3Z2JOGdcPT5jTAjSzewU8KxjAedQTp6/AOFyD3blu/z1/81V/vhamZ3c5rpcF1ydu6LRBQLqCVQCdhlZo6wQEGqMqWGtvTq3+/WW7I4vC/nwk3EMV47PyBP3GVLAc4u1NtnTcblLLp8/f3a6B7u19r/0+7QHux/x5/+3XPCb10oXtCAXuc6Tl53HIb/ceum3j4HZSLWi3zPGlDLGPGCMKWSMCTXGtAUeBBY5HZsbjQGuAm6z1iY4HYy7GWPyGWMKAKHIP0sBY4xfLh0f6D3YA+m5uoiA/H8LgtfKXOU6jyVfa228tfZAxg25LJZorT3sqX16mUUum+wBjgNDgJ7W2pmORuUmxpjLgW7IH9MBY0xc+u1hh0Nzp35AAvAy8Ej65/0cjShvctOD3V8E2nN1lgD/fwvo18rc5jrt7ayUUkp5mbaXVEoppbxMk69SSinlZZp8lVJKKS/T5KuUUkp5mSZfpZRSyss0+SqllFJepslXKaWU8jJNvkoppZSXafJVSimlvEyTr1IBwBjzYhYrOL3tdGxKqfNpe0mlAoAxJhIomOmuPsDDQDNr7RZnolJKZUWTr1IBxhjzEvAc0Mpau8npeJRS5wu0JbmUCmrGmL7IIuwtrbWbnY5HKXVhmnyVChDGmFeBp4DmeqlZKd+myVepAGCMeQ14Emhhrd3qdDxKqYvT5KuUn0s/4+0BdABOGmPKpH8r2lqb6FxkSqmsaMGVUn7MGGOAaKDwBb7d2lq70MshKaVcoMlXKaWU8jJtsqGUUkp5mSZfpZRSyss0+SqllFJepslXKaWU8jJNvkoppZSXafJVSimlvEyTr1JKKeVlmnyVUkopL9Pkq5RSSnnZ/wPQss9uW/1k+QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 576x252 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# illustrating huber loss\n",
    "\n",
    "plt.figure(figsize=(8, 3.5))\n",
    "z = np.linspace(-4, 4, 200)\n",
    "plt.plot(z, huber_fn(0, z), \"b-\", linewidth=2, label=\"huber($z$)\")\n",
    "plt.plot(z, z**2 / 2, \"b:\", linewidth=1, label=r\"$\\frac{1}{2}z^2$\")\n",
    "plt.plot([-1, -1], [0, huber_fn(0., -1.)], \"r--\")\n",
    "plt.plot([1, 1], [0, huber_fn(0., 1.)], \"r--\")\n",
    "plt.gca().axhline(y=0, color='k')\n",
    "plt.gca().axvline(x=0, color='k')\n",
    "plt.axis([-4, 4, 0, 4])\n",
    "plt.grid(True)\n",
    "plt.xlabel(\"$z$\")\n",
    "plt.legend(fontsize=14)\n",
    "plt.title(\"Huber loss\", fontsize=14)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# e.g., of a model\n",
    "\n",
    "input_shape = X_train.shape[1:]\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=huber_fn, optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 79us/sample - loss: 0.6247 - mae: 0.9969 - val_loss: 0.2884 - val_mae: 0.5913\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.2193 - mae: 0.5176 - val_loss: 0.2361 - val_mae: 0.5232\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd55cfa808>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit with this loss\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and loading custom objects:\n",
    "\n",
    "- To load the saved functions, the names should also be mapped to each custom function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save\n",
    "\n",
    "model.save(\"my_model_with_a_custom_loss.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load\n",
    "\n",
    "model = keras.models.load_model(\"my_model_with_a_custom_loss.h5\",\n",
    "                                custom_objects={\"huber_fn\": huber_fn})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 59us/sample - loss: 0.2056 - mae: 0.4982 - val_loss: 0.2170 - val_mae: 0.5037\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 42us/sample - loss: 0.2006 - mae: 0.4911 - val_loss: 0.2097 - val_mae: 0.4908\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5705a908>"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Also, the input parameters to theses functions can be saved if a function constructor is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_huber(threshold=1.0):\n",
    "    def huber_fn(y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < threshold\n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss  = threshold * tf.abs(error) - threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    return huber_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 72us/sample - loss: 0.2229 - mae: 0.4893 - val_loss: 0.2525 - val_mae: 0.4973\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.2189 - mae: 0.4856 - val_loss: 0.2338 - val_mae: 0.4765\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5706c308>"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss_threshold_2.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss_threshold_2.h5\",\n",
    "                                custom_objects={\"huber_fn\": create_huber(2.0)})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 58us/sample - loss: 0.2148 - mae: 0.4796 - val_loss: 0.2111 - val_mae: 0.4713\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.2123 - mae: 0.4775 - val_loss: 0.1970 - val_mae: 0.4534\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd593b03c8>"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Alternatively, to save the inputs to the custom function, a class may be employed in which\n",
    "  - the `get_config()` method returns a dictionary to map hyperparameters (new and base ones),\n",
    "  - `call()` computes the loss, and\n",
    "  - the constructor uses the ***reduction algorithms***, e.g., (by default) via `sum_over_batch_size` (i.e., the weighted summed loss divided by the batch size).\n",
    "- In this case, loading only requires mapping of the class name.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HuberLoss(keras.losses.Loss):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        self.threshold = threshold\n",
    "        super().__init__(**kwargs)\n",
    "    def call(self, y_true, y_pred):\n",
    "        error = y_true - y_pred\n",
    "        is_small_error = tf.abs(error) < self.threshold\n",
    "        squared_loss = tf.square(error) / 2\n",
    "        linear_loss  = self.threshold * tf.abs(error) - self.threshold**2 / 2\n",
    "        return tf.where(is_small_error, squared_loss, linear_loss)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=HuberLoss(2.), optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 81us/sample - loss: 0.6934 - mae: 0.8818 - val_loss: 0.3288 - val_mae: 0.5451\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.2402 - mae: 0.5077 - val_loss: 0.2315 - val_mae: 0.4861\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5950ca08>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.loss.threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_loss_class_.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_loss_class.h5\",\n",
    "                                custom_objects={\"HuberLoss\": HuberLoss})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other simple custom functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "def my_softplus(z): # return value is just tf.nn.softplus(z)\n",
    "    return tf.math.log(tf.exp(z) + 1.0)\n",
    "\n",
    "def my_glorot_initializer(shape, dtype=tf.float32):\n",
    "    stddev = tf.sqrt(2. / (shape[0] + shape[1]))\n",
    "    return tf.random.normal(shape, stddev=stddev, dtype=dtype)\n",
    "\n",
    "def my_l1_regularizer(weights):\n",
    "    return tf.reduce_sum(tf.abs(0.01 * weights))\n",
    "\n",
    "def my_positive_weights(weights): # return value is just tf.nn.relu(weights)\n",
    "    return tf.where(weights < 0., tf.zeros_like(weights), weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer = keras.layers.Dense(1, activation=my_softplus,\n",
    "                           kernel_initializer=my_glorot_initializer,\n",
    "                           kernel_regularizer=my_l1_regularizer,\n",
    "                           kernel_constraint=my_positive_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=my_l1_regularizer,\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 74us/sample - loss: 1.4486 - mae: 0.8727 - val_loss: 2.4208 - val_mae: 0.5681\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.5848 - mae: 0.5260 - val_loss: 1.6040 - val_mae: 0.5122\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5704c408>"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\n",
    "    \"my_model_with_many_custom_parts.h5\",\n",
    "    custom_objects={\n",
    "       \"my_l1_regularizer\": my_l1_regularizer,\n",
    "       \"my_positive_weights\": my_positive_weights,\n",
    "       \"my_glorot_initializer\": my_glorot_initializer,\n",
    "       \"my_softplus\": my_softplus,\n",
    "    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Or alternatively\n",
    "\n",
    "class MyL1Regularizer(keras.regularizers.Regularizer):\n",
    "    def __init__(self, factor):\n",
    "        self.factor = factor\n",
    "    def __call__(self, weights):  \n",
    "        return tf.reduce_sum(tf.abs(self.factor * weights))\n",
    "    def get_config(self):\n",
    "        return {\"factor\": self.factor}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considering the above subclass, note that:\n",
    "- this time we do use `.get_config()`, contrary to the loss custom subclass.\n",
    "- use `call()` for losses, layers (including activation functions), and models, while use `__call__()` for regularizers, initializers, and constraints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1, activation=my_softplus,\n",
    "                       kernel_regularizer=MyL1Regularizer(0.01),\n",
    "                       kernel_constraint=my_positive_weights,\n",
    "                       kernel_initializer=my_glorot_initializer),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 72us/sample - loss: 1.4486 - mae: 0.8727 - val_loss: 2.4208 - val_mae: 0.5681\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 39us/sample - loss: 0.5848 - mae: 0.5260 - val_loss: 1.6040 - val_mae: 0.5122\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5a8d1108>"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[\"mae\"])\n",
    "\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_many_custom_parts.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\n",
    "    \"my_model_with_many_custom_parts.h5\",\n",
    "    custom_objects={\n",
    "       \"MyL1Regularizer\": MyL1Regularizer,\n",
    "       \"my_positive_weights\": my_positive_weights,\n",
    "       \"my_glorot_initializer\": my_glorot_initializer,\n",
    "       \"my_softplus\": my_softplus,\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Metrics\n",
    "- Generally, metrics are used for evaluation, which are in comparison to the loss functions, they must be easily interpretable while can be non-differentiable with possible 0 gradients.\n",
    "- Their implementation may be sometimes similar to the custom loss.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\", metrics=[create_huber(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 58us/sample - loss: 2.0326 - huber_fn: 0.9038\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 30us/sample - loss: 0.5862 - huber_fn: 0.2682\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5bc27308>"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train_scaled, y_train, epochs=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Note that loss = metric * mean of sample weights (plus some floating point precision error)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[create_huber(2.0)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:sample_weight modes were coerced from\n",
      "  ...\n",
      "    to  \n",
      "  ['...']\n",
      "Train on 11610 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 72us/sample - loss: 0.1174 - huber_fn: 0.2375\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 34us/sample - loss: 0.1137 - huber_fn: 0.2306\n"
     ]
    }
   ],
   "source": [
    "sample_weight = np.random.rand(len(y_train))\n",
    "history = model.fit(X_train_scaled, y_train, epochs=2, sample_weight=sample_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.11744698936265671, 0.11787283955197686)"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history[\"loss\"][0], history.history[\"huber_fn\"][0] * sample_weight.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- But sometimes loss and metrics have different implementations. E.g., `keras.metrics.Precision()` updates the model gradually, batch after batch, i.e., a ***streaming or stateful*** metric, but it also considers the whole items (not just the means of each batch)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.8>"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# e.g., of such metrics\n",
    "precision = keras.metrics.Precision()\n",
    "precision([0, 1, 1, 1, 0, 1, 0, 1], [1, 1, 0, 1, 0, 1, 0, 1])  # labels and predictions (weights can be added)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=0.5>"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2nd batch\n",
    "\n",
    "precision([0, 1, 0, 0, 1, 0, 1, 1], [1, 0, 1, 1, 0, 0, 0, 0])\n",
    "precision.result() # current value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'true_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>,\n",
       " <tf.Variable 'false_positives:0' shape=(1,) dtype=float32, numpy=array([4.], dtype=float32)>]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision.reset_states()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To create a ***streaming metric***, use subclassing in which\n",
    "  - `add_weight()` initializes the variables that are updated over multiple batches,\n",
    "  - `update_state()` is called when an instance of the class is called,\n",
    "  - `result()` returns then the computed value,\n",
    "  - `get_config()` deals with saving the hyperparameters, and\n",
    "  - `reset_states()`, with the default value 0.0, can also be configured."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HuberMetric(keras.metrics.Metric):\n",
    "    def __init__(self, threshold=1.0, **kwargs):\n",
    "        super().__init__(**kwargs) # handles base args (e.g., dtype)\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        self.total = self.add_weight(\"total\", initializer=\"zeros\")\n",
    "        self.count = self.add_weight(\"count\", initializer=\"zeros\")\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        metric = self.huber_fn(y_true, y_pred) # returns for each sample\n",
    "        self.total.assign_add(tf.reduce_sum(metric))\n",
    "        self.count.assign_add(tf.cast(tf.size(y_true), tf.float32))\n",
    "    def result(self):\n",
    "        return self.total / self.count\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=14.0>"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = HuberMetric(2.)\n",
    "\n",
    "m(tf.constant([[2.]]), tf.constant([[10.]]))\n",
    "\n",
    "# total = 2 * |10 - 2| - 2²/2 = 14\n",
    "# count = 1\n",
    "# result = 14 / 1 = 14\n",
    "\n",
    "m.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(), dtype=float32, numpy=7.0>"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m(tf.constant([[0.], [5.]]), tf.constant([[1.], [9.25]]))\n",
    "\n",
    "# total = total + (|1 - 0|² / 2) + (2 * |9.25 - 5| - 2² / 2) = 14 + 7 = 21\n",
    "# count = count + 2 = 3\n",
    "# result = total / count = 21 / 3 = 7\n",
    "\n",
    "m.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Variable 'total:0' shape=() dtype=float32, numpy=21.0>,\n",
       " <tf.Variable 'count:0' shape=() dtype=float32, numpy=3.0>]"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 61us/sample - loss: 0.8646 - huber_metric: 0.8646\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 32us/sample - loss: 0.2564 - huber_metric: 0.2564\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1bd5cf0a9c8>"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.reset_states()\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])\n",
    "\n",
    "model.compile(loss=create_huber(2.0), optimizer=\"nadam\", metrics=[HuberMetric(2.0)])\n",
    "\n",
    "model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32), epochs=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.0"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.metrics[-1].threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric.h5\",\n",
    "                                custom_objects={\"huber_fn\": create_huber(2.0),\n",
    "                                                \"HuberMetric\": HuberMetric})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A simpler alternative for this case:\n",
    "# It handles shapes better and supports sample weights.\n",
    "\n",
    "class HuberMetric(keras.metrics.Mean):\n",
    "    def __init__(self, threshold=1.0, name='HuberMetric', dtype=None):\n",
    "        self.threshold = threshold\n",
    "        self.huber_fn = create_huber(threshold)\n",
    "        super().__init__(name=name, dtype=dtype)\n",
    "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
    "        metric = self.huber_fn(y_true, y_pred)\n",
    "        # it refers to one class above to compute the weighted mean.\n",
    "        super(HuberMetric, self).update_state(metric, sample_weight)\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"threshold\": self.threshold}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:sample_weight modes were coerced from\n",
      "  ...\n",
      "    to  \n",
      "  ['...']\n",
      "Train on 11610 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 83us/sample - loss: 0.4330 - HuberMetric: 0.8725\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.1286 - HuberMetric: 0.2592\n"
     ]
    }
   ],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"selu\", kernel_initializer=\"lecun_normal\",\n",
    "                       input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "])\n",
    "\n",
    "model.compile(loss=keras.losses.Huber(2.0), optimizer=\"nadam\", weighted_metrics=[HuberMetric(2.0)])\n",
    "\n",
    "\n",
    "sample_weight = np.random.rand(len(y_train))\n",
    "\n",
    "history = model.fit(X_train_scaled.astype(np.float32), y_train.astype(np.float32),\n",
    "                    epochs=2, sample_weight=sample_weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.4329961389008008, 0.4329960495905965)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history.history[\"loss\"][0], history.history[\"HuberMetric\"][0] * sample_weight.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_metric_v2.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_metric_v2.h5\",\n",
    "                                custom_objects={\"HuberMetric\": HuberMetric})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Layers\n",
    "\n",
    "- To make an exotic layer, or simply a block of layers, we may create a custom layer.\n",
    "- Custom layers with no weights (like `keras.layers.Flatten` or `keras.layers.ReLU`) can be created by `keras.layers.Lambda`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "# exponential layers may better handle values that are positive and have very different scales.\n",
    "\n",
    "exponential_layer = keras.layers.Lambda(lambda x: tf.exp(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(3,), dtype=float32, numpy=array([0.36787945, 1.        , 2.7182817 ], dtype=float32)>"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# equivalent to activation=tf.exp, activation=keras.activations.exponential, or simply activation=\"exponential\".\n",
    "\n",
    "exponential_layer([-1., 0., 1.])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 49us/sample - loss: 3.6109 - val_loss: 1.3445\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.8465 - val_loss: 0.6490\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 0s 33us/sample - loss: 0.6893 - val_loss: 0.4748\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.4939 - val_loss: 0.4235\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 0s 35us/sample - loss: 0.4542 - val_loss: 0.3933\n",
      "5160/5160 [==============================] - 0s 19us/sample - loss: 0.4390\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.43901502753413"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    keras.layers.Dense(1),\n",
    "    exponential_layer\n",
    "])\n",
    "model.compile(loss=\"mse\", optimizer=\"sgd\")\n",
    "model.fit(X_train_scaled, y_train, epochs=5,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- A ***stateful layer*** (i.e., a layer with weights) can be created by `keras.layers.Layer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [],
   "source": [
    "# E.g., a regular dense layer:\n",
    "\n",
    "class MyDense(keras.layers.Layer):\n",
    "    def __init__(self, units, activation=None, **kwargs):\n",
    "        super().__init__(**kwargs) # it extracts standard, e.g., input_shape, trainable, and name.\n",
    "        self.units = units\n",
    "        self.activation = keras.activations.get(activation)\n",
    "\n",
    "    def build(self, batch_input_shape):  # initialization of weights and bias values\n",
    "        self.kernel = self.add_weight(\n",
    "            name=\"kernel\", shape=[batch_input_shape[-1], self.units], \n",
    "            initializer=\"glorot_normal\") # batch_input_shape[-1] is number of units in the previous layer\n",
    "        self.bias = self.add_weight(\n",
    "            name=\"bias\", shape=[self.units], initializer=\"zeros\")\n",
    "        super().build(batch_input_shape) # must be at the end (it just sets self.built=True)\n",
    "\n",
    "    def call(self, X):\n",
    "        return self.activation(X @ self.kernel + self.bias)\n",
    "    \n",
    "    # better to define compute_output_shape(), as its default value may be different in other Keras implementation\n",
    "    # and is necessary for dynamic layers.\n",
    "    def compute_output_shape(self, batch_input_shape): # for tf.keras, shapes are defined by tf.TensorShape \n",
    "        return tf.TensorShape(batch_input_shape.as_list()[:-1] + [self.units]) # only the last dimension is changed\n",
    "                                                              # as_list() convert a tf.TensorShape to a python list\n",
    "\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {**base_config, \"units\": self.units,\n",
    "                \"activation\": keras.activations.serialize(self.activation)}\n",
    "                # ...serialize() returns the string identifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we should previousely generated frame['node features'] and frame['edge sets'] \n",
    "\n",
    "ds = dataset.load_dataset(dataset_dir, \"train\")\n",
    "ds = ds.flat_map(tf.data.Dataset.from_tensor_slices) # e.g., [1,2,3] ==>> [1],[2],[3] ...\n",
    "# now each element of the dataset is numerical sample (i.e., frame)\n",
    "ds = ds.shuffle(10000,seed=40).repeat(None).prefetch(10) \n",
    "\n",
    "\n",
    "\n",
    "# convert and devide each element into two sub-elements (list of nodes and edges)\n",
    "\n",
    "# Thus, each element is the first dim (baches), containing nodal/edge data (dim 2)\n",
    "\n",
    "ds = ds.map(devide_elements_and_edges, num_parallel_calls=8) \n",
    "\n",
    "# each batch should be [[nodes elements], [edge elements]]\n",
    "batch_size = num_nodes + num_elements\n",
    "ds = ds.batch(batch_size)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "from keras import layers\n",
    "\n",
    "class GraphNetBlock(layers.Layer):\n",
    "  \"\"\"Multi-Edge Interaction Network with residual connections.\"\"\"\n",
    "\n",
    "  def __init__(self, model_fn, **kwargs):\n",
    "    super().__init__(**kwargs) # it extracts standard, e.g., input_shape, trainable, and name.\n",
    "    self._model_fn = model_fn\n",
    "\n",
    "  def _update_edge_features(self, node_features, edge_set):\n",
    "    \"\"\"Aggregrates node features, and applies edge function.\"\"\"\n",
    "    sender_features = tf.gather(node_features, edge_set.senders)\n",
    "    receiver_features = tf.gather(node_features, edge_set.receivers)\n",
    "    features = [sender_features, receiver_features, edge_set.features]\n",
    "    return self._model_fn()(tf.concat(features, axis=-1))\n",
    "\n",
    "  def _update_node_features(self, node_features, edge_sets):\n",
    "    \"\"\"Aggregrates edge features, and applies node function.\"\"\"\n",
    "    num_nodes = tf.shape(node_features)[0]\n",
    "    features = [node_features]\n",
    "    for edge_set in edge_sets:\n",
    "        features.append(tf.math.unsorted_segment_sum(edge_set.features,\n",
    "                                                     edge_set.receivers,\n",
    "                                                     num_nodes))\n",
    "    return self._model_fn()(tf.concat(features, axis=-1))\n",
    "\n",
    "  def call(self, inputs):\n",
    "    \"\"\"Applies GraphNetBlock and returns updated MultiGraph.\"\"\"\n",
    "    node_features, edge_sets = inputs\n",
    "    # apply edge functions\n",
    "    new_edge_sets = []\n",
    "    for edge_set in edge_sets:\n",
    "        updated_features = self._update_edge_features(node_features,\n",
    "                                                      edge_set)\n",
    "        new_edge_sets.append(edge_set._replace(features=updated_features))\n",
    "\n",
    "    # apply node function\n",
    "    new_node_features = self._update_node_features(node_features,\n",
    "                                                   new_edge_sets)\n",
    "\n",
    "    # add residual connections\n",
    "    new_node_features += node_features\n",
    "    new_edge_sets = [es._replace(features=es.features + old_es.features)\n",
    "                     for es, old_es in zip(new_edge_sets, edge_sets)]\n",
    "    return MultiGraph(new_node_features, new_edge_sets)\n",
    "\n",
    "class EncodeProcessDecode(snt.AbstractModule):\n",
    "  \"\"\"Encode-Process-Decode GraphNet model.\"\"\"\n",
    "\n",
    "  def __init__(self,\n",
    "               output_size,\n",
    "               latent_size,\n",
    "               num_layers,\n",
    "               message_passing_steps,\n",
    "               name='EncodeProcessDecode'):\n",
    "    super(EncodeProcessDecode, self).__init__(name=name)\n",
    "    self._latent_size = latent_size\n",
    "    self._output_size = output_size\n",
    "    self._num_layers = num_layers\n",
    "    self._message_passing_steps = message_passing_steps\n",
    "\n",
    "  def _make_mlp(self, output_size, layer_norm=True):\n",
    "    \"\"\"Builds an MLP.\"\"\"\n",
    "    widths = [self._latent_size] * self._num_layers + [output_size]\n",
    "    network = snt.nets.MLP(widths, activate_final=False)\n",
    "    if layer_norm:\n",
    "      network = snt.Sequential([network, snt.LayerNorm()])\n",
    "    return network\n",
    "\n",
    "  def _encoder(self, graph):\n",
    "    \"\"\"Encodes node and edge features into latent features.\"\"\"\n",
    "    with tf.variable_scope('encoder'):\n",
    "      node_latents = self._make_mlp(self._latent_size)(graph.node_features)\n",
    "      new_edges_sets = []\n",
    "      for edge_set in graph.edge_sets:\n",
    "        latent = self._make_mlp(self._latent_size)(edge_set.features)\n",
    "        new_edges_sets.append(edge_set._replace(features=latent))\n",
    "    return MultiGraph(node_latents, new_edges_sets)\n",
    "\n",
    "  def _decoder(self, graph):\n",
    "    \"\"\"Decodes node features from graph.\"\"\"\n",
    "    with tf.variable_scope('decoder'):\n",
    "      decoder = self._make_mlp(self._output_size, layer_norm=False)\n",
    "      return decoder(graph.node_features)\n",
    "\n",
    "  def _build(self, graph):\n",
    "    \"\"\"Encodes and processes a multigraph, and returns node features.\"\"\"\n",
    "    model_fn = functools.partial(self._make_mlp, output_size=self._latent_size)\n",
    "    latent_graph = self._encoder(graph)\n",
    "    for _ in range(self._message_passing_steps):\n",
    "      latent_graph = GraphNetBlock(model_fn)(latent_graph)\n",
    "    return self._decoder(latent_graph)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "class MessagePassingLayer(layers.Layer):\n",
    "    def __init__(self, units, steps=4, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.units = units\n",
    "        self.steps = steps\n",
    "\n",
    "    def build(self, input_shape):\n",
    "        self.atom_dim = input_shape[0][-1]\n",
    "        self.message_step = EdgeNetwork()\n",
    "        self.pad_length = max(0, self.units - self.atom_dim)\n",
    "        self.update_step = layers.GRUCell(self.atom_dim + self.pad_length)\n",
    "        self.built = True\n",
    "\n",
    "    def call(self, inputs):\n",
    "        atom_features, bond_features, pair_indices = inputs\n",
    "\n",
    "        # Pad atom features if number of desired units exceeds atom_features dim.\n",
    "        # Alternatively, a dense layer could be used here.\n",
    "        atom_features_updated = tf.pad(atom_features, [(0, 0), (0, self.pad_length)])\n",
    "\n",
    "        # Perform a number of steps of message passing\n",
    "        for i in range(self.steps):\n",
    "            # Aggregate information from neighbors\n",
    "            atom_features_aggregated = self.message_step(\n",
    "                [atom_features_updated, bond_features, pair_indices]\n",
    "            )\n",
    "\n",
    "            # Update node state via a step of GRU\n",
    "            atom_features_updated, _ = self.update_step(\n",
    "                atom_features_aggregated, atom_features_updated\n",
    "            )\n",
    "        return atom_features_updated\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([\n",
    "    MyDense(30, activation=\"relu\", input_shape=input_shape),\n",
    "    MyDense(1)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 68us/sample - loss: 2.1425 - val_loss: 1.2779\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.6367 - val_loss: 0.5267\n",
      "5160/5160 [==============================] - 0s 18us/sample - loss: 0.5359\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5359284612559533"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save(\"my_model_with_a_custom_layer.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_model_with_a_custom_layer.h5\",\n",
    "                                custom_objects={\"MyDense\": MyDense})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- For the class with multiple inputs or outputs, we should use tuples of inputs or list of outputs.:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyMultiLayer(keras.layers.Layer):\n",
    "    def call(self, X):\n",
    "        X1, X2 = X\n",
    "        return X1 + X2, X1 * X2\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        batch_input_shape1, batch_input_shape2 = batch_input_shape\n",
    "        return [batch_input_shape1, batch_input_shape2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs1 = keras.layers.Input(shape=[2])\n",
    "inputs2 = keras.layers.Input(shape=[2])\n",
    "outputs1, outputs2 = MyMultiLayer()((inputs1, inputs2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Sometimes the layer’s definition are different during training (e.g., when using `Dropout` or `BatchNormalization`), then use the `training` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [],
   "source": [
    "class AddGaussianNoise(keras.layers.Layer):  # equivalent to the regulizar: keras.layers.GaussianNoise\n",
    "    def __init__(self, stddev, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.stddev = stddev\n",
    "\n",
    "    def call(self, X, training=None):\n",
    "        if training:\n",
    "            noise = tf.random.normal(tf.shape(X), stddev=self.stddev)\n",
    "            return X + noise\n",
    "        else:\n",
    "            return X\n",
    "\n",
    "    def compute_output_shape(self, batch_input_shape):\n",
    "        return batch_input_shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples, validate on 3870 samples\n",
      "Epoch 1/2\n",
      "11610/11610 [==============================] - 1s 66us/sample - loss: 0.4776 - val_loss: 0.5223\n",
      "Epoch 2/2\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.4213 - val_loss: 0.3807\n",
      "5160/5160 [==============================] - 0s 19us/sample - loss: 0.3977\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.39774652650189957"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "model.fit(X_train_scaled, y_train, epochs=2,\n",
    "          validation_data=(X_valid_scaled, y_valid))\n",
    "model.evaluate(X_test_scaled, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Models\n",
    "\n",
    "- Simply subclass `keras.models.Model`,\n",
    "- define the model using the `call()` method, and, \n",
    "- use the `get_config()` method for saving the hyperparameters, otherwise save the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQEAwADAAAD/4RDaRXhpZgAATU0AKgAAAAgABAE7AAIAAAAFAAAISodpAAQAAAABAAAIUJydAAEAAAAKAAAQyOocAAcAAAgMAAAAPgAAAAAc6gAAAAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHVzZXIAAAAFkAMAAgAAABQAABCekAQAAgAAABQAABCykpEAAgAAAAMyNwAAkpIAAgAAAAMyNwAA6hwABwAACAwAAAiSAAAAABzqAAAACAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMjAyMTowNToyMyAxMToxNjozMQAyMDIxOjA1OjIzIDExOjE2OjMxAAAAdQBzAGUAcgAAAP/hCxdodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvADw/eHBhY2tldCBiZWdpbj0n77u/JyBpZD0nVzVNME1wQ2VoaUh6cmVTek5UY3prYzlkJz8+DQo8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIj48cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPjxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSJ1dWlkOmZhZjViZGQ1LWJhM2QtMTFkYS1hZDMxLWQzM2Q3NTE4MmYxYiIgeG1sbnM6ZGM9Imh0dHA6Ly9wdXJsLm9yZy9kYy9lbGVtZW50cy8xLjEvIi8+PHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9InV1aWQ6ZmFmNWJkZDUtYmEzZC0xMWRhLWFkMzEtZDMzZDc1MTgyZjFiIiB4bWxuczp4bXA9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC8iPjx4bXA6Q3JlYXRlRGF0ZT4yMDIxLTA1LTIzVDExOjE2OjMxLjI2NjwveG1wOkNyZWF0ZURhdGU+PC9yZGY6RGVzY3JpcHRpb24+PHJkZjpEZXNjcmlwdGlvbiByZGY6YWJvdXQ9InV1aWQ6ZmFmNWJkZDUtYmEzZC0xMWRhLWFkMzEtZDMzZDc1MTgyZjFiIiB4bWxuczpkYz0iaHR0cDovL3B1cmwub3JnL2RjL2VsZW1lbnRzLzEuMS8iPjxkYzpjcmVhdG9yPjxyZGY6U2VxIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+PHJkZjpsaT51c2VyPC9yZGY6bGk+PC9yZGY6U2VxPg0KCQkJPC9kYzpjcmVhdG9yPjwvcmRmOkRlc2NyaXB0aW9uPjwvcmRmOlJERj48L3g6eG1wbWV0YT4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgPD94cGFja2V0IGVuZD0ndyc/Pv/bAEMABgQFBgUEBgYFBgcHBggKEAoKCQkKFA4PDBAXFBgYFxQWFhodJR8aGyMcFhYgLCAjJicpKikZHy0wLSgwJSgpKP/bAEMBBwcHCggKEwoKEygaFhooKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKP/AABEIAaACWAMBIgACEQEDEQH/xAAcAAEAAgMBAQEAAAAAAAAAAAAABQYDBAcCAQj/xABSEAABAwICAwgLDgMHAwUBAAAAAQIDBAUGERIhMQcTFkFRVZTRFBUiNmFxdJGTsdIyNEJSU1RWc3WBkqGyszdlwSMzQ0ZicsIkJYIIFzVj8PH/xAAZAQEAAwEBAAAAAAAAAAAAAAAAAQIDBAX/xAAjEQEAAQMEAwEBAQEAAAAAAAAAAQIRUQMSEzEhMkEUBDMi/9oADAMBAAIRAxEAPwD9UgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXYB5dIxr2tc5Ec73KKu3xHo4xurY0W1Y+scVO5VZbXb7UInHp6lb+H1nY6eZlRBHNE5HRyNR7VTjRUzQDIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAF2KABxPFOCqVuK7D23ctTV3ismdVPRVRETR7ljeRG/0OsYZoai2WWnoauRJXUyb0yT4zEXJqr4dHLPwlWx7374I8ql/QhfQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQAIeqxNaKed8LqxskzFyeyFqyq1fCjUXL7zDwstPylT0WX2SLwm0p4EDwstPylT0WX2RwstPylT0WX2ReC0p4EDwstPylT0WX2RwstPylT0WX2ReC0p4EDwstPylT0WX2RwstPylT0WX2ReC0p4EDwstPylT0WX2RwstPylT0WX2ReC0oDHvfvgjyqX9CF9OZYwu9NXYqwpVUrKqSno6iR870ppO4RW5IvuS38LLT8pU9Fl9kXgtKeBA8LLT8pU9Fl9kcLLT8pU9Fl9kXgtKeBA8LLT8pU9Fl9kcLLT8pU9Fl9kXgtKeBA8LLT8pU9Fl9kcLLT8pU9Fl9kXgtKeBA8LLT8pU9Fl9kcLLT8pU9Fl9kXgtKeBApi2zIv9pUvib8eWB7Gp41VERCZpqiGqhZNTSsliembXscjkcngVCb3RZlAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACv44rJqWzMhpXaE1bOykR6LrYj11qnhyRcvCWAq+Pve9m+04fU4irpMdvttoKa20cdNRRNihYmpETb4V5V8JtAHndu4AAAAAADF2TBnlv0efJpILDKAi57AAAAAAAAAAAAAAAFRFTJdaEPYkWz4t7X0+qgr4X1DYuKKVqt0tHkRyOzy5UJgif892byap/wCBroz/ANMtWP8AlcwAdrlAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKvj73vZvtOH1OLQVfH3vezfacPqcRV1Kae4bYAPNdwACQAABdh+fNxjc4wzirCNTc73RTT17q+pj35tVKxUaj9WprkT8j9BrsPz5uRY9oMH4WqLVeLbfOzErqiVEht8j2qjn5pkuRpRe02UqteLrhgaeTBeK79hSuuFRV2qmo23WjmqXq98MOatfGq8aIqJkfP/dK5MtrL/UYQrosKPVFSuWdiypGq6pVh2o3jNCjsl2xsuMcQ1VvqbU+5WtbXbaaqTQlSPJVVz04tJypqKJBFhTgvBbqmmxjPiLe200lj7JqG6UmSNVPiozw7Mi+2JnyreeodjxFj2WhxHT2OzWSou9fVUTa2BIpWsYrVcqKrnO1NRMk1+FEPNg3R6ass2Iam82+e11tgz7PpXuSRW9zpIrXJqciomo1rLb30e61TI2mlipocMxwNV2bkaqTJ3OlxqiIQtsp6mHHW6vM+0SXGGWKm0KVyaLapEhyc1qqmS8hTbCbynLLj68VVTaprnhOporPdHtZTVcdQ2dzdNM2LIxqZtReXiz1mS7Y7uj79cbZhTDM967WuRlZOtQyCNr1ajtBir7p2Socss8lFSXiyv3M6+/wXOepjbWWGpbI+ngi/xNPTb3OimevPxGzUWyw4fxXiaDGtTiOgfUV0lZST0U07YKmN6IqNbverTRc0XPwF9lN0bps7bgvEtLiuxR3Kjjlh7t0UsEyZPhkauTmO8KKTpRdxqip6TCLpKO2XC2w1VVJUNjr5lkmei5Ikjs9aaSJnkpejGqLT4aR5jyAAhIAABE/57s3k1T/wJYif892byap/4Gmj7wz1fWVzAB3OQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAq+Pve9m+04fU4tBB4xt01xs+VJrq6eVlTCmeWk5i56P3pmn3kVeYTHb4CKt1/t9c1yNnbDOxcpIJ10JI15Fapu9nUnzqD0iHnWmO3beGwDX7OpPnUHpEHZ1J86g9Igsm8NgGv2dSfOoPSIOzqT51B6RAXhsA1+zqT51B6RB2dSfOoPSIPKPDYPmi3S0tFM+XIwdnUnzqD0iDs6k+dQekQJ8NgGv2dSfOoPSIOzqT51B6RBY8MyMajlVGoirx5H1zWu90iL40MHZ1J86g9Ig7OpPnUHpEHlHhsA1+zqT51B6RB2dSfOoPSIE3hsA1+zqT51B6RB2dSfOoPSILF4bANfs6k+dQekQdnUnzqD0iCxeGwRP+e7N5NU/8DaqLpQU8TpZq2mjjbtc6VERPzNbDUcl4xB27RjmW+GB0FIrkyWVXKiuflydyiJy6zXRid12WrMbbLkADtcoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAj7hZbZcXo+vt9LUvTUjpYmuVPOhqcE8P8y27o7eomwBCcE8P8y27o7eocE8P8y27o7eomwBCcE8P8y27o7eocE8P8y27o7eomwBCcE8P8y27o7eocE8P8y27o7eomwBCcE8P8y27o7eocE8P8y27o7eomwuxQOL7o1Xh+wYzw/Rx2ugbC16vrGpA3JWO7lM9XFrU6YzCuHnNRW2W3Kipmi9jt6jjOM8JXC9YvoKq7TLTS3mpkihjyzWGNje4V3hXUuR2vCS1bbBRw3GNY6yBiQS5/CVnc6ScqLki/eBj4J4f5lt3R29Q4J4f5lt3R29RNgCE4J4f5lt3R29Q4J4f5lt3R29RNgCE4J4f5lt3R29Q4J4f5lt3R29RNgCE4J4f5lt3R29Q4J4f5lt3R29RNgCHhwxYoZWyQ2igZI1c0c2BqKn5Eu1qNRERMkTiPoAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFVETNVyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAoWPe/bBHlUv6EL6ULHvfvgjyqX9CF9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHMd3LEtRY7Pb4KJ+hUVFQkmki60bGqO/NcvMp044du22Opqlfe696siZPHR0sKL8DWrnr4VXZ4AOy2avZc7TR1sSorKiJsqZLmiZpnkbhWcCWeqw9b5rRO9ZqSnkV1LMu10bteivhRc/OhZgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQuLro+02Z00DUdUyvbBAi7NN65Iq+BNv3AS8kjI8tN7Wp4VyPHZUHy0X40KFTWGlRmlcM7hUu1yTVPdqq8eSLqangQzdo7VzdSeib1G0aMsuWF37Kg+Wi/Gg7Kg+Wi/GhSO0dq5upPRN6h2jtXN1J6JvUOGco5YXfsqD5aL8aDsqD5aL8aFI7R2rm6k9E3qHaO1c3Unom9Q4ZycsLv2VB8tF+NB2VB8tF+NCkdo7VzdSeib1DtHaubqT0TeonhnJywu/ZUHy0X40HZUHy0X40KR2jtXN1J6JvUO0dq5upPRN6hwzk5YY8eTxOxrglUlYqNqpc1Ryau4L32VB8tF+NCkLYrSqpnbaTV/9Teodo7VzdSeib1DhnJywu/ZUHy0X40HZUHy0X40KR2jtXN1J6JvUO0dq5upPRN6hwzk5YXfsqD5aL8aDsqD5aL8aFI7R2rm6k9E3qHaO1c3Unom9RHDOTlhd+yoPlovxoOyoPlovxoUjtHaubqT0Teodo7VzdSeib1E8M5OWF37Kg+Wi/Gg7Kg+Wi/GhSO0dq5upPRN6h2jtXN1J6JvUOGcnLC8tnie5GslY5eRHIpkKC6xWtzVRKCnavxmMRqp4lTWhuYSrZ6G8PsVXK+eF0S1FHLI5XPRqKiOY5V25ZoqLyFKtOafK1OpFU2XIAGbQAAAAAAAAAGYAGpJcqOKrbSvqYkqHLkkelm7zHlbi3s7sVtPVOdnksiRLoJ/5AboNLsyda7eOwZ96zy3/Nuj49uZugAAAB4mljhifJK9rI2Jm5zlyRE5VUq0+N6J79Cz0tXdHIuSvp2ZRJ/5uyTzZiIv0XWwFV4SXJdbbHknI+rai/kijhHdOZGdMT2S2yrCu6MrUCq8I7pzIzpieyOEd05kZ0xPZGyrBvjK1AqvCO6cyM6YnsjhHdOZGdMT2Rsqwb4ytQKrwjunMjOmJ7I4R3TmRnTE9kbKsG+nK1HON3jvPpfLovU4neEd05kZ0xPZKLuw3iurMMU8dTbW0zEq43aaVCP169WWigmmY+G6HZE2ICqpiO6ZZdpGdMT2RwjunMjOmJ7I2VYN0ZWoFV4R3TmRnTE9kcI7pzIzpieyNlWDfGVqBVeEd05kZ0xPZPL8T3CJquksMr/BDUscv55DZVg3QtgK9asXWuvqG0z3y0VY7ZBVsWJzv9uep33KpYUKrXAAAAAAAAAAAAAAAAAAAAAAqu6H7ztH2lD6nFqKruh+87R9pQ+pxMdonp5B9Ph2uQBznH1TfanHmHrDZL2+0RVlNUTSyMgZKqqzLLU7xkNiiqxZuf8Aay51mJ0vtHPWRUstFNRMie5HrlmxzV2pyFJmy1nXwVvFGNLPhuop6avknlrp26cdJSwummVvxtFqak8KmW24vslxw7NfIK5rbdAjt/fKisWFW7Ue1daL4C14VtKfBTLHuk4dvF0p6CCWsgnqs+xlqqWSFlRl8RzkyUz2W+0tJSYmrrjeuyaShr5WyPkh3tKVGtb/AGX+vLPam3Mi8JtK2A5DiLdPp7hccLU2HamrpZaq7QRTRVNK6J01O7PNW6aa255a0Ok2m/0F1uN1oaKRzqi2SthqUVqojXKmaZLx6hExPgtMQlQVmnxzYJsM1F/Wu3q1wSvhfLKxWrptdoqiJtVc9mW01cP7olivl4itkHZ1NWTNV8DKylfDvyJrVWK5NerWTeC0rgDjWL8WXujwNj+tpa98VXbru2mpZEa3OKPOLudaa/dO28p9v9LjHCmF5sRpjt1f2LE2daOroo2sl2dxpIuaKueSFdydsuyA51dMT3J+KdzuOFX0lPeGTSVdMqIqrlCj2tVcs9SrxEjet0vDlouU9FPNVTPplyqZKamfLHTr/rc1Mk/oTuhFpXQEDecX2O0WOC71lfH2DUZdjviRZFmVdiMRNblI63Y3tt/td47USVMNfQ07pHwVUDoZY82qrXaLk1pqJvBaVvBR8O4xpqXBWFKq/VMklxu1PEjGRxK+SaRzEVVRrU2cq7ELwmtMxE3J8Bo03f3aPJqj/gbxo03f3aPJqj/gU1PVbT9l6AByukAAAAADXrayCii3ypkRjc8k41VeRE4z7WyyxUz300O/TfBZpIma+FeQxU9Kr208te2KWrjRe7a3JGqu3IDzUOr31UbaZsDKbUr5HqquXlRG8XjzPb6CGSqSol3xz0yVEV66KKnGibDbAHhIo0er0Y1Hrtcia1PYAAAADQvd0gtFukq6lXK1mSIxiZue5dSNRONVU3yhVTlveJJqyVyOoKByw0rOJZU93J40XuU8Slqad02Vqq2xdidS1V6ndV3/AD3tdUVA16rFG3iV3xnePUnESrGNY1GsajWomSIiZIh9B1U0xEeHNNUz2AAsqAAAAAABWce3ie02qNtGujU1Um9Mk+ImSqq+PJB2mIvNlmKNuwd7NP5XH/Upbqmqcqq+tq3uXa5Z3Zr+ZhqWrVRoypkmmYi6SNfI5yZ8utTef5a5i129OjMeXc02A4jv9R86qvTO6zNS3m4WqVtVT1c72xrpPikkVzXt40yXZ4xP89dMXROhPbtAPFPKk8Ecrc0bI1HJnyKmZ7MGAAAMFZR01bFvVXBHMzPPJ7c8l5UNOkuFVheRzqmaessi63K9VfJS+HPa5n5p4STPj2te1WvRHNVMlRdioUqoiqF6applaopGSxskjcjmPRHNcmxUXjPZTcFVUtBcKixVCufE1q1FG9df9lmiLH/4qqZeBU5C5HLMWmzpibxcABCQAAAAAAAAAAAAAAAAqu6H7ztH2lD6nFqKruh+87R9pQ+pxMdonp8Ph9Ph2uNyzH9opr3utYSoq11Q2F1FVuVYJ3wv1aPwmqikDfMOUG5/jy1Xu6rUXHDc8iQtkrqh87rdUL7lyaSrm1eVUzQ7RLQUktfBWyU8TquBrmRTK3NzEdtRF4kU+XO3Ud1on0lypYaqlflpRTMRzVyXNNSlJpXipxSqhuv/AL34gdBiKnsk9TSQOo5ailZM2aBG90jHO1Jk7PNE2nu322zvse6FNf8AEk12oah0bK+eko1jZHI1E7pmjmjlTuc8tWrWdZu+F7HeaKCkulrpKqngRGxMkjRd7ROJq7U+427faLdbrYluoaGngoURW7wxiIxUXbmnHmRFKdzjtPcLzZb1hZtXe7Niy0VdXHFSK6FjauDSTJJG6OrUm1TSv8crsL4slairR0+MEmrURM03hqxq5VTjRNSr4jrtpwZhuz161tsslBS1evKWOJEcmfJyfcS0Vuooo6qOOkgbHVPdJO1GJlK5yZKruVVTlG03Q5Xur3Wx11/3Po6OppaiuW8QywrC5HZQ8a5psRV0fN4DLg28W6x4/wB0dl4raehc6rhnb2RIjNJm9+6TPb9xeLfgnDNuej6Gx0ED2ypO1zYUza9NjkXiyzNu6Ybsl2rYau52miq6qH+7lmha5zfvUnbPaN0dOC2C9QW7cdoF7BoqySsxBJFTPrs0ggesjlbK/wAWX5k7dp7lHuj4HivWKaG6Vi1rnpSUdMyNkDVieirpIqqueeSIq6zr8+H7RPa5bbLbKN1BK5Xvp1hboOcq5quXLnxmtbsIYdtscTKGzUMCRSpMxWRIitemeTs9uetSNkp3w41jv+G26lns7etz88BmxHaNy6DCE9VSXSkiuMNOksDoLi+aRJkTNuUavci91lqyO2VFktdTS1dNUW+lkp6uTfZ43RNVsr9XdOTjXUmteQj6bBWF6Wdk1Ph61RysXNr20rEVF5U1CaJIrhzCCsulxvW5DV3xqsuk0dW6XNqNVV3rUqpxKqZL95I7jlxtdpwJeKS/zwU9dTVlUtzjqHIj3Kq5q5UXWqK3LxnVai30dRV01VPSwyVFLmsErmIros0yXRXizQirvg7Dt4uDK66WaiqqtuWUskSK5ctmfL95O2UbocFwSx9sqdzuvvyOhsHZFd2I6oTuIVev9jpZ7M+JVOwXa52CtvN4hoY2VN5itMm/VUKI5scS56LHORcs1XNUTwFsuNroLlQOoa+jgqKNyIiwyMRW5Js1GrZMN2axUk1NaLZS0kE396yJiJp6su65dq7RFMx4JqifLiO51FPg6owliDE1YlbbbtbY6KnqHsyS3vXJWMTka5upXcqaz9CJrQ0KizW2otbLbPQUstvYjWtpnxIsbUbsRG7NRvMY2NjWMREa1MkROJCaYsiZu+mjTd/do8mqP+BvGjTd/do8mqP+BXU9VtP2XoAHK6QAAAABowwVC3aeeV//AE6RtZExHas9auVU8xvGjQU80NbXySuzjle10aZ55IjURfFrN4AAAAAAAADRv1Z2vstfWZ5djwPlT/xaq/0Kdhqm7EsVHG7NZFjSSRV2q93dOXzqpP7oPePffI5f0qRtF7zg+rb6jbRZarMADoYAACAAAAAAKLuqf3Fp8od+hS9FF3VP7i0+UO/QpNPtC+n7QpBJWK0T3qrfT0ro2vZGsiq9ckyQjm5I5FdrTjOmYFrLDNdZm261z08yU71c90yuRW6s0yO/W1JopvEO2XMjBW+85/8AYvqLDfKuyz07G2q3TUsyOzc+SVXoqcmRX633nN/sX1F5ndRMzFh261//ABlH9Sz1IbJrWv8A+Mo/qWepDZPMcEgACAABKOrp1or3YqlvwqrsZ6/6ZEVMvxI0vqHOsTe6s32rSfuodFOXV9nRp+oADNoAAAAAAAAAAAAAAAAFa3QKaWaxxz07Ve+iqY6pWImaua1e6/JVX7iyhURUyXYOhTqaohqoGT00rJYXpm17FzRTKZKzBNsmqZJqSSstz5F0npRzLG1y8ujrTPxIYeA8HPV96UnsnRzRhhxS9A88B4Oer70pPZHAeDnq+9KT2RzRg4pegeeA8HPV96UnsjgPBz1felJ7I5owcUvQPPAeDnq+9KT2RwHg56vvSk9kc0YOKXoHngPBz1felJ7I4Dwc9X3pSeyOaMHFL0Cq4osclsxJhuhpr1eN4uE745tOoRVyRuaZatRaOA8HPV96UnsjmjBxS9A88B4Oer70pPZHAeDnq+9KT2RzRg4pegeeA8HPV96UnsjgPBz1felJ7I5owcUvQPPAeDnq+9KT2RwHg56vvSk9kc0YOKXoHngPBz1felJ7I4Dwc9X3pSeyOaMHFL0aNhRbljTsmDuqS3074nSp7l0r1TNqLx5I3X4zdTAtG7VUXO81ES7Y5KvuXePJEX8yyW+hprdSR0tFCyCCNMmsYmSIUr1d0WhajTtN2yADJqAAAAANGgp5Ya2vkkdmyZ7XRpnnkiNRPWbxo0FNJDW18sj0Vk0jXMTPYiNRPWbwAAAAAAAAFf3Qu8e++Ry/pUjaL3nB9W31Eluhd4998jl/SpG0XvOD6tvqN9H6x1fjMADdgAAAAAAAAFF3VP7i0+UO/Qpel1bSt4yt9Pe7Y2OKrgjqoH77CrnpkqomWS+BcxE2mJXo8VRLmRt2y41VsndNRSrFI5isVURF1LtTWYewrikm9rb5nPzyyY5jkVfAqLrPtTQ3KljSSotdZGxXI3Sc1ETNVyRNvKehzaVUWmXZujLCYK33nP8A7F9RJdqbxzPXfgTrNq3YWut0qWwVFJNRUq/3ssuSLo8aNTPapFevRtm0omuI+up2v/4yj+pZ6kNk8xRtiiZGxMmsRGongQ9HA4pAAEAACUNib3Vm+1aT91DopzrE3urN9q0n7qHRTl1fZ0afqAAzaAAAAAAAAAAAAAAAadyulDa4FmuNXDTRJ8KV6NT8wNwHPq3dXsTZlgtUNddZ9iNpoVVF+9eo+x40xFVsRaHBNfr46idIk/NAOgAoXb/HCpmmEadE5Frm5+o8uxjialRVrcE1ionHTVDZPyRAL+DndNur2hkqRXqiuNpl2ZVEKqnnTX+RdLRerbeId9tlbT1TNv8AZvRVTxptQCQAAAAAAABQse9++CPKpf0IX0oWPe/fBHlUv6EL6AAAAAAAAAAAAAAAAAAAAAAaNBTSQ1tfK96ObM9rmIi7ERqJ6zeNGgpH09bXyveitne1zUTiRGon9DeAAAAAAAAAr+6F3j33yOX9KkbRe84Pq2+okt0LvHvvkcv6VI2i95wfVt9Rvo/WOr8ZgAbsAAAADWuNwpLbTrPX1EVPEnwpHZEJbIVUamarknhKQ7G810nfT4XtU9c9NXZEncRJ4c//AOHhcI3a9ypLii7udAutaOkzYxPAq8ZG6/SduUlfMb2m2yLTROdXVq6kp6ZNNVXkVdiHNMZWu4XHsevdZoLZLUSJFFTxqqyzOXjVE1Jl4jslos1vs8CRW6kigbxq1Na+Ndqm26nidUNncxFla3Ra5dqJx5FZpmrtMVRHSnYBwRDh+NKutVs9yem3LNIvA3rILdvuqxQW+3Qvye5yzyIi60RNTfzz8x1I5jupWeKCgluczt9qqiriYiqmqONM8mp61Iqi1NoTTN6ryvOF7k274foa1q5rLEmn/uTUv55kqRdjs8VnWqjpHZUs0m+si+TcvukTwLqUlDSOlZ7AASqAAAAAlDYm91ZvtWk/dQ6Kc6xN7qzfatJ+6h0U5dX2dGn6gAM2gAAAAAAAAAABF4hv1uw9b3Vl1qWwQpqTPW568iJxqRmNsXU+GqeFjYn1dyqVVlNSxpm57vDyJmQuGcFVNXWresaypX3GTXHSu7qGmReJG7MwNVLnizGsX/ZIu0NofsrJ9c0qf6WpsTw/mSti3N7Jb3pUV7ZLtX7XT1rlkzXwNXUXVrUa1ERERE2Ih9AxwQRQM0YY2Rt5GNREMgAAAAeJYmSt0ZWNe3kcmZUr/ud2C76UrKZbfWZ5pUUa725F8KJqUuAA5pp4vwTErqly4ks0e1W6qmJOX/V+Zb8K4ntmJqFKi2To5yf3kLtUka8jkJzIo+LMEb/U9uMLypbL7HrR8fcxzeB6bF8fnAvAKhgnF3bh8tsu0XYV/pUyqKZ2pHf6mcqFvAAAChY9798EeVS/oQvpQse9++CPKpf0IX0AAAAAAAAAAAAAAAAAAAAAA0qGlfBWV0rpEc2d7XNanwcmon9DdNGgpXwVtfK96ObPI1zUT4KI1E/obwAAAAAAAAFf3Qu8e++Ry/pUjaL3nB9W31Eluhd4998jl/SpG0XvOD6tvqN9H6x1fjMDXrq2moKd09bPHBC3WrpHZIVOTHCV8roMM26pucyLlvmjoRN8KuU1mqIYxF102bVK1fcaWi0v3rflq6pVySCmTTdn4eJCKbhrEF7lWTEl3dBBsSloXaKKnIq//wBLNaMP2q0MRLfQwxOT4ejm9fG5dZF5npa0Qrjp8W4hYvYsUdipHJqfL3czk8XEbdmwNbKKVKmvWS51u1ZapdJM+VG7PWWWtrKahgWasnjgiTVpSOREIvhVZeKuaviY7qH/ADHcl5npNMY1jUaxqNamxETJD6QnCqzfPm+jd1DhVZvnzfRu6id9OUbZwmwQnCqzfPm+jd1DhVZvnzfRu6hvpybZwmyjbsHexT+Vx/1J7hVZvnzfRu6io7pt4obnYIYKCZZ5UqWPVrWO1Imea7CtVVNu1qaZv06SmwEJwqsvz5v4HdQ4VWb5830buotvpyrNM4TYIThVZvnzfRu6hwqs3z5vo3dQ305Ns4TYIThVZvnzfRu6hwqs3z5vo3dQ305Ns4TYIeLE9mlkbG24Qtc5ck082Z/eqEwmtM02ExMT0i0x2hsTe6s32rSfuodFOdYm91ZvtWk/dQ6Kc2r7OjT9QAGbQAAAAAAAAIjFV+pMN2ae4Vzu4YmTGJtkeuxqeMl12HNLhGzGu6O2iflJaLCiSTNz1STrsRfAmX5Lygbu57hurWeTEeJ1369VaZxsfrSmj4mt5F1l+CJkgAAAAAYK6rhoaOaqqXpHDE1XvcvEiAZwUiTEN/rnrJbKCjpKb4C1yudI9OXRaqaPiVczz20xd/I/RS+0X2VYV305XkFG7aYu/kfopfaHbTF38j9FL7Q46sI305XkFG7aYu/kfopfaHbTF38j9FL7Q46sG+nLLug4Wfco2XeyZwYhou7glZqWRE+A7lRfCSWBcSxYnsjapGLDVRO3qphVMljkTaniIjtpi7+R+il9ogrdbsSW7EVddqKa0xLWtTf6dI5N7e5Ph5Z5ov3jjqwb6curAo3bTF38j9FL7Q7aYu/kfopfaHHVg305Y8e9++CPKpf0IX05deKTE11u1pr55bQyW3SOkjayOTJyqmXdZuJjtpi7+R+il9ocdWDfTleQUbtpi7+R+il9odtMXfyP0UvtDjqwb6cryCjdtMXfyP0UvtDtpi7+R+il9ocdWDfTleQUbtpi7+R+il9odtMXfyP0UvtDjqwb6cryCjdtMXfyP0UvtDtpi7+R+il9ocdWDfTleQUbtviyPunw2Wdqf4bN8jVfEqqqJ5iw4dvsN4jlZvbqetgVEnpnrm6NV2a01Ki8SoRNM09piqJ6TAAKrAAAAADRoKV0FbXyue1zZ3tc1E+Dk1EyXzG8aNBSrBW18qyNck8jXI1Pg5NRNfmN4AAAAAAAACv7oXePffI5f0qRtF7zg+rb6iS3Qu8e++Ry/pUjaL3nB9W31G+j9Y6vxRsZU0NdugYcpauNJqZzJFdG7W1V8KF6paaClhbFTRRwxN2MY1ERCl4l/iXhn6uT1KXk0p7lnV1AAF2F1UJhuljvE9Td65rZ27++KkjembYmMXR0kT4yqirmWjQb8VPMV7AHezF9dP8AuuLGeHq1TNczL19OmIpizzoN+KnmGg34qeY9Azu0edBvxU8w0G/FTzHoAedBvxU8w0G/FTzHoAfNBvxU8x80G/FTzHoAedBvxU8w0G/FTzHoC486Dfip5hoN+KnmPQFxhqaWnqYXRVEMcsbkyVr2oqKVy0RSW28V1r03PpGMZUU2kuasa5VRWZ8iK1cvApaSAk776jyKL9x51fyVTGpZz/1RE0Xa2JvdWb7VpP3UOinOcTbbL9q0n7qHRjs1fZyafqAAzaABQN1jG1wwHTW26NtqVtkdUJFcJGq7fKZi7JERNSpt2+DlAv5AYqxjh/CkUUmIbrS0CS5722V/dPy26LU1r9yFUu26SlVjCx4cwdFTXerrWJVVU2+LvVLTKiKj1VONc0yTwpymzd8G2K343rMf4ir1elPSJExlXo7xSNTJFc3kVdf4lAsuF8WWHFUEk2HrpS17IlRJN5fmrM/jJtT7ycPz5gOsocT/APqKuV9wWrEsUFtSCunjTRZUyrsybqzy1a8vgryn6DTYBq3WqbQ2yrq3qiNghfKqrxZIq/0KZuNW11NhJLhUJnV3OZ9VK5dqoq6vy1/eSO6tUupdz+9PYuSuhSP8Tkb/AFJvDdM2jw/badiZNip42p9zUAkQAAAAAqu6KqutVvgVV3ue4QMkRPhNzV2Xnahaiq7ofvO0faUPqcTHaJ6eQfT4drjAVzE2Mbbh2vo6GsjrZ6urY58UVJTOmcrW5Zrk1OLM0qTdEsc1wpqOqbcLdNUuSOFa+jkp2yOXY1HOTLPwEXha0rgACUABo2q5w3J9a2COdi0lQ6mfvsat0nNRFVW57W69oG8CJv8AiChsUltZcHva64VTaODRZpZyOzyReRNS6yWAAAIAVa646s1ss96udS+fsa0VKUlVoR5qknc6kTjTu2kazdQsrdB9fRXu3Uz1ROya23yRxJnszflkieFSu6FrSvYIeuxHbqK7We3SyudUXbT7FVjdJrtBukqqvEmSkwWQAGKqnZS0s1RKqpHExXuyTNckTNQMoNOy3KnvNoo7lQq51LVxNmiVyZKrXJmmaG6B8NCg/scfUD49S1FHMyT/AFI1Wq3zZr5zfNGm7+7R5NUf8DPU9V9P2XoAHK6QAAD497Y2Oc9yNa1M1VVyREPksjIo3PkcjWNTNVXYiEa9kd8o1bI2oip982KmjvzU/PRUDNbKbeX1c++pJ2TLvqK3YjckRE8yG8fGtRrUa1ERETJETiPoAAAAAAAAFf3Qu8e++Ry/pUjaL3nB9W31Eluhd4998jl/SpG0XvOD6tvqN9H6x1fimYl/iXhn6uT1KXko2Jf4l4Z+rk9Sl5NKe5Z1dQBQFLqovAHezF9dP+64sZXMAd7MX10/7rixng6ntL2aPWAAFVnILRh6HFW6LjlLpX3ZrKKqgZAynr5YWsR0KKupq5bTZo6ivwLumWfD7rjWXCwXyKRYOzZN9lp52JmqI9datVMtS8poWTGFjwruk49jv1b2I6pq6d8WcT3I5EhRF1tRTIkrt0XdQsl0tMVS3D9ip5XtrJYXRpPPImijWaSIqoiIi5m3n71ZlE4WKs3TKZKyqZZ7HebzR0cixVNZQwo6KNye6RFVUVypx5G1et0ix2yCwztWorYL3p9hupI98V6tRF0ctuaqqJlynHcDUlLYMO11txNjO+4duNHPMj6BkscbZkVyqj4kcxVejvBnrJm1WiO2Vu4/TwUdwp4G1NbKkNdk6WPSYippZIiJypqGykiqXT8J44gv14qLVUWq52i5QwpUJBXxIxZIs9HSbkq5pnqIibdVo40kq22K9yWKORY3XaOBHQanaKuTXpK3PjyMtcj03cbe9rXK3tDMi5JqX+2ZqOZSVlusdouNywhia5Wa8tmkdwYq1bM1ZdJc42w5ZoirsVM9pEUxKZmYdfxJjuitNfT26goq283WeJKhtHQMRzmxLse5VVEamtNpIYSxRSYkgqEhhqKOtpXpHVUVUzQmgcqZppJyKmtFTUpxeopayHdKlu2LLvc8LRXe10z21VK9scTZmsbpwOc9qoioukqIXPchoKBcS4lultrbxdIpt6gW51sjHR1StTbHk1M9H3OewiqiIgiqZl1QAGbQICXvvqPIov3Hk+QEvffUeRRfuPOn+T/WHP8A0/5y1cTbbL9q0n7qHRjnOJttl+1aT91Dox26vs5NP1AAZtAou7Nie2YXwFcJrrAys7KatJBRuTPsiR6KiMy5ONfEXogL1hCzXu+2y73SldU1ltVXUunK7e43L8LQz0VXwqnEBwb/ANOdNUbnmLp8LYot0VJcrzSx1tJVcb0RqZwKq8bc11cqL4DsdJjq03PdAuuC6mnfFXUcLJkWoRuhUNVEXuE48s0JbFeELNiptEl6pVlfRTJUU8scjo5Inpxo5qov3bDRxtueYcxm2Bb3RudUwaoqqGR0UzE5Ee3Xl4FA5VjqnorZ/wCo7A78MIyK6VTXtucdMmSOhy1K9E1bNLzIfoIqOCNzrDeC3TS2SiclXNqkqp5HSzPTk03a0TwIW4Cn7rkSzbnl4RqKqtja/wAz2r/QsVimbUWWgmYqK18Ebky8LUPmIKJLjY7hRqme/wBO+PztVCs7kF07Y4GomP8A7+jV1LIi7UVq6vyVALqAAAAAFV3Q/edo+0ofU4tRVd0P3naPtKH1OJjtE9Ph8Pp8O1xuUbpFzqrTuq4TqqC1VN1nSiq2pTU7mo9UXRzXulRNRBX+93LdPvNJhNLHNY30dVFW1b66VqStYxf8Nqa1z5U1HS7rh6qq90Ow32N8SUlDTVEMjVVdNVfllkmWzUaW6NhOsvclruuHpoKXEFsnSSGaXNGyR/DjcqfBVP8A9rM5iWkTHhTsZY0ZU4/r8PVeJ3YbtdthjV80KIk1RK5M8kcqLk1EVPOebLj26zYUxnT2Wr7e11la11HXJHmtRG9M81RMs3Nydxa8kLHVYVvNvxhVYms8Fvq5LlTxx11vqXq1u+MTJHxyaK+LJU1m3R4cxNJZL46e7w2+7XByOpm0rEdFRI3Y1FVEV2eS5r4dRHk8KtgK7yXO7Wuez49fdHSO/wCvt1yRrHo3Jc97ZooqKi8mo2K/Gd2obDih0dQj65cRLaaB8jc2wo/QRNScSZuU3qrCl+v91sUt6t1koHWyqjqpK6kkc+adWfBb3KaLVXbmqmer3PqmssuIqWSqhjqqu8rd6GVqK5InJoqzSTJPiqi5cSkxEpvCq46sF5st6wN2Zf6u8UUl8gV/ZbW6UcuvJWqiJk1U0tXgQu2ArvX3DF+OqWsqXy09DWxx0zHbI2rHmqJ95F33D2McT3TDc9zS00NNaq+OrlihmfIs6t2uRVamWSbG+HabjMM4lseLL/cMOy2yalvb2SyJWK9rqd7W6OaI1F008GoWtN4Jta0oLCd+xLddy2orILpSxXDtjNDJXVzka2CBsioqpqyVUTZmaloxGy2Y8sFDbMb1GI4rhM+Crp59F6R9w5yPY5rURNbcss9imddy28RYCoLT2ZRVNbRXZ1yWORHJBUtVyroP1as8+RSbrcNYju2IcN3KentFtpLXVb86jger3PRWK1XaeiiZpnqblx7SPJ4UfHWf/ttuo8a9vW+uA2sbYkxjXYCqqKuwVLRWyanbHUVm/smWKLJNJ6Mauaqiayz4k3Prnc8JYytdPUUjai83JKyBz3O0WNRY1ydq29wuzPabFbbN0W42aW0yy4Zo4JoVp31ECzSSNYqZKqNVETPIWkvCErZKFcUbkTrXULU0DYahsMy7XtSBEzXw6jNhrhHug2usxFTYirLTH2RLHb6Sma3elYxcmrIioquVVRc/AS9NudyW6twI231ETqHDzJ2zb6qo+TfGZZoiJl7rNdpismG8U4Poa+z4cbbqy2zzSTUs9TM6N9JprrarUaumiLrTWmeZNpReLK7QY/vmLaLC1moJ2W67XKSojuFTGxFWJsGpyxouaIqlvp7PiCyQ3ynrrvNd7M+3vfFNVqm/xS5OzbqRM2qmS6+Qh6bcwrLFQ4bqrBXQyXuzvmkkfUNVsdXvq5yNVUzVvgXWWGktWJbpXXGsvstPRxSUT6Smt9NO6SNHO2ySLkma7ETJNSCIn6ePig7l+JqjFlPh7Dljr30FJabdBLXzMREknciI3e2Zovcou13mO5pq1HM34CuVFYsIy2Wejp8Q2KJkL3rmkVRHookkblRM8lyzTVtOlx6SsbpoiPy1oi5oik03+oqtPT6aNN392jyao/4G8Rb4OycZWqHfJIs6ed2nE7RcmWhxkanqnT9nQBmmeWaZkPLY1dUJPDcq+J+tckl0m+ZUVDzJZnQ17aq3yRRSLnprJGr1cq7Vzz1HK6UiyupX1K08c8bpkRVVjXIqpkYqOsnqp5G9hzQQtTVJLkiuXwJnnkeap1RTOatHQxTNX3atejHJ501+cyT3Klp6llPUStileiK1H6kXwZ7MwPNDRPglklnqpqiR+ru1ya1ORETUbyJkEAAAAAAAAAAAAV/dC7x775HL+lSNovecH1bfUSW6F3j33yOX9KkbRe84Pq2+o30frHV+KZiX+JeGfq5PUpeSjYl/iXhn6uT1KXk0p7lnV1AF2AKXVRe5/wB7MX10/wC64sZVsK1DbbU1VkqlSORsr56dztSSxvcrtXhRVVFQtJ4erTNNcxL2NOYmmJgABmu+aKLtRPMfQAMckMUjmukjY5zdiuaiqniMmQADIxrDEsqSLGzfE+Fopn5zIAPMkbJWaEjGvbyOTND6xrWNRrGo1qbERMkQ+gAAABAS999R5FF+48nZZGRRukkc1rGpmrlXJETxlXs9Q663muusSKlE5jKanVUy3xGq5VengVXZJ4jq/kiZ1Luf+mYiiz7ibbZftWk/dQ6Mc6xN7qzfatJ+6h0U7NX2cmn6gAM2gAAAAAAAApzW2IzB+6VV0cqaFuv+U1MufctmT3Tfvz/NDpRWse4ZZieyOgY7eq6Fd9pZk1LHImzXyKBZUXMFP3PcV9vaSWiuDFp73Q/2dVC5MlVU1aSeBS4AAAAKpuiLlQ2lV2Jcoc1/EhayPv8AbI7vap6ORdFXoisem1j0XNrk8SohMeJRPSDBGPmvNvjbHcbPU1MyalloUSRj/DkqoqeIx9tqz6PXvo6dZ1b6cubZVhLgiO21Z9Hr30dOsdtqz6PXvo6dY305NlWEuCI7bVn0evfR06x23q/o/e+jp1jfTk2VYS4IR98nY9jH2K8Ne9cmtdC1Fd4u61mTttWfR699HTrG+nJsqwlwRHbas+j176OnWO21Z9Hr30dOsb6cmyrCXBBS4glhnhhlsd4ZLMqpGxYERXqiZrl3XIZu21Z9Hr30dOsb6cmyrCXBEdtqz6PXvo6dY7bVn0evfR06xvpybKsJcER22rPo9e+jp1jttWfR699HTrG+nJsqwlwRHbas+j176OnWO21Z9Hr30dOsb6cmyrCXBEdtqz6PXvo6dY7bVn0evfR06xvpybKsJc0KVc8eWlE2pS1Cr4s2GKK5V0z9CPD93Ry7N8jaxvnV2om8LWOppqypul1Vi187UjZGxdJsEaLnoovGqrrVSmpXExaF9OiYm8rKADnbh5kjZIiI9rXZLmmaZnoAaMlDJ2e2qgqpI0XJJIl7pj08CLsXwofaW4xT1k1KrXxVEevQemWk34ycqG6YKqJ74nrArWVGiqMkVueioGcGvb5JpaSJ1VHvc+Xdt4kXweA2AAAAAAAAAK/uhd4998jl/SpG0XvOD6tvqJLdC7x775HL+lSNovecH1bfUb6P1jq/FMxL/EvDP1cnqUvJRsS/xLwz9XJ6lLyaU9yzq6gABdVr1lFTVsaR1cLJWIuaI5Nni5CMXCtoVc+x5ekSe0TYKzTE9wmKpjpB8FbR83l6RJ7Q4K2j5vL0iT2icA2U4N9WUHwVtHzeXpEntDgraPm8vSJPaJwDZTg31ZQfBW0fN5ekSe0VHdHhtmHLZTPpoJeyJpkRP+ok9y1UV3wuTV950o5BuuU1ZcKlKx2cdvpXtpY8/wDEe7NXOTwJkiZ+ApXTER4heiqZnzK/UmHLLVUsNRHBLoSsR7f+ok2KmfxjLwVtHzeXpEntGDA8NbQWpbXckXf6JdBr+KSNfcqi/l9xYy0U026Vmqq/aD4K2j5vL0iT2hwVtHzeXpEntE4CdlOEb6soPgraPm8vSJPaHBW0fN5ekSe0TgGynBvqyh4cM2mJ+klLp+CSR708zlVCXa1GtRrURERMkREyRD6CYiI6RMzPaGxN7qzfatJ+6h0U51ib3Vm+1aT91Dopzavs6NP1AAZtAAAAAAAAAAAUnHOFKmrqI73hmVKS/wBOmpyakqG/EfxGzg3GdNe5Ft1cxaG+QJlPSSpkuabVbyoW0rWLcHW3EaNmm3ymuMX9zWQLoyMXi18aAWUHM6e/Yowg90WKaJ90tTFybcaVub0TiV7f/wB95cbBiqyX9P8Atdxgmfxx6Wi9P/FdYE2AAAyAAZDIETfMR2mxRI+7V8FMi7Gud3S+JqawJVdSLkma8hxvdPx9iG1Vr7bboaSlerVdmyRJp0aiZ5qiam6kz1k1W4oxBixyUuC6KWloXu0XXWpboty49Bq7fH6iTpNz+ho7FX00blnuldE6Kevm7qR2ltXwbeICvbilrrbiyfFF9qJaqqnzipnSuz0Wp7pU5M11fcdZyNW10MFtt9PRUbEjp4GIxjU4kQ2gGQAXYoHCN1zGMlHug2plG/ubS5HvRONzvdJ+HV953CiqI6ykgqYVzimY2Ri8qKmaes5JijCFtpcVYZjqo+yprlWTOrZZNsqq3PLwImepEOn4dtzrTa4qDfVligzZE5dqR59yi+FE1fcBJ5DIABkMgAGQyAAZDIABkAAAAAAAAAANGgSpSsr+yFXeVkbvOfxdFM/zzN40qFars2u7Iz3nTbvOeWzRTP8APM3QAAAAAAAAK/uhd4998jl/SpG0XvOD6tvqJLdC7x775HL+lSNovecH1bfUb6P1jq/FMxL/ABLwz9XJ6lLyUbEv8S8M/VyepS8mlPcs6uoAAXUARVddZEq3UVrpVra5qI6RmmjGRIuxXO4vFtPKNxN82tSeDf3+yZVa9FE2mWtOjXVF4hLgiNHE3za1enf7I0cTfNrV6d/slf06WVvz6mEuCI0cTfNrV6d/sjRxN82tXp3+yP06WT8+phLlG3YEywxT+Vx/1LFo4m+bWr07/ZIXFmH8RYit0dJK22QoyVsuk2V6rq4vc+ErV/RpzHaY0K4npcU2AiEbib5vavTP9kaOJvm9q9O/2S36dPKODUwlwRGjib5tavTv9kaOJvm1q9O/2R+nSyfn1MJcERo4m+bWr07/AGRo4m+bWr07/ZH6dLJ+fUwlwQ0kmI4E3yS3UNQxNrIKhUfl4NJuS+c3rZcILjTrLTqvcuVj2OTJ0bk2tcnEpejVpr9ZUq06qO4aOJvdWb7VpP3UOinOsTe6s32rSfuodFMdX2a6fqAAzaAAAAAAAAAAAAAAqZpkpUb9ueYdvNR2TNRrTVWee/Urt7cq/dq/ItwAoHBPFFqT/sOKpJYm+5p7jEkieLT2/kaz6rdMo3LnQ2Wub/8AW5WL+aodIAHN0xDugomiuE6RXfG7JTL1htbul1a5R22zUKcssiv9SqdIAHP+DOL7qn/ecUpSRrtit0Ojq/3rrNyz7muHrbV9lPgkrqrbvlY/fVz5cthdAB8Y1GNRrURGomSIiZIh9AAAAAAAKFj3v3wR5VL+hC+lCx7374I8ql/QhfQAAAAAAAAAAAAAAAAAAAAADRoOyeza/sjS3nTbvOezLRTPL78zeNGgZUtra9Z1XeXPbvOa/B0Uzy+/M3gAAAAAAAAK/uhd4998jl/SpG0XvOD6tvqJLdC7x775HL+lSNovecH1bfUb6P1jq/FMxL/EvDP1cnqUvJRsS/xLwz9XJ6lLyaU9yzq6gCgKXVRWA2o+zzVbkznqqqaSR3KqPVqeZERCyFcwB3sxfXT/ALrixnhanmuZexRH/MAAKLgKTct0my0F7rrV2LeKqqonNZP2JQSTNYqtRyZq1F4lJXC2MLLid08drqX9kwL/AGtNPE6GaPwqxyIuXh2E7Z7ReFhABCQAAAAAAAArbIGwYwuG96knpYpXt4lfpPbn48kTzFkICXvvqPIov3HnT/J/rDn/AKf85auJvdWX7VpP3UOjHOcTbbL9q0n7qHRjt1fZyafqAAzaAAAAAAAAAAAAAAAAAcqNRVcqIicagrePp5WWWKlhdoLXVMdI5ycTHL3X5IqfeRM28keSqxjbYqh0NKysr3MXJy0cDpGIvJpbPzMXDKHmm89EXrNimgipoI4aeNscUbUa1jUyREMhzT/RPyHRGjDT4ZQ803noi9Y4ZQ803noi9ZuAj9E4Tww0+GUPNN56IvWOGUPNN56IvWbgH6JwcMNPhlDzTeeiL1jhlDzTeeiL1m4B+icHDDT4ZQ803noi9Y4ZQ803noi9ZuAfonBwwpWKbtLcsS4arqa0XbeLfO+SbSplRclbkmXKWnhlDzTeeiL1m4B+icHDDT4ZQ803noi9Y4ZQ803noi9ZuAfonBww0+GUPNN56IvWOGUPNN56IvWbgH6JwcMNPhlDzTeeiL1jhlDzTeeiL1m4B+icHDDT4ZQ803noi9Y4ZQ803noi9ZuAfonBww01xpSMyWot93gj45H0blRPHlmpYaCtprhSsqKKZk0LtjmLn93gXwEUQ9qRbbjjeYNVPcqd8skfEksatTSTxo7JfEaaetum0qV6W2LwuoCA3YgAAAADRoGVLa2vdUKqxOe1Yc1zyTRTPxa8zeNGgjqW1te6dVWF72rCirnkmimfi1m8AAAAAAAABX90LvHvvkcv6VI2i95wfVt9RJboXePffI5f0qRtF7zg+rb6jfR+sdX4pmJf4l4Z+rk9Sl5KNiX+JeGfq5PUpeTSnuWdXUAUBS6qLwB3sxfXT/uuLGVzAHezF9dP+64sZ4Op7S9mj1gABVZzrc7/AIi7o3ltN+whFY9Vlt3bsC11J3NRVw1NNVI34cLW6SZpx5Kq+YkODOMrVi7EV0w5WWHsW7TRzLHWxyuexWRo34KohvYcwZcVxYuJ8X3CmuF1ZTrS00VLCscNOxVVXaOaqqque1TW8RN2dptZzi244qcVU9TeKnGlbYH789tHb6ShWWNGNXJFkVWLpquWvXqJSfG+J77T7nXa6btPWXqWpgrGvgRyIrGommjXJnq1uRPCmeZYbHhPFWDqGstGGJ7VV2qWV8tM+uV7JKTTXNW5NRUeiLrTWhuyYGuDrjgepnu61slhfPJUzVOe+VCyMy1ZbMl5eIm9KLS18MS3yw7ozsO3W+z3qjqra6uifUxMZJG9siNVubUTNFRSj0WKrpWV9QtwxvUWDEral7GWu4UjY6HJHqiMR6t7pFT4WlmdaqsOTzbodLiFJ40p4rbJRLFr01c6RHI5OLLUVG9YRxfdsOVWGrhPZbhRTK5jLpVI91QxirmiqzLJXompFRUIpqhMxKNxfjaetxzPhx2IJLBQUNLHNPV0MCzSTyvRFRrV0XaLERc88s1J/coxRV3S53qzVVXPdILfvb6a6S06wuqGPTWjkyRM2rqzRNZip8BXPDN8prxhKqpqiXtfDb6ulrlVrZ0iajWyI9qKrXJknEpZ8IWu90tTca/EVwZNUVj2qyjp1XeKVrUyyZnrVV2qurWJmm3giJv5WYAGTQICXvvqPIov3Hk+QEvffUeRRfuPOn+T/WHP/T/nLVxNtsv2rSfuodGOc4m22X7VpP3UOjHbq+zk0/UABm0AAAAAAAAAAAAAAAACr4+972b7Th9Ti0FXx973s32nD6nEVdSmnuG2ADzXcAAkAAANVLhRK7RSsp1dsy31ufrNlThO4ZgfDWIMFVFberNSVlW64VLFmkauloo/UmZamImJmVZmb2h3dMlTNFzQHJ8ISx4ExriXDr6qaSxx0DbxSpNIr1p2IqtkjzVdmaIqG4/dCvkOH24mqMLKzDasSbNKpFqmwrskWPRyyyyXLSzyJmifhFTpgKNdscVbMU01hsVnW41VVb23CKV06RRtYrsu7XJck2bM81X7zBad0Zj7ZiiS922Whr8OpnW08ciSo5NFXNVjtWaKicaIRsk3Q6AChWPF+IK1LTW1WHGLZ7mrEimo6nf5IEcmbXStRqJlyqi6j3PjC83C83WhwpYoq6K1ybzUVNTVbyx8mijljjyauapmmarkmY2yboXoEBgnEsOKrKtdDTzUskcz6aenmy0opWLk5qqm3xk+VmLJibgACQAACJ/z3ZvJqn/gSxE/57s3k1T/AMDTR94Z6vrK5gA7nIAAAAANGgiqGVte+dVWJ72rEmlnkmiiL4tZvGjQRVDK2vfM5Vie9qxJpZ5Jooi+LWbwAAAAAAAAFf3Qu8e++Ry/pUjaL3nB9W31Eluhd4998jl/SpG0XvOD6tvqN9H6x1fimYl/iXhn6uT1KXkpWM6G4x4ktF7oKJ1bFRNekkUbkR658iGxbsfWepnWnrHTW6pTVvdWzQ/PZ5zSJiJm7OYvEWW0KeYpGTRo+J7XsXWjmrminosqi8Ad7jG/CZUTtcnIu+uLGVeOCrs9yqamgi7Joqld8lpkcjXMk43Mz1LnxouWvWZFxNUIuXB+7fhj9o8nV0K4rnw9PT1qJpi8rICtcJ6j6P3b8MftDhPUfR+7fhj9oz4NTDTmoysoK1wnqPo/dvwx+0OE9R9H7t+GP2hwamDmoysoK1wnqPo/dvwx+0YqrF7qWJJKmyXSNiuRuk5I0TNVyRPdcqjh1MHLRlagVrhPUfR+7fhj9ocJ6j6P3b8LPaHBqYOajKygrXCeo+j92/DH7Q4T1H0fu34Y/aHBqYOWjKygrXCeo+j92/DH7Q4T1H0fu34Y/aHBqYOajKylfkei4xqWoqKraGLNOTu5DyzENZNmyGw17XqmpZ3MY3PwrpL6j5ZLfLSrUVVdK2a4VTkdM9qZNaiamsb4ET+p0/y6NcV7phz/ANGrTNNolhxNtsv2rSfuodGOdYm91ZvtWk/dQ6KdGr7MNP1AAZtAAAAAAAAAAAAAAAAAq+Pve9m+04fU4tBV8fe97N9pw+pxFXUpp7htgA813AAJAAAfFOHbkt/rMHYWntV0wriiSp7OnmRYLc5zFa52aa1VDuQLRVaLKzF5u5LR4XvOKuGN6vVGtrq7vb1ttvpZXor4YURdb8tSK5youXEVCCktLMPQ2eXA+IqjEzYWwSUsjqhKZ70REVyyaejvfH4j9EgnkRsc/tFpqqTdTp5+wnxUMWHGUqPairG16TJ/Zo5dqoiePIhrfbLvTY23T6yGz9lsq46bsSKqarYavKLJzUdlkvIdZBG5O1+eKSibJdrTwBtOJbDfOyon11JIySOhjj/xUdpdyqbcstpuy2e2YZxHiFmKsP36tZW1z6ujrLbvz2SMkyXQVI3JouRc017TvQLciNim7lNAtDhh7u0i2RKmpknbSvldJJoqqIjnq5VVHKiJmhcgCkzebrRFoAAQkAAAif8APdm8mqf+BLEV/nuzeTVP/A00feGer6rkADucgAAAAA0aCGojra98zlWKR7ViTSzyTRRF8WvM3jQt7altdcFn0t6dI3ec11ZaKZ5ffmb4AAAAAAAAFf3Qu8e++Ry/pUjaL3nB9W31E1jGmdWYUvFOxFV0lJK1ETjXRUr9nnbU2mimYqK2SFjs08Rvo/WOr8bhp3C2UNxZo19JBUIiZJvjEXI3AbT5YxKl1WBG07lkw5dKu1yZ6W9ter41XxZnmK5YtszP+622G606bZaN2UiJyq3jLsCNuE7sqvasdWOvmWF1Q6kqE1LHVN3tc+TNdRZ2ua9qOYqOaqZoqLmikfdbHbLsiJcaKGdU2Oc3uk+9NZWKvBlbb2ufhS7VFE7PPseZ6viXwJqXIXqhPiV4BQ6TEuILRm3E9nkkgZqWro2o9PGqIvUWG04qsl1c1lHcIllX/Df3DvFkoiqETTKbABZAcs3brssUFvt0Mite5yzyInImpv55+Y6mcw3UrPFDb5LlPlLWT1cbEcuxkaZ5NT1qU1L2Wo8SvWFrk274foaxrtJZI008/jJqX88yVIux2eOzrVRUi6NHK/fWRfJuX3SJ4F1EoWi9vKs9gAJQAAAAAlDYm91ZvtWk/dQ6Khz2+MWausUDUzV9yhf9zM3r+k6Emw5dX2dGl6gAM2gAAAAAAAAAAAAAAAAQmMLbLcrMqUvvunkZUwJyvYueX3pmn3k2BPk6UykxDbpmo2aoZSVKJ3dPUqkcjF5FRfWmo2O3Ft5wo/TN6ywVltoa1UWso6aoVNiyxNfl50Nfg/Zuabf0dnUc/wCeMtuacIftxbecKP0zesduLbzhR+mb1kxwfs3NNv6OzqHB+zc02/o7OofnjKeecIftxbecKP0zesduLbzhR+mb1kxwfs3NNv6OzqHB+zc02/o7OofnjJzzhD9uLbzhR+mb1jtxbecKP0zesmOD9m5pt/R2dQ4P2bmm39HZ1D88ZOecIftxbecKP0zesduLbzhR+mb1kxwfs3NNv6OzqPnB+zc02/o7OofnjJzzhEdt7bzhSemb1jtxbecKP0zeso26Rd7NYca4epIrdQJFG5X1bUgZloP7lM9XEma+Y6eyw2V7Uc21W9WqmaKlOzX+Q/PGTnnCJ7cW3nCj9M3rHbi284Ufpm9ZMcH7NzTb+js6hwfs3NNv6OzqH54yc84Q/bi284Ufpm9Y7cW3nCj9M3rJjg/Zuabf0dnUOD9m5pt/R2dQ/PGTnnCH7cW3nCj9M3rHbi284Ufpm9ZMcH7NzTb+js6hwfs3NNv6OzqH54yc84Q/bi284Ufpm9Y7cW3nCj9M3rJjg/Zuabf0dnUOD9m5pt/R2dQ/PGTnnCEmvlqgjV8tyo2tTj35vWeMNwTXTEL7zJG6OighWno9NFasmkqK+TJdeWpET71LBDY7VDIkkNsoY3psc2BiKn35EiiZFqNKKJurXqzVFgAGzIAAAAAaM8VQ24088Un/AE+i5k0arq5Ucnhz1febkb2vYjmORzVTNFRc0U9KmaZLsI1lL2qpZ1oY5JmaWk2nRyIjeVG5+fICSBr0dWypp2y6MkWa5aMrVa5F5MlNgAAAAAA+OajkyXYu05/Zo2W2prLKqqj6ORXRNXjhequYqeBM1b9x0ErWLrLLVb3c7an/AHSkaqMbnkk7F1rG7x8S8Sl6Kts3Urp3RZgBp2y4w3CJVYj4pmapYJU0ZI15HIuw3DqibuaYsAAlAAABBXnCVku7lfV0MaTfKxZsd502/eToItdMTZSpMOX+0NVcOXp8kKa201ciPTxI7iPMGL7nbc2YmsdTA1q5LU0yb5H41yLuFRFTJURUXiUjbhO7KKteIrRdckoLhTyuVM9BHZO8y6yt7sHezT+Vx/1JW9YKsl1dvklKlPUbUlp13tyL92pfMUfdCslytFlgat4mrbctQxEiqERXsdryydxoVqvbytTa/h1xNgCbEBeFAAEoAAAAIu4VtRNUNttmY2e4yKiOXaynavw38mrYm1SJmI8ymImemzYFdc8WvkjbpUdtjcxZOJZ35ZonhRu3/cXcjrDaobNbYqSnzcjc3Pe7a96rm5y+FVJE5Kp3Td10xaLAAKpAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALqRQAOB4xwVPccW2yovFS6KpvlVI3e2ZKkEbW9wnhXLLM7LhRlXBYqWluDdGppm7w5U2PRupHJ40RF+8rOPe/fBHlUv6EL6AAAAAAAAAAAAAAAAAAAAAAAABgraSCtgWKpjSSPPPJdWS8qchhWlmgpI4aGdWLHsWZFk0k5FXPM3QBpVM1bBBEsdK2pk/wARGPRmXizPlTcOxYIZJqapVXprbFGsisXLYuRvDIDRqrnBSwRSzb61smxEicqpq40RNRtwyNlibIzPRcmaZpke8gAAAEDfsM0t1qG1kcktHcWJotqYFycqcjkXU5PApB1NNiO3u7qhp7nAie7pn73J97HavMpegWiqaelZpie3Pku1QmqSx3prk2olNpfmi5DtvNzLeuhqdBGRflqV4qXPu283Mt66Go7bzcy3roanQchkOWo4qXPu283Mt66Go7bzcy3roanQchkOWo4qXPu283Mt66Go7bzcy3roanQchkOWo4qXPu283Mt66GpTN1W4SVGHYGuttyp0SqYulUU6sau3VnyndMjm+7x3oUvl0X9SJ1ZlMacQz9tpeZb10NR23m5lvXQ1OgNTUh9yJ5akcVLn3bebmW9dDUdt5uZb10NToOQyHLUcVLn3bebmW9dDUds66RdGlsF2kd/riSJPO5ToOQyHLUcVKkwWW/3KNza+aC1Qu+DTrvs2XJpL3KfcillslmorLSLT0EOg1V0nuVVc57uVzl1qpIgzmqZ7XimI6AAQkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFCx7374I8ql/QhfTQuFoo6+uoauqi06iier4HaSpoKqZKvhN8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAc43eO8+l8ui9Tjo5D4pw7RYlt7KO4rKkLJWypvbtFdJNnF4QJhNiAJsAAAAAAAAAAAAAAB//9k=",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 136,
     "metadata": {
      "image/jpeg": {
       "height": 277,
       "width": 400
      }
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# E.g., a resedual block:\n",
    "\n",
    "from IPython import display\n",
    "display.Image(\"./images/ResidualBlock-.jpg\", width=400, height=277)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new_scaled = X_test_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualBlock(keras.layers.Layer):\n",
    "    def __init__(self, n_layers, n_neurons, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(n_neurons, activation=\"elu\",\n",
    "                                          kernel_initializer=\"he_normal\")\n",
    "                       for _ in range(n_layers)]\n",
    "\n",
    "    def call(self, inputs):\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        return inputs + Z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ResidualRegressor(keras.models.Model):    \n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden1 = keras.layers.Dense(30, activation=\"elu\",\n",
    "                                          kernel_initializer=\"he_normal\")\n",
    "        self.block1 = ResidualBlock(2, 30)\n",
    "        self.block2 = ResidualBlock(2, 30)\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "\n",
    "    def call(self, inputs):\n",
    "        Z = self.hidden1(inputs)\n",
    "        for _ in range(1 + 3):\n",
    "            Z = self.block1(Z)\n",
    "        Z = self.block2(Z)\n",
    "        return self.out(Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 108us/sample - loss: 11.4379\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 1.7623\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 1.4744\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.6444\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.5185\n",
      "5160/5160 [==============================] - 0s 36us/sample - loss: 0.6048\n"
     ]
    }
   ],
   "source": [
    "model = ResidualRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\user\\.conda\\envs\\atml_202021\\lib\\site-packages\\tensorflow_core\\python\\ops\\resource_variable_ops.py:1786: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "If using Keras pass *_constraint arguments to layers.\n",
      "INFO:tensorflow:Assets written to: my_custom_model.ckpt\\assets\n"
     ]
    }
   ],
   "source": [
    "model.save(\"my_custom_model.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.load_model(\"my_custom_model.ckpt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 125us/sample - loss: 0.6816\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 1s 43us/sample - loss: 0.5100\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 1s 45us/sample - loss: 0.6881\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 1s 44us/sample - loss: 0.5656\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 1s 46us/sample - loss: 0.7512\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(X_train_scaled, y_train, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11610 samples\n",
      "Epoch 1/5\n",
      "11610/11610 [==============================] - 1s 93us/sample - loss: 0.9281\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.4480\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - 0s 40us/sample - loss: 0.7364\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.4003\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - 0s 38us/sample - loss: 0.5058\n",
      "5160/5160 [==============================] - 0s 32us/sample - loss: 0.4289\n"
     ]
    }
   ],
   "source": [
    "# keras.models.Model is a subclass of keras.layers.Layer, but layers are suggested to be defined by the latter.\n",
    "# Here, the layer object could be implemented into the model using the Sequential API.\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "block1 = ResidualBlock(2, 30)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", kernel_initializer=\"he_normal\"),\n",
    "    block1, block1, block1, block1,\n",
    "    ResidualBlock(2, 30),\n",
    "    keras.layers.Dense(1)\n",
    "])\n",
    "\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=5)\n",
    "score = model.evaluate(X_test_scaled, y_test)\n",
    "y_pred = model.predict(X_new_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Beyond defining losses and metrics based on labels, predictions, and sample weights, they can also be defined by model internals using the `add_metric()` and `add_loss()` methods of a custom model.\n",
    "- E.g., a reconstruction loss (using the auxiliary output of a hidden layer) is added to the regular loss as a form of (sometimes useful) regularization (since it preserves some information)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ReconstructingRegressor(keras.models.Model):\n",
    "    def __init__(self, output_dim, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        self.hidden = [keras.layers.Dense(30, activation=\"selu\",\n",
    "                                          kernel_initializer=\"lecun_normal\")\n",
    "                       for _ in range(5)]\n",
    "        self.out = keras.layers.Dense(output_dim)\n",
    "        self.reconstruct = keras.layers.Dense(8) # workaround for TF issue mentioned below\n",
    "        self.reconstruction_mean = keras.metrics.Mean(name=\"reconstruction_error\")\n",
    "\n",
    "    #Commented out due to TF issue #46858\n",
    "    #def build(self, batch_input_shape): # the extra layer with unknown number of input can be implemented by build().\n",
    "    #    n_inputs = batch_input_shape[-1]\n",
    "    #    self.reconstruct = keras.layers.Dense(n_inputs)\n",
    "    #    super().build(batch_input_shape)\n",
    "\n",
    "    def call(self, inputs, training=None): # it processes the inputs, and passes the outputs\n",
    "        Z = inputs\n",
    "        for layer in self.hidden:\n",
    "            Z = layer(Z)\n",
    "        reconstruction = self.reconstruct(Z)\n",
    "        recon_loss = tf.reduce_mean(tf.square(reconstruction - inputs))\n",
    "        self.add_loss(0.05 * recon_loss) # 0.05 is just a hyperparameter\n",
    "        if training:\n",
    "            result = self.reconstruction_mean(recon_loss)\n",
    "            self.add_metric(result)\n",
    "        return self.out(Z) # it also print losses and metrics over the epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "model = ReconstructingRegressor(1)\n",
    "model.compile(loss=\"mse\", optimizer=\"nadam\")\n",
    "history = model.fit(X_train_scaled, y_train, epochs=2)\n",
    "y_pred = model.predict(X_test_scaled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Training Loops\n",
    "\n",
    "- They can be written after defining a model (without compiling),\n",
    "- we rarely need them (e.g, the *Wide & Deep* paper as requires two optimizers),\n",
    "- they are very error-prone, and\n",
    "- most importantly, such training loops do not tackle layers with different behaviors in training and testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [],
   "source": [
    "# e.g. a simple model is defined\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "l2_reg = keras.regularizers.l2(0.05)\n",
    "model = keras.models.Sequential([\n",
    "    keras.layers.Dense(30, activation=\"elu\", kernel_initializer=\"he_normal\",\n",
    "                       kernel_regularizer=l2_reg),\n",
    "    keras.layers.Dense(1, kernel_regularizer=l2_reg)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "def random_batch(X, y, batch_size=32): # Data API may have better alternatives.\n",
    "    idx = np.random.randint(len(X), size=batch_size)\n",
    "    return X[idx], y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_status_bar(iteration, total, loss, metrics=None):\n",
    "    metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result())\n",
    "                         for m in [loss] + (metrics or [])])\n",
    "    end = \"\" if iteration < total else \"\\n\"\n",
    "    print(\"\\r{}/{} - \".format(iteration, total) + metrics, end=end) # ensuring the status bar is in the same line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 - loss: 0.0900 - mean_square: 858.5000\n"
     ]
    }
   ],
   "source": [
    "# showing the status bar\n",
    "\n",
    "import time\n",
    "\n",
    "mean_loss = keras.metrics.Mean(name=\"loss\")\n",
    "mean_square = keras.metrics.Mean(name=\"mean_square\")\n",
    "for i in range(1, 50 + 1):\n",
    "    loss = 1 / i\n",
    "    mean_loss(loss)\n",
    "    mean_square(i ** 2)\n",
    "    print_status_bar(i, 50, mean_loss, [mean_square])\n",
    "    time.sleep(0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A fancier version with a progress bar:\n",
    "\n",
    "def progress_bar(iteration, total, size=30):\n",
    "    running = iteration < total\n",
    "    c = \">\" if running else \"=\"\n",
    "    p = (size - 1) * iteration // total\n",
    "    fmt = \"{{:-{}d}}/{{}} [{{}}]\".format(len(str(total)))\n",
    "    params = [iteration, total, \"=\" * p + c + \".\" * (size - p - 1)]\n",
    "    return fmt.format(*params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "' 3500/10000 [=>....]'"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "progress_bar(3500, 10000, size=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_status_bar(iteration, total, loss, metrics=None, size=30):\n",
    "    metrics = \" - \".join([\"{}: {:.4f}\".format(m.name, m.result())\n",
    "                         for m in [loss] + (metrics or [])])\n",
    "    end = \"\" if iteration < total else \"\\n\"\n",
    "    print(\"\\r{} - {}\".format(progress_bar(iteration, total), metrics), end=end)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "50/50 [==============================] - loss: 0.0900 - mean_square: 858.5000\n"
     ]
    }
   ],
   "source": [
    "mean_loss = keras.metrics.Mean(name=\"loss\")\n",
    "mean_square = keras.metrics.Mean(name=\"mean_square\")\n",
    "for i in range(1, 50 + 1):\n",
    "    loss = 1 / i\n",
    "    mean_loss(loss)\n",
    "    mean_square(i ** 2)\n",
    "    print_status_bar(i, 50, mean_loss, [mean_square])\n",
    "    time.sleep(0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first define the hyperparameters, optimizer, loss, and metric functions.\n",
    "\n",
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)\n",
    "\n",
    "n_epochs = 5\n",
    "batch_size = 32\n",
    "n_steps = len(X_train) // batch_size\n",
    "optimizer = keras.optimizers.Nadam(lr=0.01)\n",
    "loss_fn = keras.losses.mean_squared_error\n",
    "mean_loss = keras.metrics.Mean()\n",
    "metrics = [keras.metrics.MeanAbsoluteError()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "WARNING:tensorflow:Layer dense is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "11610/11610 [==============================] - mean: 1.3955 - mean_absolute_error: 0.5722\n",
      "Epoch 2/5\n",
      "11610/11610 [==============================] - mean: 0.6774 - mean_absolute_error: 0.5280\n",
      "Epoch 3/5\n",
      "11610/11610 [==============================] - mean: 0.6351 - mean_absolute_error: 0.5177\n",
      "Epoch 4/5\n",
      "11610/11610 [==============================] - mean: 0.6384 - mean_absolute_error: 0.5181\n",
      "Epoch 5/5\n",
      "11610/11610 [==============================] - mean: 0.6440 - mean_absolute_error: 0.5222\n"
     ]
    }
   ],
   "source": [
    "# then create the custom loop\n",
    "\n",
    "for epoch in range(1, n_epochs + 1):\n",
    "    print(\"Epoch {}/{}\".format(epoch, n_epochs))\n",
    "    for step in range(1, n_steps + 1):\n",
    "        X_batch, y_batch = random_batch(X_train_scaled, y_train)\n",
    "        \n",
    "        # make a prediction on one batch using the model as the function\n",
    "        with tf.GradientTape() as tape:\n",
    "            y_pred = model(X_batch)\n",
    "            main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "            # model.losses are relevant to the hidden layer regulizers\n",
    "            \n",
    "            # add_n() sums multiple tensors of the same shape and data type\n",
    "            loss = tf.add_n([main_loss] + model.losses)\n",
    "        \n",
    "        gradients = tape.gradient(loss, model.trainable_variables)\n",
    "        \n",
    "        # the optimizer’s clipnorm or clipvalue hyperparameter are taken into account authomatically\n",
    "        # any other transformation to the gradients should be added before apply_gradients()\n",
    "        optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "        \n",
    "        # update the training loop if you used kernel_constraint or bias_constraint\n",
    "        for variable in model.variables:\n",
    "            if variable.constraint is not None:\n",
    "                variable.assign(variable.constraint(variable))\n",
    "        \n",
    "        # update the mean loss and the metrics\n",
    "        mean_loss(loss)\n",
    "        for metric in metrics:\n",
    "            metric(y_batch, y_pred)\n",
    "        \n",
    "        print_status_bar(step * batch_size, len(y_train), mean_loss, metrics)\n",
    "    print_status_bar(len(y_train), len(y_train), mean_loss, metrics)\n",
    "    for metric in [mean_loss] + metrics:\n",
    "        metric.reset_states()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f55aa06cc0d345358a51e2c8615c2b05",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='All epochs', max=5.0, style=ProgressStyle(description_wid…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7d165e2812934678a1edf9d3abccf292",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 1/5', max=362.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Layer reconstructing_regressor is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because it's dtype defaults to floatx.\n",
      "\n",
      "If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.\n",
      "\n",
      "To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3b507d41c9674e3a80e8daf8b7e29e23",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 2/5', max=362.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a7c5a5580c464b328b1577dfa979ac64",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 3/5', max=362.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de188a9513bb476d8585ff99c7e2b9e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 4/5', max=362.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e5b60049a0c48e9a0aa1cdd142eabce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Epoch 5/5', max=362.0, style=ProgressStyle(description_wi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# alternatively\n",
    "\n",
    "try:\n",
    "    from tqdm.notebook import trange\n",
    "    from collections import OrderedDict\n",
    "    with trange(1, n_epochs + 1, desc=\"All epochs\") as epochs:\n",
    "        for epoch in epochs:\n",
    "            with trange(1, n_steps + 1, desc=\"Epoch {}/{}\".format(epoch, n_epochs)) as steps:\n",
    "                for step in steps: # loops for sampled batches.\n",
    "                    X_batch, y_batch = random_batch(X_train_scaled, y_train)\n",
    "                    with tf.GradientTape() as tape:\n",
    "                        y_pred = model(X_batch)\n",
    "                        main_loss = tf.reduce_mean(loss_fn(y_batch, y_pred))\n",
    "                        loss = tf.add_n([main_loss] + model.losses)\n",
    "                    gradients = tape.gradient(loss, model.trainable_variables)\n",
    "                    optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
    "                    for variable in model.variables:\n",
    "                        if variable.constraint is not None:\n",
    "                            variable.assign(variable.constraint(variable))\n",
    "                    status = OrderedDict()\n",
    "                    mean_loss(loss)\n",
    "                    status[\"loss\"] = mean_loss.result().numpy()\n",
    "                    for metric in metrics:\n",
    "                        metric(y_batch, y_pred)\n",
    "                        status[metric.name] = metric.result().numpy()\n",
    "                    steps.set_postfix(status)\n",
    "            for metric in [mean_loss] + metrics:\n",
    "                metric.reset_states()\n",
    "except ImportError as ex:\n",
    "    print(\"To run this cell, please install tqdm, ipywidgets and restart Jupyter\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom Optimizers\n",
    "\n",
    "- By `keras.optimizers.Optimizer` subclassing, optimizers can be customized (although rarely)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyMomentumOptimizer(keras.optimizers.Optimizer):\n",
    "    def __init__(self, learning_rate=0.001, momentum=0.9, name=\"MyMomentumOptimizer\", **kwargs):\n",
    "        \"\"\"Call super().__init__() and use _set_hyper() to store hyperparameters\"\"\"\n",
    "        super().__init__(name, **kwargs)\n",
    "        self._set_hyper(\"learning_rate\", kwargs.get(\"lr\", learning_rate)) # handle lr=learning_rate\n",
    "        self._set_hyper(\"decay\", self._initial_decay) # \n",
    "        self._set_hyper(\"momentum\", momentum)\n",
    "    \n",
    "    def _create_slots(self, var_list):\n",
    "        \"\"\"For each model variable, create the optimizer variable associated with it.\n",
    "        TensorFlow calls these optimizer variables \"slots\".\n",
    "        For momentum optimization, we need one momentum slot per model variable.\n",
    "        \"\"\"\n",
    "        for var in var_list:\n",
    "            self.add_slot(var, \"momentum\")\n",
    "\n",
    "    @tf.function\n",
    "    def _resource_apply_dense(self, grad, var):\n",
    "        \"\"\"Update the slots and perform one optimization step for one model variable\n",
    "        \"\"\"\n",
    "        var_dtype = var.dtype.base_dtype\n",
    "        lr_t = self._decayed_lr(var_dtype) # handle learning rate decay\n",
    "        momentum_var = self.get_slot(var, \"momentum\")\n",
    "        momentum_hyper = self._get_hyper(\"momentum\", var_dtype)\n",
    "        momentum_var.assign(momentum_var * momentum_hyper - (1. - momentum_hyper)* grad)\n",
    "        var.assign_add(momentum_var * lr_t)\n",
    "\n",
    "    def _resource_apply_sparse(self, grad, var):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def get_config(self):\n",
    "        base_config = super().get_config()\n",
    "        return {\n",
    "            **base_config,\n",
    "            \"learning_rate\": self._serialize_hyperparameter(\"learning_rate\"),\n",
    "            \"decay\": self._serialize_hyperparameter(\"decay\"),\n",
    "            \"momentum\": self._serialize_hyperparameter(\"momentum\"),\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "keras.backend.clear_session()\n",
    "np.random.seed(42)\n",
    "tf.random.set_seed(42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = keras.models.Sequential([keras.layers.Dense(1, input_shape=[8])])\n",
    "model.compile(loss=\"mse\", optimizer=MyMomentumOptimizer())\n",
    "model.fit(X_train_scaled, y_train, epochs=5)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "e478721be6ae00b21c1ef490b997b2300b1e4a4e739631b0ef12889023ed85ed"
  },
  "kernelspec": {
   "display_name": "Python 3.7.11 ('TF-Agents')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}