Jax_apps.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Jax.apps.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyOdDX0vJdtHm7ZP0dix14Vw",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rahuldave/LearningJax/blob/main/Jax_apps.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Jax applications\n",
        "\n",
        "We finally showcase two applicattions using Jax.\n",
        "\n",
        "Entire frameworks for deep learning have been created using Jax: flax or haiku, optax for optimization and much more. Jax is being used in MCMC inference engines; the easy gradients make for good Hamiltonian Monte Carlo implementations. People have written gradient boosting libraries in Jax..\n",
        "\n",
        "Here we look at two relatively simple applications. The first is solving a differential equation using Jax, and the second is writing custom losses for Xgboost. Both are made possible by the automatic differentiation capabilities of Jax..."
      ],
      "metadata": {
        "id": "pMj1CEwAPGIH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!cat /proc/cpuinfo"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lKRfiCiPdHm0",
        "outputId": "eb1e41b1-7602-40f7-93a9-a4cb1f49668d"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "processor\t: 0\n",
            "vendor_id\t: GenuineIntel\n",
            "cpu family\t: 6\n",
            "model\t\t: 79\n",
            "model name\t: Intel(R) Xeon(R) CPU @ 2.20GHz\n",
            "stepping\t: 0\n",
            "microcode\t: 0x1\n",
            "cpu MHz\t\t: 2199.998\n",
            "cache size\t: 56320 KB\n",
            "physical id\t: 0\n",
            "siblings\t: 2\n",
            "core id\t\t: 0\n",
            "cpu cores\t: 1\n",
            "apicid\t\t: 0\n",
            "initial apicid\t: 0\n",
            "fpu\t\t: yes\n",
            "fpu_exception\t: yes\n",
            "cpuid level\t: 13\n",
            "wp\t\t: yes\n",
            "flags\t\t: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities\n",
            "bugs\t\t: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa\n",
            "bogomips\t: 4399.99\n",
            "clflush size\t: 64\n",
            "cache_alignment\t: 64\n",
            "address sizes\t: 46 bits physical, 48 bits virtual\n",
            "power management:\n",
            "\n",
            "processor\t: 1\n",
            "vendor_id\t: GenuineIntel\n",
            "cpu family\t: 6\n",
            "model\t\t: 79\n",
            "model name\t: Intel(R) Xeon(R) CPU @ 2.20GHz\n",
            "stepping\t: 0\n",
            "microcode\t: 0x1\n",
            "cpu MHz\t\t: 2199.998\n",
            "cache size\t: 56320 KB\n",
            "physical id\t: 0\n",
            "siblings\t: 2\n",
            "core id\t\t: 0\n",
            "cpu cores\t: 1\n",
            "apicid\t\t: 1\n",
            "initial apicid\t: 1\n",
            "fpu\t\t: yes\n",
            "fpu_exception\t: yes\n",
            "cpuid level\t: 13\n",
            "wp\t\t: yes\n",
            "flags\t\t: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities\n",
            "bugs\t\t: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa\n",
            "bogomips\t: 4399.99\n",
            "clflush size\t: 64\n",
            "cache_alignment\t: 64\n",
            "address sizes\t: 46 bits physical, 48 bits virtual\n",
            "power management:\n",
            "\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "USE_TPU = False\n",
        "\n",
        "if USE_TPU:\n",
        "  import jax\n",
        "  import jax.tools.colab_tpu\n",
        "  jax.tools.colab_tpu.setup_tpu()\n",
        "else:\n",
        "  # x8 cpu devices  \n",
        "  import os\n",
        "  os.environ[\"XLA_FLAGS\"] = '--xla_force_host_platform_device_count=2'"
      ],
      "metadata": {
        "id": "8bZAo0kHdNeR"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Solving an ordinary differential equation\n",
        "\n",
        "The code for this demo is taken from https://towardsdatascience.com/how-to-solve-an-ode-with-a-neural-network-917d11918932 .\n",
        "\n",
        "Say we want to find the solution $y=f(x)$ to the differential equation:\n",
        "\n",
        "$$y’=-2xy,$$\n",
        "\n",
        "under some initial conditions, say  $y(0)=1$.\n",
        "\n",
        "Remember that neural neyworks are big honking non-linear function approximators, so we ought to be able to come up with a scheme to solve differential equations approximately using them!\n",
        "\n",
        "Lets say we use the following network as the approximation:\n",
        "\n",
        "![](https://miro.medium.com/max/930/1*ABVRsdzou1-jjtUwH7qUKw.png)\n",
        "\n",
        "It has 31 trainable parameters. We'll arange these parameters in another `pytree`, here an array."
      ],
      "metadata": {
        "id": "JSV9yWUNR59A"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "f4gy_n2E6t6T"
      },
      "outputs": [],
      "source": [
        "import jax.numpy as jnp\n",
        "\n",
        "def sigmoid(x):\n",
        "    return 1./(1. + jnp.exp(-x))\n",
        "\n",
        "def f(params, x):\n",
        "    w0 = params[:10]\n",
        "    b0 = params[10:20]\n",
        "    w1 = params[20:30]\n",
        "    b1 = params[30]\n",
        "    y = sigmoid(x*w0 + b0)\n",
        "    y = sigmoid(jnp.sum(y*w1) + b1)\n",
        "    return y"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lets seed a random key and initialize these params using a standard normal distribution."
      ],
      "metadata": {
        "id": "8aWSzfsGhzj7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from jax import random\n",
        "\n",
        "key = random.PRNGKey(0)\n",
        "params = random.normal(key, shape=(31,))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eubwGzZ3di1U",
        "outputId": "12d78020-54f2-41b9-881a-7c013a0e1cc6"
      },
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now we want to find $f'$ or $\\frac{df}{dx}$ so we take the gradient with respect to the second argument:"
      ],
      "metadata": {
        "id": "ySwczKc0iKSj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from jax import grad\n",
        "\n",
        "dfdx = grad(f, 1)"
      ],
      "metadata": {
        "id": "bh0nPaYTdj1r"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We need to find a range on which we need to solve the differential equation: these are the appropriate boundary conditions.."
      ],
      "metadata": {
        "id": "526zCMSQif58"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "inputs = jnp.linspace(-2., 2., num=401)"
      ],
      "metadata": {
        "id": "3V-Oh6wxdtCC"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Since Jax is being fed a sclar function to calculate the gradient, and we are not summing over the $x$ as we normally do in a loss calculation, we'll use out old friend `vmap` to vectorize out gradient over the input range..."
      ],
      "metadata": {
        "id": "1YshuymoixWK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from jax import vmap\n",
        "\n",
        "f_vect = vmap(f, (None, 0))\n",
        "dfdx_vect = vmap(dfdx, (None, 0))"
      ],
      "metadata": {
        "id": "KVUqmckidwqV"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The `(None, 0)` once agin maps to the arguments: map over the 0-axis of the second argument (x), while the first argument (params) is left untouched (or more precisely, with `None`, it will be broadcasted across the mapping).\n",
        "\n",
        "All this is very nice, but how do we _enforce_ our differential equation? We do it the usual way, by defining a loss and minimizing it. Since our ODE is:\n",
        "\n",
        "$$y’=-2xy,$$\n",
        "\n",
        "we'll want:\n",
        "\n",
        "$$y’ + 2xy$$\n",
        "\n",
        "to be as small as possible. We'll also want $y(0)$ to be as close to 1 as possible. Putting in these two constraints into separate squared losses and adding them gives us:"
      ],
      "metadata": {
        "id": "I1rArDhIjO-i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from jax import jit\n",
        "\n",
        "@jit\n",
        "def loss(params, inputs):\n",
        "    eq = dfdx_vect(params, inputs) + 2.*inputs*f_vect(params, inputs)\n",
        "    ic = f(params, 0.) - 1.\n",
        "    return jnp.mean(eq**2) + ic**2"
      ],
      "metadata": {
        "id": "ca4Z6EVEeG1A"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now everything is as before. Get the usual gradient of the loss with respect to the parameters and set up gradient descent. "
      ],
      "metadata": {
        "id": "zxHR0r74kxyV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "grad_loss = jit(grad(loss, 0))"
      ],
      "metadata": {
        "id": "59lyaLX7eRMv"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We use Nesterov momentum for faster convergence..we dont need to but it illustrates how we might evaluate the jitted `grad_loss` function at slightly different parameter values.\n",
        "\n",
        "![](https://miro.medium.com/max/740/1*KCRAjTK4244WKBZC6Zm8RQ.png)"
      ],
      "metadata": {
        "id": "NDc8Yph7lTR3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "epochs = 1000\n",
        "learning_rate = 0.1\n",
        "momentum = 0.99\n",
        "velocity = 0.\n",
        "\n",
        "for epoch in range(epochs):\n",
        "    if epoch % 100 == 0:\n",
        "        print('epoch: %3d loss: %.6f' % (epoch, loss(params, inputs)))\n",
        "    gradient = grad_loss(params + momentum*velocity, inputs)\n",
        "    velocity = momentum*velocity - learning_rate*gradient\n",
        "    params += velocity"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ms6emVx-ebNP",
        "outputId": "acf6d465-4240-4e16-a1bb-6c7e06bc4771"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch:   0 loss: 0.954726\n",
            "epoch: 100 loss: 0.061131\n",
            "epoch: 200 loss: 0.024128\n",
            "epoch: 300 loss: 0.008868\n",
            "epoch: 400 loss: 0.003202\n",
            "epoch: 500 loss: 0.001567\n",
            "epoch: 600 loss: 0.000880\n",
            "epoch: 700 loss: 0.000505\n",
            "epoch: 800 loss: 0.000306\n",
            "epoch: 900 loss: 0.000224\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "How did we do? We plot our solution against the exact solution which is $exp(-x^2)$:"
      ],
      "metadata": {
        "id": "83f56-7Dll_q"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(inputs, jnp.exp(-inputs**2), '.', alpha=0.5,  label='exact')\n",
        "plt.plot(inputs, f_vect(params, inputs), label='approx')\n",
        "plt.legend()\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        },
        "id": "YNAr7InWekHd",
        "outputId": "7ee63bc3-5cd3-4b21-d267-333a77666c80"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3iUVfr/8feZSZuEkJkUSkgCCEiHBAIGpSPSFFBEQGVBFNeCuq6udXVdZdd1l12/Ytu1YKVJR0Q6CEjHUAMRREjCUFJmQsmEJDPn90eCv4iUBCaZkvt1XVxMeZK58xA+eXKfM+corTVCCCF8n8HTBQghhHAPCXQhhPATEuhCCOEnJNCFEMJPSKALIYSfCPDUC0dHR+tGjRp56uWFEMInbd++PUdrHXOx5zwW6I0aNWLbtm2eenkhhPBJSqkjl3pOWi5CCOEnJNCFEMJPSKALIYSf8FgPXQghzisuLiYrK4vCwkJPl+I1QkJCiIuLIzAwsMIfI4EuhPC4rKwswsPDadSoEUopT5fjcVprcnNzycrKonHjxhX+uCu2XJRSU5RSJ5VSey7xvFJKTVZKHVRK7VJKdahE3UIIQWFhIVFRURLmZZRSREVFVfo3lopcoX8KvAN8fonnBwDNyv7cALxf9rcQPsNqd7Aj086h7DPkO4oxmwKJCA0iMiyIxHgzsWaTp0v0exLmv3Y15+OKga61XquUanSZQ4YAn+vSdXg3KaXMSqn6Wutjla5GiGp0PsR3ZNpYk55Nzpki7AVFULaidHCggVrBAVjCgujVPIb28RYJd+HV3NFDbwBklrufVfbYbwJdKfUg8CBAQkKCG15aiMqz2h3M3Z7Bvh/WcV3BLhqVZPECOVjUaYyBLhSaU4RxijAOF9Zlf0ECy7ObMMuUQHStYIa0j+X2DnES7KJC5s+fz/XXX0+rVq2q/LWqdVBUa/0B8AFAcnKy7KwhqpXV7mDtnsOc2/AeQ84uYYI6CUCeoRZWHU2OjqAEIwDhqoAmykpPYyrBASUAHCquz+zcHnyxqhdL9h5nZKcEerWoI8EuLmv+/PnceuutPhPoR4H4cvfjyh4TwmukZthYMecD7st/l2jy+V635i3nMNaWtCFHWQgyKsKCAujfph6hwQGYTYHkFxazYNsRoosySNZ7GWDcwjMBM3hEz+fDE7fxf0uGMOeHSF66tRVJCRZPf4k1jtXuIMvmIM5icssP1S+//JLJkydTVFTEDTfcwLhx4xg/fjxbtmzB6XTSuXNnZs6cSaNGjRgyZAg2m43i4mImTpzIkCFDAPj888+ZNGkSSinatWvHww8/zMKFC/nuu++YOHEic+bMoUmTJtdc66WoimxBV9ZDX6S1bnOR5wYBE4CBlA6GTtZad77S50xOTtaylouoala7gzVpR4lY8wKDipayy3Udrxb/jh+4ngCjIrwsxOtGmOjWLPo3wZyaYWPdgRxO5DtYsvcE9QoP8ZhhFv2NWzngasAzrkc5G9WGMV0aydX6Ndi3bx8tW7as8PFWu4O3Vh6gxOkiwGjgiT7Nrunc79u3j2eeeYa5c+cSGBjII488QkpKCj/++COFhYU4HA7i4uJ4/vnnKSkpoaCggNq1a5OTk0NKSgoHDhwgLS2N22+/nQ0bNhAdHU1eXh6RkZGMHTuWW2+9lTvvvPOq6rrwvCiltmutky92/BWv0JVS04GeQLRSKgv4CxAIoLX+L7CY0jA/CBQA91W6aiGqgNXu4O9zNjEm8890Yg/vlwzmLddwtDGAhmYTN7eoy8B29S97dZ2UYPnl+TuTbSze3YB/pV3PrPxN/M34P2YY/8JzuQ8xaVl3Vu0/yWtD20ioV4Msm4MSp4s4SyhZtgKybI5rOu8rV65k+/btdOrUCQCHw0GdOnV4+eWX6dSpEyEhIUyePBkonSP+wgsvsHbtWgwGA0ePHuXEiROsWrWK4cOHEx0dDUBkZOS1f6GVVJFZLqOu8LwGHnVbRUK4gdXuYOqaXTyU+RTN9WGedj7MQld3TEFGmtSpdVVtkvPhPrBtfSZ+E8SIE02Z5JrEmwFv807JcT48PJwp639mXNfGEupVLM5iIsBoIMtWQIDRQJzl2s631poxY8bw+uuv/+rxY8eOcebMGYqLiyksLCQsLIypU6eSnZ3N9u3bCQwMpFGjRl7zDldZy0X4ndQMGy/N3MCAHQ9zvT7MoyVPsojuNIoJ45n+LXjn7g7X1PNOSrDw9qgOjO/fmVfMf2O+qxsTDLN4rORT5v6QyVOzdpKaYXPjVyQuFGs28USfZozolHDN7RaAPn36MHv2bE6eLBsoz8vjyJEj/P73v+e1117jnnvu4dlnnwUgPz+fOnXqEBgYyOrVqzlypHQ12969ezNr1ixyc3N/+RwA4eHhnD59+prqqyh567/wK6kZNt74OpXnsp+nuT7MkzzJ5oDONI0K5bWhbdw2eBlrNnFvSkNax9bmpXnP4LCF8UDAYgxOI28fHc2/lqYzaXh7uVKvQrFm9wyGArRq1YqJEydyyy234HK5CAwMZMiQIQQGBnL33XfjdDq58cYbWbVqFffccw+33XYbbdu2JTk5mRYtWgDQunVrXnzxRXr06IHRaCQpKYlPP/2UkSNHMn78eCZPnszs2bM9PyhaFWRQVLib1e7g6a9+YLz1FXrorfzR9QTrg7vSrG44z/RrXmUzUVIzbPxryX6GWt/kLpbyjutOPgsayRM3X8+9KQ2r5DX9TWUHRWsKtw+KCuErVu87wdBjb9GLLfxNjyEtsjdPVsPsk6QEC5PuSmTKupdZ+EMhEwyzsRZFM2NLCK1ja8uURlFtpIcufJ7V7uDLTUcoXPNv7tJL+UTfxvyg2/hdl0bck9KwWtoesWYT47o14av6z7CJdrxq+IBG9o08O3sXy/cer/LXFwIk0IWPs9odvDR/D2lLP+b+c5+zWN/Iu8Z7aVY3nN4t6lRrLbFmE0/1b83b0S/zE/G87vwPKu8gz83dLaEuqoUEuvBpq/efpPDINl52vccWVwteURNIiA7nT/2ae2RAMinBwtODk3mt1kuUYOQd43/Q507xr6XpMvNFVDkJdOGzUjNsfLtpJ5Nc/ySXCJ5wPUmdyNoefyt+UoKFMQO68bzxKa7jKK8b/kve2XO8tihNQl1UKQl04ZNSM2y8/vVO/pT/N8yc4Y/qT0TVaeDWqYnXom/regwbdjcfhdxHP8MWfuecx1FbgYS6qFIS6MLnWO0O3l51kOG5/6W93s/EgEcprtPWa8L8vL6t69H57pdYH9ydR5lBy+J9HM8vrd1qd3i6POGHJNCFz8myOWh/Zj3DXd8y3TCIfVF9Pd5muZSkhpGED3+XbEMdXuctIo0OXC4XWTYJdH9XUlJS7a8pgS58TmFOBmNzJrGf63g/YDQP92jilWF+XvumCRzpOZkYbPzB8S7px0+RX1Dk6bLEBYYOHUrHjh1p3bo1H3zwAQC1atXiySefpHXr1vTp04fs7GwAevbsyRNPPEFiYiJt2rRhy5YtALzyyiuMHj2am266idGjR3P48GF69+5Nu3bt6NOnDxkZGeTn59O8eXPS09MBGDVqFB9++KFbvgZ5Y5HwKamHc4hcNoFASphoeorrYiKJCA3ydFlXpOI7MS/idwzP/4RNzpW8/10o0eHBXv2DyGO+fQ6O73bv56zXFgb847KHTJkyhcjISBwOB506dWLYsGGcPXuW5ORk3nzzTV599VX++te/8s477wBQUFDAjh07WLt2LePGjWPPnj0ApKWlsX79ekwmE7fddhtjxoxhzJgxTJkyhccff5z58+fzzjvvMHbsWJ544glsNhvjx493y5cpV+jCZ6Rm2Eib9VfaFO/mde7nkKseBsO1r7RXHeIsJpaYR7JdteFJ5ye48g7LAKmXmTx5Mu3btyclJYXMzEwOHDiAwWBgxIgRANx7772sX7/+l+NHjSpdiLZ79+6cOnUKu90OwODBgzGZSr8nN27cyN133w3A6NGjf/n4vn370rZtWx599FE++ugjt30NcoUufILV7mDqomW8fnYaS+jCsoCeNIgw8Vjvpj6xAFas2cSEPs3574Kn+E/uwzxb/D4Tcl6SRbwu5gpX0lVhzZo1rFixgo0bNxIaGkrPnj0vuiSuUuqit8vfDwsLu+LruVwu9u3bR2hoKDabjbi4uGv8CkrJFbrwCWv2HWP0yUmcJYRXnWOJqW3y2oHQS0lKsPDQkF58GjqOLmo3tzlXcODEaVbtP+np0mq8/Px8LBYLoaGh7N+/n02bNgGlwTt79mwApk2bRteuXX/5mJkzZwKwfv16IiIiiIiI+M3nvfHGG5kxYwYAU6dOpVu3bgC8+eabtGzZkmnTpnHfffdRXFzslq9DAl14PavdQfGmD2jPj7yhx1IcHMXIzgk+FebnJSVYMHd7kK2qDU+rLwgvOsHi3cdkGqOH9e/fn5KSElq2bMlzzz1HSkoKUHq1vWXLFtq0acOqVat4+eWXf/mYkJAQkpKSeOihh/j4448v+nnffvttPvnkE9q1a8cXX3zBW2+9RXp6Oh999BH//ve/6datG927d2fixIlu+Tpk+Vzh9VZv2kqXpbey09CKx9TzNK1b26fbFFa7gzemL+EfJ37PTmNr3oyZyJibrmNg2/qeLs1jvHX53Fq1anHmzJnfPN6zZ08mTZpEcvJFV7F1m8ounytX6MKrpWbYiFj9Ai4NbwQ8SJO6tT22Tou7xJpNjB3Uk1kRY0lx/kDT3NV8semIDJCKayaBLryW1e5g6YIv6XBuC58FjiAwsiGjUxr6ZKvlQkkJFmJ6P8aRgMY8UTKFoyey+dfSdGm9eJmLXZ1D6SBqVV+dXw0JdOG1dh7JZlTe+xyhPv89dwtGg4HEeLOny3Kb9g2j+cT8GHV0LmNLZpGRe5YdmXZPl+Uxnmr/equrOR8S6MJrxez7nIb6KO8H30dISAgD29b36VbLhWLNJpol38wiQy9+xyIaODPJO1sz30EaEhJCbm6uhHoZrTW5ubmEhIRU6uNkHrrwSrvTD9Iy/T02qkRWuzrQpkFEtW9YUR16tajDP/c+QvfMLTxd8hH/2dW8Rm5bFxcXR1ZW1i9vrRelP+QqOz9dAl14ndQMGyfmv0RL7eCjWuNpFB7GsI5xfnV1fl6s2cQtndsyO38M4069R9zJNby2yOVzc+yvVWBgII0bN/Z0GT5PWi7Cq1jtDmYuXcPNjiXMpi/7iusRFhzoV73zCyXGm9loGcwRFcsjJZ+TbT8tS+yKqyKBLrxKls3BUNunFBPEZwHDqedDb++/WrFmE4/0acG02vfTGCt3sFKW2BVXRQJdeBXX0R2kFKxhumEQdqPF65fGdZekBAvJfe8hVbVibNEMso4flyV2RaVJoAuvYbU7CPv+7+RTizkhd9C8brhPLI3rLhFhwSyo8wgWTjHaOZ/pWzOl7SIqRQJdeI2MH5bR1rGVmcF3cppQn1ka113iLCYyTC1YZuzOSOciggqO1+h56aLyJNCFV0g9kodl0xtkE8l0+tHAEur3vfMLxZpLxwsWR4/DiJO+uV/KkgCiUiTQhcdZ7Q6WL55N86K9zAi5izqRZr95i39lJSVYuKVrCqtD+zHYuQJn7mGZ8SIqrEKBrpTqr5RKV0odVEo9d5HnE5RSq5VSqUqpXUqpge4vVfirHZl2Btk+J0dFskD18vtpileSGG9mSeRoXCjGlHwlM15EhV0x0JVSRuBdYADQChillGp1wWF/Br7SWicBI4H33F2o8E9Wu4Pd3y+mddFuPjcMJSbSXONaLReKNZvo36UD84z96FeymoJj6TLjRVRIRa7QOwMHtdaHtNZFwAxgyAXHaKB22e0IwOq+EoU/25Fpp8/Jz8hTZuYbbmZQ2/o1stVyoYjQIDbUG00xgTzgnCkzXkSFVCTQGwCZ5e5nlT1W3ivAvUqpLGAx8JhbqhN+z5C1hWTXTmYG3YEKCiUyrOZMU7ycOIuJM4FRzAkYxM2u7zGf+UlmvIgrcteg6CjgU611HDAQ+EIp9ZvPrZR6UCm1TSm1TRbhEakZNurtmEwetZnh6kPTmFo1unde3vkZL+uiR3COIPrmTZMZL+KKKhLoR4H4cvfjyh4r737gKwCt9UYgBIi+8BNprT/QWidrrZNjYmKurmLhF6x2B4u+XUTiuW3MCbmdetGRfrsA19VKSrAw5Kb2rAwbxC2udejcn2TGi7isigT6VqCZUqqxUiqI0kHPhRcckwH0AVBKtaQ00OUSXFzSjkw7A/O+IF+FM1v1q/EzWy4lMd7MKstdlGDkd875nD1XLK0XcUlXDHStdQkwAVgK7KN0NstepdSrSqnBZYc9BYxXSu0EpgNjtaxULy7BanewdcNqOp7bzDTDbURGRtX4mS2XEms2MfqWFL4L60/f4lUU5GQwZ3uWXKWLi6pQD11rvVhrfb3WuonW+m9lj72stV5YdjtNa32T1rq91jpRa72sKosWvm1Hpp3uJ7/gLKHMMvSXmS1XkJRg4Wzyoyg041jIwewzcpUuLkreKSqqXdHJg3R3bmJh0ACcQbVlZksFBEc3YmlATwYWLye0KKfGblUnLk8CXVQrq91B9K7/UYKRz/UAmdlSQYnxZtbXHU0AJYx2fc2mQ7nSdhG/IYEuqlXagQN0zl9CqqU/deonyMyWCoo1m+iWksKW0B4M08twFtik7SJ+QwJdVJvUDBtnv3uHAEqYXDiA8BCZ2VIZifFm1kaPItjloO3xuTIvXfyGBLqoFla7gw9X7KT3ma9ZH9AFp7kxA9vWl6vzSog1m2jbqTu7gpIY4fyGk3n5Mi9d/IoEuqgWWTYH3U8tIpwCPlVDZN75VUqMN7PcMoIobeMW51pZiVH8igS6qBanTp+hj302W1Rb9hmaMqpTvFydX4VYs4l23Ybyo2rEXUXz+fF4vqzEKH4hgS6qnNXuIPO7T4nBxszgYTVur1B3iwgLZnXkCBqTxU2u7bISo/iFBLqocjsy8rjFPpMDhuvYqtrVuL1C3S3OYmJbrV6cUNH8Ti+U5QDELyTQRZWy2h0c3TibBs4svjAOpUFkmLzN/xrFmk080qcFy2oPo60zjUjbLhbvPiZX6UICXVStHRk2euVOIzcwlsx6fWvsXqHulpRgoU6P8ZxVtXgk6BtOF8pVupBAF1XIanewe8Nimhbt5xN9K2GmEJnZ4kZtr4vje8tgWp9ax9lj6bJol5BAF1VnR6adbtnTsakIFqqepFwXJa0WN4o1mzjV/gFKMDKaxbJol5BAF1Wn5EQ6Nzq3sTBwoGwvV0VCI2NZGdCNW4pXEVRkl0W7ajgJdFElrHYHkbs/5BxBTNN9ZRGuKpIYb2ZTnZGYOMcdeoUs2lXDSaCLKpF28BCdTy0j1dKPevXjZBGuKhJrNnFDlx7sDU5kFEspcDik7VKDSaALt7PaHRRt/IAgXcS7hf1kEa4qlhhvZl3UXZhLsml0coVMYazBJNCF2+06fJyutvmkhd+IIeZ6WYSrisWaTSSkDOVYQBz3GRZz2lEkV+k1lAS6cCur3cGJdZ9R22Xn/cL+cnVeTRITIlkXOYx4x36Cj22VKYw1lAS6cKsdGXn0zJvFj4YmbKGVTFWsJrFmE852o8inFqP5RqYw1lAS6MKtTIdX0VBnMTtoCCFBATJVsRpFRJj5OrAfN5ZsxlJklSmMNZAEunAbq91Bg30fc4IoFjlvkKmK1Swx3kxq3eFoFKNci2UKYw0kgS7c5tDuDVzvSGVrvbtoVt8iUxWrWazZRJ8bEtkS2oMhehXFBXZpu9QwEujCLVIzbLDhHc5i4r1TXWUw1EMS482sj76LEFcBrY8vlH1HaxgJdHHNrHYHXy7bSIrjO5YE3UJ4RKRMVfSQWLOJNp16si+wNcOd33A877TsO1qDSKCLa7Yj004P2xwUmhmGgbJfqIclxptZab6T+vok3V1bZN/RGkQCXVwTq93Bqp0/0evsYlYbuhAQ1Ug2sPCwWLOJ5j1GcJS6DCtawJG8AgKNytNliWoggS6uSZbNwQ32bwingFWW4bKBhZeICDPxffQw2ul0ugQf5lh+oadLEtVAAl1ck0DlpFvuLHYaWrHxXCPqR4R4uiRB6b6jW8yDOEMofexzZH2XGkICXVwTve9r6ulsNtYZRYt64RQ7tadLEpS2XXq3b8K68AH0dG4g8MxRmcJYA0igi6uWeiSP2j/8l0xVnxmnWmM0GIizSO/cWyTGm9kUfSeg6XhitkxhrAEk0MVVsdodLPl2Pk2L05kXPIS6ESaZquhlYs0mbuiQxJaQmxjiXE5eXq5MYfRzFQp0pVR/pVS6UuqgUuq5Sxxzl1IqTSm1Vyk1zb1lCm+zI9NOH9ss8lU4X6ueMlXRSyXGm1llHk44ZxnoXC1TGP3cFQNdKWUE3gUGAK2AUUqpVhcc0wx4HrhJa90a+EMV1Cq8hNXuYNv2rSSf28g8Qz+iIy0yVdFLxZpNdO7Wj93qeoYVfU1m7mmZwujHKnKF3hk4qLU+pLUuAmYAQy44ZjzwrtbaBqC1PuneMoU3ybI56GGbhVMFsCHqdpmq6OUiQoPYVHckcRxnQFCqTGH0YxUJ9AZAZrn7WWWPlXc9cL1S6nul1CalVP+LfSKl1INKqW1KqW3Z2dlXV7HwuALbCW7IX8IS1Y0DBWEyVdHLxVlM7AnvzjFVh772OTI46sfcNSgaADQDegKjgA+VUr9pqGqtP9BaJ2utk2NiYtz00qI6We0O8tb+jxCKmBM8lIaRoTJV0cvFmk30axfHsvDbaa/3UTtnhwyO+qmKBPpRIL7c/biyx8rLAhZqrYu11j8DP1Ia8MLP7Dp8kl6n5rPF2IFDKh6DTFX0CYnxZjabB3GaUEa5FnH2XLHMS/dDFQn0rUAzpVRjpVQQMBJYeMEx8ym9OkcpFU1pC+aQG+sUXsBqd5Cz8UssLhtTDbfRwBIqg6E+ItZsYvzN7VhTaxBdi79H2zPl3aN+6IqBrrUuASYAS4F9wFda671KqVeVUoPLDlsK5Cql0oDVwJ+01rlVVbTwjB0ZNnrkfYU1pAn2ejfKYKiPSUqwENbtEUAxIXQFpwvlKt3fVKiHrrVerLW+XmvdRGv9t7LHXtZaLyy7rbXWf9Rat9Jat9Vaz6jKokX1s9od/Lz5a+KLD/OpaxDhpiCZd+6DWjRvSWp4Tzrmfk12TrZcpfsZeaeoqJAsm4NeeV9hN0axJ/JmeVeoj4o1mzjX6WHCcHC/aR1OeaORX5FAFxXiPLabVo5tTKc/x864ZKqiDwtrlMwOQ2tuyp3N7oxc8guKPF2ScBMJdHFFVruDkg3v4iCYxcH9Zaqijyt2atZE3kU9cujt2sj0rZnSdvETEujiitIOHCDlzCqWBfYhn1oyVdHHxVlM7A7rQoaK5R79NWcLi2Rw1E9IoIvLstodlGz8H0ZKmKoGyVRFPxBrNjGhz/UsjxhGc+dB6thTZXDUT0igi8va9fMxutoXkBbeFVO9ZjJV0U8kJVho0GMcpw21+X3gtzKF0U9IoItLstodZK/7hFqu07x3rj/hIbJErj9p17g+35uH0Or095w9ls6c7Vlyle7jJNDFJe3IyKOnbTb7Dc3YrpuTcl2UtFr8SKzZxJn291GCkdEs5mD2GblK93ES6OKSQg8vJ15bmR00hJCgACLDgjxdknCz0MhYVgT0oF/xSoKK7OSdlSmMvkwCXVyU1e6gwb6POU40i52daRpTS9otfigx3symOiMJoYg79TI2HcqVtosPk0AXF/XzrnU0c+xia70RNKtvZljHOGm3+KFYs4mULt3YHdKRu1lCoeOstF18mAS6+I3UDBtsfJezmHjv1E0yGOrnEuPNrIm+m/CSPJodWyQbYPgwCXTxK1a7g6nLvucGx1q+DbqF2hEWWbfFz8WaTTTpPJCfAptxr2sBx/JOywYYPkoCXfxKls1Bn/x5KGAqAwgLlqvzmiAxwcIS8ygauI7Rw7UZlyza5ZMk0MWvBDvP0O30YlYZunBcxTCqU7xcndcAsWYT1/cYSYaqz8iiuRzJPUugUXm6LFFJEujiVwJ3TaUWBfwQew+J8WYiQmWqYk0REWZibfQoWuqf6BO8j2P5hZ4uSVSSBLr4Rerhk8Ts/YQdqhWLbbEYZRGuGiXOYmJ7RD9ylIV+9hkyOOqDJNAFUDoYumXRFGKcJ5hrup16tYNlMLSGiTWb6Nu+IUtq3UEn1y5q5eySwVEfI4EugNL9QvvZZ3LEEM93uoMMhtZQifFmNlgGc5pQRjvncfacLNrlSyTQBVa7g8ObF9Ko5BBfGgYTG1lLlsitoWLNJsbf3J5VtW6ja8lGjPafZWldHyKBLkqnKuZNxxYQzd7ofrJEbg2XlGAhrMdjlBDAI4HfyL6jPkQCXaCzttHcsYMv9SDZL1QAEFU3jm8D+tD51FIyM36WfUd9hAR6DWe1OzBsfJvThPJtcD/ZL1QApfuOro8ZiREXI52LZN9RHyGBXsP9mLaDjgXrWBQ0kDOYZL9QAZROYcwNjmON8UZudy4FR54MjvoACfQazGp3ELD5HYoJYJoaIPuFil/Emk081rspy6LuIQwHXfPmyuCoD5BAr8HSDhygc/5SUi39iawbL4Oh4leSEiz06NaL7aabGOlcRHGBXa7SvZwEeg2VmmHj7HfvEEAJkwsHyBK54qIS482sjBlNqOsM7Y/Nkn1HvZwEeg1ktTv4cMUuep/5mu8DuuA0N5Z3hYqLijWbiG11I9+rDtzj+pqjJ7PlKt2LSaDXQFk2Bz1OLSScAj5VQ+RdoeKyIsOC+DJ4BBH6NAPPfSv7jnoxCfQa6PSpfPraZ7FJtSfN0FSWyBWXlRhvpqheR7aodtzrXMDynT/Lol1eSgK9hrHaHZz47gMiyWdq8F00rxsuS+SKy4o1mxjWMY5vIn9HJPl0zFkgi3Z5qQoFulKqv1IqXSl1UCn13GWOG6aU0kqpZPeVKNxp1+ET9LfPZKexDTsNrWTeuaiQxHgzmeGJpBpac3fJfIoKZTNpb3TFQFdKGYF3gQFAK2CUUqrVRY4LB54ANru7SOEeqRk2jq75iEhXLlMMw2Teuaiw8/PSF1tGE63zaJ/zjayX7oUqcoXeGTiotT6ktWk3sjYAABmsSURBVC4CZgBDLnLca8AbgGxz4oWsdgfvrdzPwPwZpBmbczzqBpl3LiolKcFCUo8hpAe2ZIxzLify8qX14mUqEugNgMxy97PKHvuFUqoDEK+1/uZyn0gp9aBSaptSalt2dnalixVXL8vmIOX0MurrbKaoYYSFBMnMFlFpiQkWvrGMpo7OYZBztWwm7WWueVBUKWUA/gM8daVjtdYfaK2TtdbJMTEx1/rSohIClZN+tumkq8Z8b+ggM1vEVYk1m2jb/Q52q+u5t+grrLl22Uzai1Qk0I8C8eXux5U9dl440AZYo5Q6DKQAC2Vg1LuovfOI08f4ru4YEhMsMrNFXLWIsGBW1h9PXXIZZVwlm0l7kYoE+lagmVKqsVIqCBgJLDz/pNY6X2sdrbVupLVuBGwCBmutt1VJxaLSUg/nEJ36Nj+pBKadaiebP4trEmcx8VNYR1INbRh8agYzN6TL4KiXuGKga61LgAnAUmAf8JXWeq9S6lWl1OCqLlBcG6vdwdZFHxJXksH0kJHUjTDJ2/zFNYk1mxjQLpZ5lrFEYeeGnLkyOOolAipykNZ6MbD4gsdevsSxPa+9LOEuO49kM9j2GT8ZGrOcG2gib/MXbpAYb2Z2eCKbbUncUzKXzY7B7Mi0y4WCh8k7Rf2Y1e7gxNpPqOc8xvuGEbL5s3Cb8/PSv466jwh9mpSc2bISoxeQQPdj36Vl0S/3c9IMzVinkhnUtr7MOxduk5RgoUXHnqxVnbjXuYAjR4+yav9JT5dVo0mg+ymr3UHh5k+oTw6TXSMINBqIDJOZLcK9IsOC+DhwJOEUMKxogexq5GES6H5q18/HGHp6GjsNrfghoD0JUWHSOxdulxhvpiimDcvVjYxVi6lVnCtrvHiQBLofSs2wkbP6PSwuGx8EjKJJ3dr8qV9z6Z0Lt4s1m3imX3O+rfMAgRTTL/dzWePFgyTQ/YzV7uDj5akMPjWd7QGJ5EQly5otokolJVi4ueuNrAwdwFDnMlTuAZnG6CES6H4my+ZggH06tTjL2+oe2Y1IVIvEeDPfRo2hkCB+XzxV1njxEAl0P3Mu5zA3n57HEtWNdMN1smaLqBaxZhODUtoxzTiEHs6NBBzbTn6BbFVX3STQ/YjV7sCw5u8AfBZyr+xGJKpVRGgQ2+rfTZ4yM6Hkc6ZvyZC2SzWTQPcjP+3eSJezK5gfdBvHVIzsRiSqVZzFRHFAGJ8EjCBR76P5qfUy46WaSaD7idQMGxHrJ3KaMKao22U3IlHtzr97dEfMYDJULHfZpzB14yGZ8VKNJND9gNXu4NsF02h3bjtfBN2FOTJaZrYIj0hKsDCqSxNmRYyjsc6k1YkF/GtpurReqokEuh/YeSSbUXnvk0ldPizsjdFgkJktwmMS481sD+3KD7o5DzlnkJtzUlov1UQC3Q9E7/uCxjqT94LGERIiy+MKz4o1mxjULpa3Au/HwilGF88i76zMeKkOEug+bnf6QVqlv8tGlcgq3ZE2sRH0blHH02WJGq5XizoExHXgG0MvRrgWsz11m/TSq4EEug9LzbBxbN6LBOtCPgp7kEbRYQzrGCdX58LjYs0mhnWM4+voBygmkDuy3+e1RWkS6lVMAt1HpWbYmDpvATcXLmUaA9hXXFfeFSq8SmK8mZLQOnxqHEY311bq5WyQAdIqJoHug6x2B2+vPMAY+3vYqM2nAcOpF2GSaYrCq5yfxris9jAydB2ecn3K0ZxTMkBahSTQfVCWzcGNp5fSVqfzvvFuzJExvHRrK5mmKLxOUoKF4Tc04e3AsTQhi6Eli2WAtApJoPugAttxhuf9l52qBYuNvXm4RxMJc+G1erWoQ16DvmxQHXjQOZ0NO3ZLL72KSKD7mNQMGyx9gVAK+U/Io1xfL0LWaxFeLdZsYlhyPNOjHyMAF8NOviMDpFVEAt2HpGbYWDj3S3qeW80UhnDAFSvrtQifkBhv5mxYPJ8Zh9HHtZG43A0S6lVAAt1HWO0O/rdiDw/Y3+Yw9fkyYJgMhAqfcX6AdEXkSI4Qy9POD8mz22UjDDeTQPcRWTYHt9q/oAEn+Kfx98REmmUgVPiUpAQLz9/WninmCSRwgrF6gWyE4WYS6D7CdTSV/qdms0j1IjWgrQyECp+UlGCha99hLDN04+7iORQe2ycbYbiRBLoP2PHzcWJXP4mN2vzPNE42rhA+LSI0iEX1J+DAxIsl7/K/NT9KL91NJNC9XGqGjUOz/kyC8wh/5SFynaEyECp8WpzFxJnAKP4TcD9t9Y90zZ0tA6RuIoHuxax2B/MWzmVIwRzm0JutgR1lIFT4vPMDpLstfVmrknnIOQ1XzkFZFsANJNC92No9h3kg558cJ4q/O+8lJjxEBkKFX0hKsPDSba35KOIxigjkz873OXg8n1X7T3q6NJ8mge6lUjNshK2bSALHeUk/jCG4NiM7J0iYC7+RlGChX0oSbxrHkqz2cWvxt8zYkiGtl2sgge6FUjNsfDvnE247t4gv9UB2BrSlWd1wWedc+J1eLeqwv95gvieRPxmmYTr1k/TTr0GFAl0p1V8pla6UOqiUeu4iz/9RKZWmlNqllFqplGro/lJrBqvdwUeL1/NI/n/YRyP+F3gvDaPC+FO/5tI3F34n1mzimf4tmBL1FEUE8krxWxzNsUs//SpdMdCVUkbgXWAA0AoYpZRqdcFhqUCy1rodMBv4p7sLrSnW7DvGuON/J4gi/uh8nIjataVvLvxaUoKFCUO68U6tx2mtDjHe+RUHTpyWfvpVqMgVemfgoNb6kNa6CJgBDCl/gNZ6tda6oOzuJiDOvWXWDKkZNvjun3QkjYn6AXKCE6RvLmqEpAQLjbqOZK66mXFqAS3P7ZR++lWoSKA3ADLL3c8qe+xS7ge+vdgTSqkHlVLblFLbsrOzK15lDZCaYWPO3JmMKJzJQt2dpQE9pW8uapReLeqwsN5jZFKffxrf49ypbOmnV5JbB0WVUvcCycC/Lva81voDrXWy1jo5JibGnS/t06x2Bx8u/p4/2F8nk7r8J3C89M1FjRNrNvHEgPa8Y3mGaOw8V/weR3LOSD+9EioS6EeB+HL348oe+xWl1M3Ai8BgrfU595RXM3yXlsXDx1/BRCETnH+kVm2L9M1FjZSUYOHu24fyqWksfdRW7nEukH56JVQk0LcCzZRSjZVSQcBIYGH5A5RSScD/KA1zOfOVkHokj8jvXqAtB3hRP8qJ4MbSNxc1WlKCBVP3x1mhUnhCTZN+eiVcMdC11iXABGApsA/4Smu9Vyn1qlJqcNlh/wJqAbOUUjuUUgsv8elEOakZNjbPnkS/c8v4UN/OuoAU6ZsLAfRqWZep9Z4lg1jeNE6GU0eln14BSmvtkRdOTk7W27Zt88hrewOr3cEHX07lxZxn2URbngl4jvqR4dJqEaJMaoaNKfOX8EbeH0jXDXnI+Bea1I9i0vD2NXpsSSm1XWudfLHn5J2iHrJt+1b+kPMKR4nhKeejRNUOkzAXopykBAvjhvZnctgTJKl0nnZO4cDxU9JPvwwJdA/YnX6Q5O8fxKXhYf08OtgsfXMhLiIpwUJ8t3uYom5nuFrBsOKF0k+/DAn0arbj0DGCZt1DpCuXx3mWkwH1pW8uxGX0alGHFfUfZAUpPGP4khb566SffgkS6NVo+R4rp6eNo1lJOi/wOAeCWsh8cyGuINZs4k/9W/JRzDPsU014reT/CD65i5fm75FQv4AEejVZvucY+XMep1vJBl533svagBQaWEKlby5EBSQlWHj2tg78I+Jl8gjnHf13Sk7+KKF+AQn0apB6JI/8hc9yp17OB64hfK4HEBkaJGEuRCUkJVjon5LIo4Y/o9B8ZPw7RXlZ0n4pRwK9iqVm2Phx5ovcWbSAz539+LdrBLWCg/hTv+YS5kJUUq8WdYhq2IbHDH/GzFneYyKn805IqJeRQK9CqRk2Uqf/lREF05hLL94OHEfDyFr844629G1dz9PlCeFzYs0mXhvahgH9BvC32i8RzwkmF79Kzsnj0n5BAr3KpGbY2Dn9L4xzfMIiVxf+XPIAdc1hvHFnOwlzIa5BrNnEvSkNuWv43fyt1p+5jiz+q/9KzklrjQ91CfQqsHzPMfZ/+TRjHZ+xwHUTL/IoIUHBMtdcCDdKSrDQvOvt/MHwLNdh5RPjRM7kHufZ2btYvve4p8vzCAl0N1u+x0runCcZVTSbac4+POt6BENAMEkJZplrLoSb9WpRh6KGPfmD4Xkac4yPeZmSvMM8N3d3jQx1WcvFTax2B6vTjhK16mn6l6xmimsQb7juITYilAe6XUevFnVkrrkQVcBqd7Bq/0k2r1nERMffOEcQDzif5XREC7/8vydruVQxq93B3+dsoumysfQvWc3/lQznDdc91AoO4oWBLbknpaFffUMJ4U3O99QH3zaMB4wTcaKYavwrCae2MmlZOi/N31NjNsiQQL9GVruDuSu/58nMR+mg0/hTycN8Yhwus1mEqGZ9W9fjwWEDeTr8X5xQ0XxseJ07S75h2+E8pqz/uUaEeoCnC/Bly/ceZ/WS2Tx9+g2MuoSxJS+wXbWiaVQorw1tIwOgQlSzvq3rER3eh+fnhfNQ7hu8aPiUZs7D/HPbeDYeyuUPfZr59UWW9NCvgtXuYO72TIrXvcXjeio/6/o8ztMcVXHES5gL4XGpGTZenreLW22f8XvmsMt1HU86Hyc3qAHjuzXm9g5xPtsGlR66G6Vm2PjLzPU0XzeBJ/mSZboTdxS/xlFDHK3iIiTMhfACSQkWXr29Hd81eJA/8DQN1XEWBDxPz6Lv+GDtIZ6atdMv56vLFXolLN97nGXfzOKpgjeJ0nYmOUfyiR5EaFCgz//UF8IfWe0O5v6QxTdrtzDR9RYdDeks0N2YxH1ERNXxyQuwy12hS6BfgdXuYEemnT1HjlFn25v8Tn/NYerxjGsCPwVeT6zZ5Pd9OSF83fK9x3l7xX4G2r7gfj0PO7V41TWO9MheDGkfS+OYWiTGm33igkwC/SqlZtiY+E0a0Sc38bzrfzRSJ5jh7MM/9L1YIix+OcdVCH9ltTuYsv5ndm1bxyv6PVqpwyx3JfMP173YghrQKNo3toGUQK+k829U+Pr7VO7Kn8Iww3ccdtXlpZL72azaEh4SKFMShfBBVruDl+bvYfeRHO4qmc/DhnkE4ORj1yA+0kOIjopmTJdGXn2hJoFeQeeDfN7mA/SwzWKcnk8QxXzsHMRk5x0EhYQSZwmVFosQPuyX/+epWZw5mcHvS77kDuN6TmozbzuHsTigD/UjazOqc4JXBrsEegUs33ucd1ek0dH2LeP0XBqoHJY6OzHJOYqTQXHER4Z67T+wEKLyzgf7zC0ZROTt5EnXZ3Q0/IhVR/F+yWC+DbyZupERXncBJ4F+CecHPHcfOY5r2+eM1fOpr/L4wdWUSc5RbFetaBgV5vW/ggkhrt75YP98w880yNvEo4Y5JBt+5JiO5EtXPxYZ+9K/c0vax1u8YuBUAr2c8yF+KPsMP+zcxU35XzNEryRanWKrqzmTS+5gE20IDwnymUESIcS1Oz8J4ueTZ2hTtIOHDPO50ZhGoQ5kketG5gUMIN/ShgFt6nl0VowEOv//p/DszYeIt29miHMZvdQPAKxydeBTZ3+2qtaEBgUyolOc1/w0FkJUn/MXfDsybXy1NYsGRT9zt1rK7cb1hKpz7HfF87XuysqAHgRY4jzShq2xgX4+xNOO2tCHN9Lx9Ep6641EqjPk6Np85ezJNGcfrMQQHhIgA55CiF8s33uct1YeINPmgEI7QwwbGGJcT0fDAVxasVm3ZJ2hE3tr3UiLVu2q7SKwxgR6+XZKbs5JAn9eTauzm+imdhClTlOgg1nh6shCVxfWOtujjIGEBQcQazbJgKcQ4jfKD5wetTs4c66YBq7jDDWup79hK80NmQAccDVgQ0AyP9bqhDH+BupGWaqsLeO3gZ6aYWPdgRyMCkpOnyA/fR1NHbtp60yjlTqMUWnydC3Wudqz0pXEGt2BcwYTYUGlIe7pXpgQwjeUv1hcuuc4R+0OThcWU1+f5GbjD/QxbKez2k+gclKkjezWTdhpbM1BUzvqNE8hIDwGl4ZuzaKveUzOrwLdaitg34EfyTm4lZwD22ni/ImWHKah4SQADh1Eqm7KdlcLVjvbsVM3RSsDQUZFs7rh9G8tIS6EuHrl++xzth/lzLliip2a2qqADupHOhv2cYPaR1t1iADlAuCojiZNN+KAsSlRzToTeX0XWjdtfFUZ5DeBbrU72P7Fi9yW+/Evjx3WdUlzNWSnqwnbdQt2uhpTQgAGBYFGRcOoMDokWGjTIEJaKkIItzrfJTh7rpgNP+Vy4MRpipwarSGUQpKMP9GKn2lj+Jm26mcaG0r3OX0z4H5+uu5eXhjYstKZdLlAr9AGF0qp/sBbgBH4SGv9jwueDwY+BzoCucAIrfXhSlVZAVk2B9sCk/gpcDzbzsWxsziOM9qEBoxlAV47JID+beoRGhxAo6gwCXEhRJVJSrD80kI5328/knuWgnMlLNl7gq3n2vB9SWuUs/T4cFVAYmAmZwPiCSssIcvmcGs+XTHQlVJG4F2gL5AFbFVKLdRap5U77H7AprVuqpQaCbwBjHBblWXiLCbyardmhT0WuyomNMxI/bAgejWPITwk0G09KiGEqKzze5ued2fy/x/jyy8sZk16NraCYLYVhWMhkMSQAOIsbh4wvVLLRSnVBXhFa92v7P7zAFrr18sds7TsmI1KqQDgOBCjL/PJr7qHXta/yjtbRGRYkPTChRA+wV3Zda0tlwZAZrn7WcANlzpGa12ilMoHooCcSld7BbFmkwS4EMLnVEd2VesWdEqpB5VS25RS27Kzs6vzpYUQwu9VJNCPAvHl7seVPXbRY8paLhGUDo7+itb6A611stY6OSYm5uoqFkIIcVEVCfStQDOlVGOlVBAwElh4wTELgTFlt+8EVl2ufy6EEML9rthDL+uJTwCWUjptcYrWeq9S6lVgm9Z6IfAx8IVS6iCQR2noCyGEqEYVmoeutV4MLL7gsZfL3S4Ehru3NCGEEJVRrYOiQgghqo7H3vqvlMoGjlzlh0dTBVMi3UDqqhypq/K8tTapq3Kupa6GWuuLzirxWKBfC6XUtktNrPckqatypK7K89bapK7Kqaq6pOUihBB+QgJdCCH8hK8G+geeLuASpK7Kkboqz1trk7oqp0rq8skeuhBCiN/y1St0IYQQF5BAF0IIP+ETga6U+pdSar9SapdSap5SynyJ4/orpdKVUgeVUs9VQ13DlVJ7lVIupdQlpyAppQ4rpXYrpXYopa5tZ2z31lXd5ytSKbVcKXWg7O+L7kSilHKWnasdSqkL1w1yZz2X/fqVUsFKqZllz29WSjWqqloqWddYpVR2uXP0QDXVNUUpdVIptecSzyul1OSyuncppTp4SV09lVL55c7Xyxc7zs01xSulViul0sr+Lz5xkWPcf7601l7/B7gFCCi7/QbwxkWOMQI/AdcBQcBOoFUV19USaA6sAZIvc9xhILoaz9cV6/LQ+fon8FzZ7ecu9u9Y9tyZajhHV/z6gUeA/5bdHgnM9JK6xgLvVNf3U7nX7Q50APZc4vmBwLeAAlKAzV5SV09gUTWfq/pAh7Lb4cCPF/l3dPv58okrdK31Mq11SdndTZQu4XuhzsBBrfUhrXURMAMYUsV17dNap1fla1yNCtZV7eer7PN/Vnb7M2BoFb/e5VTk6y9f72ygj1JKeUFdHqG1Xkvp4nuXMgT4XJfaBJiVUvW9oK5qp7U+prX+oez2aWAfpRsBlef28+UTgX6BcZT+VLvQxXZWuvAEeooGlimltiulHvR0MWU8cb7qaq2Pld0+DtS9xHEhZRuhbFJKVVXoV+Tr/9VOXMD5nbiqUkX/XYaV/Zo+WykVf5HnPcGb/w92UUrtVEp9q5RqXZ0vXNaqSwI2X/CU289XhVZbrA5KqRVAvYs89aLWekHZMS8CJcBUb6qrArpqrY8qpeoAy5VS+8uuKjxdl9tdrq7yd7TWWil1qTmzDcvO13XAKqXUbq31T+6u1Yd9DUzXWp9TSv2e0t8ienu4Jm/2A6XfU2eUUgOB+UCz6nhhpVQtYA7wB631qap+Pa8JdK31zZd7Xik1FrgV6KPLGlAXqMjOSm6vq4Kf42jZ3yeVUvMo/bX6mgLdDXVV+/lSSp1QStXXWh8r+9Xy5CU+x/nzdUgptYbSqxt3B3plduLKUpfZiau669Jal6/hI0rHJrxBlXxPXavyQaq1XqyUek8pFa21rtJFu5RSgZSG+VSt9dyLHOL28+UTLRelVH/gGWCw1rrgEodVZGelaqeUClNKhZ+/TekA70VH46uZJ85X+Z2txgC/+U1CKWVRSgWX3Y4GbgLSqqAWb92J64p1XdBnHUxpf9YbLAR+VzZ7IwXIL9di8xilVL3zYx9Kqc6U5l6V/mAue72PgX1a6/9c4jD3n6/qHPm9hhHjg5T2mnaU/Tk/8yAWWHzBqPGPlF7NvVgNdd1Oad/rHHACWHphXZTOVthZ9mevt9TlofMVBawEDgArgMiyx5OBj8pu3wjsLjtfu4H7q7Ce33z9wKuUXjgAhACzyr7/tgDXVfU5qmBdr5d9L+0EVgMtqqmu6cAxoLjs++t+4CHgobLnFfBuWd27uczMr2qua0K587UJuLEaaupK6djZrnK5NbCqz5e89V8IIfyET7RchBBCXJkEuhBC+AkJdCGE8BMS6EII4Sck0IUQwk9IoAshhJ+QQBdCCD/x/wCYUpOv3H2L5wAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2: Supplying xgboost a gradient and a Hessian\n",
        "\n",
        "This demo is taken from https://towardsdatascience.com/jax-vs-pytorch-automatic-differentiation-for-xgboost-10222e1404ec which also shows how to do this using pytorch.\n",
        "\n",
        "The basic idea of gradient boosting is the incremental learning of the residual from the previous weak learner..quite like a resnet. This residual can be described as a gradient in function space.\n",
        "\n",
        "Any loss function can be minimized in gradient boosting, since there, the basic idea is to replace it by the mean squared error followed by a beam serach. You can think of this as roughly needing to model the gradient at any step by a week learner in the MSE sense, and thus we need both the gradient, and the gradient of the gradient, or the Hessian.\n",
        "\n",
        "xhboost provides tons of loss functions, but also gives us the ability to write custom loss functions which we want to minimize. This is a usual trick in Kaggle competitions, and I have personally used it for losses like focal loss which are not defined in the package.\n",
        "\n",
        "We'll use a custom loss function here, calles the Squared Log Error (SLE):\n",
        "\n",
        "![](https://miro.medium.com/max/1400/0*sCTEDe1H7ee8gfX0.png)\n",
        "\n",
        "As the article says:\n",
        "\n",
        ">Note that this loss penalizes an under-predicted estimate greater than an over-predicted estimate. It could reflect a real business requirement when predicting house prices, and we are able to fulfill it by choosing a custom loss function.\n",
        "\n"
      ],
      "metadata": {
        "id": "7dzm9KeMmPXK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "import sklearn.datasets\n",
        "from sklearn.model_selection import train_test_split\n",
        "from xgboost.sklearn import XGBRegressor\n",
        "import jax\n",
        "import jax.numpy as jnp\n",
        "from tqdm.notebook import tqdm\n",
        "import plotly.express as px\n",
        "from functools import partial\n",
        "from typing import Callable"
      ],
      "metadata": {
        "id": "XTdm75RbicmM"
      },
      "execution_count": 20,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We use `grad` to calculate the gradient of the SLE loss, and then use `hvp` provided by jax to calculate the Hessian. As the article says:\n",
        "\n",
        ">Note that we use the hvp (Hessian-vector product) function (on a vector of ones) from [JAX’s Autodiff Cookbook](https://jax.readthedocs.io/en/latest/notebooks/autodiff_cookbook.html#hessian-vector-products-using-both-forward-and-reverse-mode) to calculate the diagonal of the Hessian. This trick is possible only when the Hessian is diagonal (all non-diagonal entries are zero), which holds in our case. This way, we never store the entire hessian, and calculate it on the fly, reducing memory consumption."
      ],
      "metadata": {
        "id": "WMN3z2rroWzM"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def jax_sle_loss(y_true: np.ndarray, y_pred: np.ndarray):\n",
        "    \"\"\"Calculate the Squared Log Error loss.\"\"\"\n",
        "    return (1/2 * (jnp.log1p(y_pred) - jnp.log1p(y_true))**2)\n",
        "\n",
        "def hvp(f, inputs, vectors):\n",
        "    \"\"\"Hessian-vector product.\"\"\"\n",
        "    return jax.jvp(jax.grad(f), inputs, vectors)[1]\n",
        "\n",
        "def jax_autodiff_grad_hess(\n",
        "    loss_function: Callable[[np.ndarray, np.ndarray], np.ndarray],\n",
        "    y_true: np.ndarray, y_pred: np.ndarray\n",
        "):\n",
        "    \"\"\"Perform automatic differentiation to get the\n",
        "    Gradient and the Hessian of `loss_function`.\"\"\"\n",
        "    loss_function_sum = lambda y_pred: loss_function(y_true, y_pred).sum()\n",
        "\n",
        "    grad_fn = jax.grad(loss_function_sum)\n",
        "    grad = grad_fn(y_pred)\n",
        "\n",
        "    hess = hvp(loss_function_sum, (y_pred,), (jnp.ones_like(y_pred), ))\n",
        "\n",
        "    return grad, hess"
      ],
      "metadata": {
        "id": "TGPODy85inbm"
      },
      "execution_count": 21,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We create our training data..."
      ],
      "metadata": {
        "id": "Y8eKwsBOo-sw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "X, y = sklearn.datasets.fetch_california_housing(return_X_y=True)\n",
        "X_train, X_test, y_train, y_test = train_test_split(\n",
        "    X, y, train_size=0.05, random_state=0)\n",
        "print(f\"Train Data: {X_train.shape[0]} examples, {X_train.shape[1]} features\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rNmtrpZditkT",
        "outputId": "c13b9f6a-2c45-42ed-bd9b-36d17f0f9863"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Train Data: 1032 examples, 8 features\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We `jit` the function returning both the loss and the hessian after it has been curried with the input loss function using python's `partial`. This is needed because we need to both `grad` and `hvp` this loss, but the xgboost specification only wants us to provide `y_true` and `y_pred` as inputs. (you saw an example of this \"closure\" strategy earlier when we were talking about functional programming.\n",
        "\n",
        "Then we fit for 100 epochs or 100 trees and do are regular gradient boosting stuff.."
      ],
      "metadata": {
        "id": "koeNEp-0panA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "jax_objective = jax.jit(partial(jax_autodiff_grad_hess, jax_sle_loss))\n",
        "reg = XGBRegressor(objective=jax_objective, n_estimators=100)\n",
        "reg.fit(X_train, y_train, eval_set = [(X_test, y_test)], verbose=True)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mqoC2u8AjY9a",
        "outputId": "333dbd37-4bd1-4920-8661-e29a33b46d2c"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[13:33:11] WARNING: /workspace/src/objective/regression_obj.cu:152: reg:linear is now deprecated in favor of reg:squarederror.\n",
            "[0]\tvalidation_0-rmse:1.89628\n",
            "[1]\tvalidation_0-rmse:1.84526\n",
            "[2]\tvalidation_0-rmse:1.79421\n",
            "[3]\tvalidation_0-rmse:1.74338\n",
            "[4]\tvalidation_0-rmse:1.69278\n",
            "[5]\tvalidation_0-rmse:1.64275\n",
            "[6]\tvalidation_0-rmse:1.59351\n",
            "[7]\tvalidation_0-rmse:1.54489\n",
            "[8]\tvalidation_0-rmse:1.49703\n",
            "[9]\tvalidation_0-rmse:1.45083\n",
            "[10]\tvalidation_0-rmse:1.4057\n",
            "[11]\tvalidation_0-rmse:1.36191\n",
            "[12]\tvalidation_0-rmse:1.3194\n",
            "[13]\tvalidation_0-rmse:1.27866\n",
            "[14]\tvalidation_0-rmse:1.23975\n",
            "[15]\tvalidation_0-rmse:1.20226\n",
            "[16]\tvalidation_0-rmse:1.16705\n",
            "[17]\tvalidation_0-rmse:1.13363\n",
            "[18]\tvalidation_0-rmse:1.1021\n",
            "[19]\tvalidation_0-rmse:1.0719\n",
            "[20]\tvalidation_0-rmse:1.04363\n",
            "[21]\tvalidation_0-rmse:1.01775\n",
            "[22]\tvalidation_0-rmse:0.993397\n",
            "[23]\tvalidation_0-rmse:0.970323\n",
            "[24]\tvalidation_0-rmse:0.948407\n",
            "[25]\tvalidation_0-rmse:0.929901\n",
            "[26]\tvalidation_0-rmse:0.910937\n",
            "[27]\tvalidation_0-rmse:0.894408\n",
            "[28]\tvalidation_0-rmse:0.87842\n",
            "[29]\tvalidation_0-rmse:0.863459\n",
            "[30]\tvalidation_0-rmse:0.849843\n",
            "[31]\tvalidation_0-rmse:0.838266\n",
            "[32]\tvalidation_0-rmse:0.826477\n",
            "[33]\tvalidation_0-rmse:0.813676\n",
            "[34]\tvalidation_0-rmse:0.804198\n",
            "[35]\tvalidation_0-rmse:0.795037\n",
            "[36]\tvalidation_0-rmse:0.787153\n",
            "[37]\tvalidation_0-rmse:0.779947\n",
            "[38]\tvalidation_0-rmse:0.772318\n",
            "[39]\tvalidation_0-rmse:0.764513\n",
            "[40]\tvalidation_0-rmse:0.758461\n",
            "[41]\tvalidation_0-rmse:0.752558\n",
            "[42]\tvalidation_0-rmse:0.745905\n",
            "[43]\tvalidation_0-rmse:0.737557\n",
            "[44]\tvalidation_0-rmse:0.727824\n",
            "[45]\tvalidation_0-rmse:0.722478\n",
            "[46]\tvalidation_0-rmse:0.71812\n",
            "[47]\tvalidation_0-rmse:0.71314\n",
            "[48]\tvalidation_0-rmse:0.707483\n",
            "[49]\tvalidation_0-rmse:0.703849\n",
            "[50]\tvalidation_0-rmse:0.701057\n",
            "[51]\tvalidation_0-rmse:0.695743\n",
            "[52]\tvalidation_0-rmse:0.69159\n",
            "[53]\tvalidation_0-rmse:0.687288\n",
            "[54]\tvalidation_0-rmse:0.683926\n",
            "[55]\tvalidation_0-rmse:0.681515\n",
            "[56]\tvalidation_0-rmse:0.67754\n",
            "[57]\tvalidation_0-rmse:0.675063\n",
            "[58]\tvalidation_0-rmse:0.672494\n",
            "[59]\tvalidation_0-rmse:0.668424\n",
            "[60]\tvalidation_0-rmse:0.665971\n",
            "[61]\tvalidation_0-rmse:0.663711\n",
            "[62]\tvalidation_0-rmse:0.661035\n",
            "[63]\tvalidation_0-rmse:0.659729\n",
            "[64]\tvalidation_0-rmse:0.658141\n",
            "[65]\tvalidation_0-rmse:0.656401\n",
            "[66]\tvalidation_0-rmse:0.65518\n",
            "[67]\tvalidation_0-rmse:0.651945\n",
            "[68]\tvalidation_0-rmse:0.650803\n",
            "[69]\tvalidation_0-rmse:0.649685\n",
            "[70]\tvalidation_0-rmse:0.646986\n",
            "[71]\tvalidation_0-rmse:0.645028\n",
            "[72]\tvalidation_0-rmse:0.64308\n",
            "[73]\tvalidation_0-rmse:0.64099\n",
            "[74]\tvalidation_0-rmse:0.64006\n",
            "[75]\tvalidation_0-rmse:0.639639\n",
            "[76]\tvalidation_0-rmse:0.638205\n",
            "[77]\tvalidation_0-rmse:0.637453\n",
            "[78]\tvalidation_0-rmse:0.636822\n",
            "[79]\tvalidation_0-rmse:0.635136\n",
            "[80]\tvalidation_0-rmse:0.634576\n",
            "[81]\tvalidation_0-rmse:0.633622\n",
            "[82]\tvalidation_0-rmse:0.633258\n",
            "[83]\tvalidation_0-rmse:0.631846\n",
            "[84]\tvalidation_0-rmse:0.631209\n",
            "[85]\tvalidation_0-rmse:0.63025\n",
            "[86]\tvalidation_0-rmse:0.629237\n",
            "[87]\tvalidation_0-rmse:0.627656\n",
            "[88]\tvalidation_0-rmse:0.627522\n",
            "[89]\tvalidation_0-rmse:0.627112\n",
            "[90]\tvalidation_0-rmse:0.625914\n",
            "[91]\tvalidation_0-rmse:0.625933\n",
            "[92]\tvalidation_0-rmse:0.625666\n",
            "[93]\tvalidation_0-rmse:0.625118\n",
            "[94]\tvalidation_0-rmse:0.624764\n",
            "[95]\tvalidation_0-rmse:0.624705\n",
            "[96]\tvalidation_0-rmse:0.624332\n",
            "[97]\tvalidation_0-rmse:0.623676\n",
            "[98]\tvalidation_0-rmse:0.623723\n",
            "[99]\tvalidation_0-rmse:0.623329\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "XGBRegressor(objective=<CompiledFunction of functools.partial(<function jax_autodiff_grad_hess at 0x7fbe8ef99170>, <function jax_sle_loss at 0x7fbe8ef48440>)>)"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    }
  ]
}