Part 4 - Callbacks.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import pandas as pd\n",
    "from functools import partial\n",
    "from pathlib import Path\n",
    "import torchvision\n",
    "\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import re\n",
    "\n",
    "\n",
    "from scripts.dataloader import Dataset, Transforms, Resize, ToTorch, Sampler, collate, DataLoader\n",
    "from scripts.custom_models import Lambda, flatten, MixedInputModel, TabularModel, CNNModel, CustomResnet\n",
    "from scripts.optimizer_and_loss import Optimizer, MSE_Loss, get_example_dataloaders, get_example_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 0 - Imports from previous lessons\n",
    "\n",
    "The main things that we need from the previous lessons are the dattaloaders, the model, the loss function and the optimizer.  We don't need the training function because we will be rewriting it to include callbacks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dl_train, dl_valid = get_example_dataloaders()\n",
    "model = get_example_model(dl_train)\n",
    "loss_function = MSE_Loss(reduce = True)\n",
    "opt = Optimizer(model, 0.02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1 - Introduction to the Callback System\n",
    "\n",
    "A callback is a supporting function, or group of functions, that can be called at various preset points within the code.  This allows the user to customize the functionality of the code without modifying the core components directly.  For instance, after each epoch of training, we may wish to change the learning rate.  This could be achieved with a callback that gets triggered before (or after) every completed epoch of training.  Alternatively, we might also want a callback that can break the training cycle if the validation loss starts to rise (early stopping).  Callbacks may even provide core functionality, such as the ability to use assess custom metrics during validation (e.g. an accuracy score)\n",
    "\n",
    "We can therefore create a list of desired features from our callbacks:\n",
    "- The ability to be called at specific points in the training cycle\n",
    "- The ability to mix and match different types of callbacks\n",
    "- The ability to change core parameters related to training\n",
    "- The ability to store parameters\n",
    "- The ability to stop training\n",
    "- A callback system that is easy to program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to call callbacks at specific landmarks in the code is to assign each of these landmarks a name.  For instance, we can call the point right after we calculate the loss \"after loss\", or the point right after we end a single batch \"after batch\".  Each of our callbacks should have functions related to these points, but we also don't want to have to program each function for every callback.  This is especially true when you just want the callback to access a single point (e.g. decreasing the learning rate by 10% after every batch).  We can get around this issue by creating a callback class which accepts a string when called (e.g. 'after batch') and only performs the operation if it has that function  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Callback():\n",
    "\n",
    "    _order = 0 #This allows you to control the order callbacks are called in\n",
    "    def __init__(self): self.learn = None\n",
    "    def set_learner(self,learn): self.learn = learn\n",
    "    def __getattr__(self, attr): return getattr(self.learn, attr)  #Will look here if it can't find it in the current class\n",
    "    \n",
    "    def __call__(self, callback):\n",
    "        f = getattr(self, callback, None)\n",
    "        if f and f(): return True   #f() won't be called if f is None\n",
    "        else: return False\n",
    "        \n",
    "    @property\n",
    "    def name(self):\n",
    "        name = re.sub(r'Callback$', '', self.__class__.__name__)\n",
    "        return camel2snake(name or 'callback')\n",
    "    \n",
    "    \n",
    "        \n",
    "_camel_re1 = re.compile('(.)([A-Z][a-z]+)')\n",
    "_camel_re2 = re.compile('([a-z0-9])([A-Z])')\n",
    "def camel2snake(name):\n",
    "    s1 = re.sub(_camel_re1, r'\\1_\\2', name)\n",
    "    return re.sub(_camel_re2, r'\\1_\\2', s1).lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all there is too it.  For instance, we can create a simple callback to test out the functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "It works\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class TestCallback(Callback):\n",
    "    def example_callback(self):\n",
    "        print (\"It works\")\n",
    "        \n",
    "class TestCallback2(Callback):\n",
    "    def begin_batch(self):\n",
    "        print(\"Starting the batch!\")\n",
    "    \n",
    "    \n",
    "test_callback = TestCallback()\n",
    "#test_callback.set_learner(None)\n",
    "\n",
    "test_callback(\"example_callback\")\n",
    "test_callback(\"unimplemented_callback\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that when the callback has the function it is called, but when it doesn't it returns False.  We will later incorporate the callbacks into a larger \"learn\" class, but just ignore the `set_learner` and `__getattr__ ` for now.\n",
    "\n",
    "We can create a callback manager class to handle all of our callbacks.  Its main purpose will be to go through each of the registered callbacks whenever they're by the the training cycles.  We will accept two forms of callbacks as inputs.  The first is the initalized callback instances (e.g. test_callback from above).  The other is the uninitialized class (e.g. TestCallback), which must be called by the callback mananger to be initialized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CallbackMananger():\n",
    "    def __init__(self, callbacks, callback_functions):\n",
    "        self.callbacks = []\n",
    "        self.add_callbacks(callbacks)\n",
    "        self.add_callbacks([cbf() for cbf in callback_functions])\n",
    "        \n",
    "    def add_callbacks(self, cbs):\n",
    "        for cb in cbs: \n",
    "            cb.set_learner(None)\n",
    "            self.callbacks.append(cb)\n",
    "            \n",
    "    def __call__(self, callback_name):\n",
    "        result = False\n",
    "        for cb in sorted(self.callbacks, key = lambda x: x._order): result = cb(callback_name) and result\n",
    "        return result\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "cbm = CallbackMananger(callbacks = [TestCallback()], \n",
    "                       callback_functions = [TestCallback2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a simplified version of our training cycle.  Note, we'll be able to clean up aspects like move_data_to_GPU in just a bit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def move_data_to_GPU(xb, yb):\n",
    "    xb = tuple(map(lambda x: x.cuda(), xb))\n",
    "    yb = yb.cuda()\n",
    "    return xb, yb\n",
    "\n",
    "model.cuda()\n",
    "training = True\n",
    "\n",
    "xb, yb = next(iter(dl_train))\n",
    "xb, yb = move_data_to_GPU(xb, yb)\n",
    "def one_batch(xb, yb):\n",
    "    cbm('begin_batch')\n",
    "    preds = model(xb)\n",
    "    cbm('after_pred')\n",
    "    loss = loss_function(preds, yb)\n",
    "    cbm('after_loss')\n",
    "    if training:\n",
    "        loss.backward()\n",
    "        cbm('after_loss')\n",
    "        opt.step()\n",
    "        cbm('after_step')\n",
    "        opt.zero_grad()\n",
    "        cbm('after_batch')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting the batch!\n"
     ]
    }
   ],
   "source": [
    "one_batch(xb, yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next feature we want to add in is the ability to cancel the batch early.  We can do this using exceptions.  Let's add a callback that stops the batch after the loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "class StopAfterLoss(Callback):\n",
    "    def __init__(self):\n",
    "        self.temp_variable = None #To show that callbacks can store variables or be initialized\n",
    "        \n",
    "    def after_loss(self):\n",
    "        print('About to cancel the batch')\n",
    "        raise CancelBatchException(\"Cancelling the training cycle\")\n",
    "        \n",
    "    def after_cancel_batch(self):\n",
    "        print('Cancelled the batch!')\n",
    "        \n",
    "        \n",
    "class CancelBatchException(Exception): pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_batch(xb, yb):\n",
    "    try:\n",
    "        cbm('begin_batch')\n",
    "        preds = model(xb);                 cbm('after_pred')\n",
    "        loss = loss_function(preds, yb);   cbm('after_loss')\n",
    "        if training:\n",
    "            loss.backward();               cbm('after_loss')\n",
    "            opt.step();                    cbm('after_step')\n",
    "            opt.zero_grad()\n",
    "            \n",
    "    except CancelBatchException:           cbm('after_cancel_batch')\n",
    "        \n",
    "    finally: cbm('after_batch')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting the batch!\n",
      "About to cancel the batch\n",
      "Cancelled the batch!\n"
     ]
    }
   ],
   "source": [
    "cbm = CallbackMananger(callbacks = [TestCallback()], \n",
    "                       callback_functions = [TestCallback2, StopAfterLoss])\n",
    "\n",
    "one_batch(xb, yb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2 - The Learner Class\n",
    "\n",
    "This is all we need for the foundations of a callback system.  We've addressed all of the requirements except for the ability to change core parameters associated with training.  For that, we want a larger structure than the callback mananger.  One that completely controls the entire training process.  To that end, we will create a Learner class, which will do double duty as the callback manager.  We will also create a `DataBlock` class that will contain our dataloaders (rather than have multiple possibilities in terms of the training, validation, and test sets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Datablock():\n",
    "    def __init__(self, train_dl, valid_dl = None, test_dl = None):\n",
    "        self.train_dl, self.valid_dl, self.test_dl = train_dl, valid_dl, test_dl\n",
    "        \n",
    "\n",
    "\n",
    "class PreLearner():\n",
    "    def __init__(self, model, data, loss_func, opt, cbs, cbfs):\n",
    "        self.model = model\n",
    "        self.loss_func = loss_func\n",
    "        self.opt = opt\n",
    "        self.data = data\n",
    "        \n",
    "        self.callbacks = []\n",
    "        self.add_callbacks(callbacks)\n",
    "        self.add_callbacks([cbf() for cbf in callback_functions])\n",
    "        \n",
    "    def add_callbacks(self, cbs):\n",
    "        for cb in cbs: \n",
    "            cb.set_learner(self)\n",
    "            self.callbacks.append(cb)\n",
    "            \n",
    "    def one_batch(self, xb, yb):\n",
    "        try:\n",
    "            self.xb, self.yb = xb, yb;                          self('begin_batch')\n",
    "            self.preds = self.model(self.xb);                   self('after_pred')\n",
    "            self.loss = self.loss_func(self.preds, self.yb);    self('after_loss')\n",
    "            if training:\n",
    "                self.loss.backward();                           self('after_loss')\n",
    "                self.opt.step();                                self('after_step')\n",
    "                self.opt.zero_grad()\n",
    "\n",
    "        except CancelBatchException:                            self('after_cancel_batch')\n",
    "\n",
    "        finally:                                                self('after_batch')\n",
    "            \n",
    "            \n",
    "    def __call__(self, callback_name):\n",
    "        result = False\n",
    "        for cb in sorted(self.callbacks, key = lambda x: x._order): result = cb(callback_name) and result\n",
    "        return result\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test out the ability to access and write data to the learner using a new callback:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PredictionDoubler(Callback):\n",
    "    def after_pred(self):\n",
    "        pred = self.preds #This will call the __getattr__ class method when it doesn't find self.pred\n",
    "        self.learn.test_output = \"If you see this, we've set a value to the learner\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "callbacks = [TestCallback()]\n",
    "callback_functions = [TestCallback2, StopAfterLoss, PredictionDoubler]\n",
    "learn = PreLearner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting the batch!\n",
      "About to cancel the batch\n",
      "Cancelled the batch!\n",
      "If you see this, we've set a value to the learner\n"
     ]
    }
   ],
   "source": [
    "learn.one_batch(xb,yb)\n",
    "\n",
    "print(learn.test_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have something that is working and fulfils all of the requirements.  There is more to the training cycle than the ability to perform one batch.  Similarly, we can start to push some of the features to the callbacks themselves (e.g. moving the data to the GPU).  The main things that we need to add to our final learner are:\n",
    "- The ability to switch between training and validation mode\n",
    "- The ability to iterate through an entire epoch\n",
    "- The ability to iterate over multiple epochs\n",
    "- The ability to check that any callbacks are in the defined list\n",
    "\n",
    "In terms of callbacks, we want to add the following features:\n",
    "- The ability to move the batch data to the GPU\n",
    "- The ability to keep track of the loss over time\n",
    "- The ability to add another validation metric\n",
    "- The ability to stop the training after 3 batches and 2 epochs (so that we can test more easily)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class Learner():\n",
    "    def __init__(self, model, data, loss_func, opt, cbs, cbfs):\n",
    "        self.model = model\n",
    "        self.loss_func = loss_func\n",
    "        self.opt = opt\n",
    "        self.data = data\n",
    "        \n",
    "        self.callbacks = []\n",
    "        self.add_callbacks([TrainEvalCallback()] + cbs)\n",
    "        self.add_callbacks([cbf() for cbf in cbfs])\n",
    "        \n",
    "    def add_callbacks(self, cbs):\n",
    "        for cb in cbs: \n",
    "            cb.set_learner(self)\n",
    "            setattr(self, cb.name, cb)\n",
    "            self.callbacks.append(cb)\n",
    "          \n",
    "    def fit(self, num_epochs):\n",
    "        self.num_epochs = num_epochs\n",
    "        try: \n",
    "            self.loss = torch.tensor(0.);                       self('begin_fit')\n",
    "            for epoch in range(num_epochs):\n",
    "                self.epoch = epoch\n",
    "                if not self('begin_epoch'): #Will return False unless you want to skip the epoch\n",
    "                    #train cycle\n",
    "                    #self.in_train = True\n",
    "                    #self.model.train()\n",
    "                    self.dl = self.data.train_dl\n",
    "                    self.all_batches()\n",
    "                \n",
    "                    #valid cycle\n",
    "                    if not self('begin_validate'):  #Will return False unless you want to skip validation\n",
    "                        #self.in_train = False\n",
    "                        #self.model.eval()\n",
    "                        self.dl = self.data.valid_dl\n",
    "                        with torch.no_grad(): self.all_batches() # May have to move torch no grad up\n",
    "                self('after_epoch')\n",
    "            \n",
    "        except CancelTrainException:                            self('after_cancel_train')\n",
    "        finally:                                                self('after_fit')\n",
    "        \n",
    "    def all_batches(self):\n",
    "        try:\n",
    "            for i, (xb, yb) in enumerate(self.dl):\n",
    "                self.one_batch(i, xb, yb)\n",
    "        except CancelEpochException: self('after_cancel_epoch')\n",
    "\n",
    "    def one_batch(self, i, xb, yb):\n",
    "        try:\n",
    "            self.iter = i\n",
    "            self.xb, self.yb = xb, yb;                          self('begin_batch')\n",
    "            self.preds = self.model(self.xb);                   self('after_pred')\n",
    "            self.loss = self.loss_func(self.preds, self.yb);    self('after_loss')\n",
    "            if self.in_train:\n",
    "                self.loss.backward();                           self('after_loss')\n",
    "                self.opt.step();                                self('after_step')\n",
    "                self.opt.zero_grad()\n",
    "\n",
    "        except CancelBatchException:                            self('after_cancel_batch')\n",
    "\n",
    "        finally:                                                self('after_batch')\n",
    "           \n",
    "    ALL_CALLBACKS = {'begin_batch', 'after_pred', 'after_loss', 'after_backward', 'after_step',\n",
    "        'after_cancel_batch', 'after_batch', 'after_cancel_epoch', 'begin_fit',\n",
    "        'begin_epoch', 'begin_validate', 'after_epoch',\n",
    "        'after_cancel_train', 'after_fit'}\n",
    "        \n",
    "    def __call__(self, callback_name):\n",
    "        result = False\n",
    "        assert callback_name in self.ALL_CALLBACKS\n",
    "        for cb in sorted(self.callbacks, key = lambda x: x._order): result = cb(callback_name) and result\n",
    "        return result\n",
    "        \n",
    "\n",
    "        \n",
    "        \n",
    "class TrainEvalCallback(Callback):\n",
    "    def begin_fit(self):\n",
    "        self.learn.n_epochs = 0\n",
    "        self.learn.n_iter = 0\n",
    "        \n",
    "    def begin_epoch(self):\n",
    "        self.learn.n_epochs = self.epoch\n",
    "        self.model.train()\n",
    "        self.learn.in_train = True\n",
    "        \n",
    "    def begin_validate(self):\n",
    "        self.model.eval()\n",
    "        self.learn.in_train = False\n",
    "        \n",
    "        \n",
    "        \n",
    "        \n",
    "class CancelTrainException(Exception): pass\n",
    "class CancelEpochException(Exception): pass\n",
    "class CancelBatchException(Exception): pass\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class MoveBatchGPUCallback(Callback):\n",
    "    _order = 100 #We want this to be done last, potentially (some operations may be requied before or after it)\n",
    "    def begin_batch(self):\n",
    "        self.learn.yb = self.yb.cuda()\n",
    "        self.learn.xb = tuple(map(lambda x: x.cuda(), xb))\n",
    "    def begin_fit(self):\n",
    "        self.model.cuda()\n",
    "        \n",
    "class EarlyStoppingCallback(Callback):\n",
    "    def __init__(self, max_epochs, max_batches):\n",
    "        self.max_epochs = max_epochs\n",
    "        self.max_batches = max_batches\n",
    "    def after_epoch(self):\n",
    "        if self.epoch >= self.max_epochs: raise CancelTrainException()  \n",
    "    def after_batch(self):\n",
    "        if self.iter >= self.max_batches: raise CancelEpochException()\n",
    "            \n",
    "class OutputStepsCallback(Callback):\n",
    "    def begin_batch(self): print(f\"   Batch {self.iter}\")\n",
    "    def begin_epoch(self): print(F\"Epoch {self.epoch}\", \"\\nBeginning the Training\")\n",
    "    def begin_validate(self): print(\"Beginning the validation\")\n",
    "    \n",
    "class SqueezePredsCallback(Callback):\n",
    "    def after_pred(self):\n",
    "        self.learn.preds = torch.squeeze(self.learn.preds, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "callbacks = [OutputStepsCallback(), EarlyStoppingCallback(2, 3)]\n",
    "callback_functions = [MoveBatchGPUCallback, SqueezePredsCallback]\n",
    "learn = Learner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0 \n",
      "Beginning the Training\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n",
      "Beginning the validation\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n",
      "Epoch 1 \n",
      "Beginning the Training\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n",
      "Beginning the validation\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n",
      "Epoch 2 \n",
      "Beginning the Training\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n",
      "Beginning the validation\n",
      "   Batch 0\n",
      "   Batch 1\n",
      "   Batch 2\n",
      "   Batch 3\n"
     ]
    }
   ],
   "source": [
    "learn.fit(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've shown some simple callbacks that handle basic actions, such as early stopping and outputting the current steps so that we can track progress during training.  The good thing about using callbacks to perform these tasks is that we can easily mix and match at runtime.  It also increases the flexibility of the system.  For instance, we need a vey specific unpacking and repacking process when moving the xb to the GPU, which is made simple by having this behaviour in its own callback.  There are other steps that could have been pushed to a callback as well, such as identifying the `model.eval()` and `model.train()` steps.  What you choose to push to callbacks and what you choose to incorporate into the code is up to you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3 - More Sophisticated Callbacks\n",
    "\n",
    "In addition to the simple callbacks we saw in the previous section, we can also look at more complex callbacks that interact with our learner in more complicated ways.  We will go over two of the most critical ones here:\n",
    "- The ability to track loss and other specific metrics\n",
    "- The ability to schedule specific parameters like the learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stats and Metrics\n",
    "\n",
    "The ability to track specific stats is a very useful tool when training your models.  Although the loss function is what the system uses for training the parameters, a metric such as accuracy may provide a more tangible representation of the performance of the model.  Alternatively, you may wish to extract very specific information that is not easily parsed from a single loss value.  For instance, you may want to separate the predictions based on each class and give a summary of each.  This is easily calculated using the preds and yb components stored in the learner.\n",
    "\n",
    "To calculate stats, we will create a new 'AvgStats' class that will accumulate the metrics of interest across all the batches of an epoch and then append them to a amoving tally.  This class should have some key functionalities, including:\n",
    "- The ability to reset the statistics easily\n",
    "- The ability to average in batches of varying sizes (since the last batch sometimes has fewer elements - not a huge deal)\n",
    "- The ability to accept a list of custom \"metric functions\"\n",
    "\n",
    "To keep things simple, each metric will take two inputs: the predictions (output) and the yb (target).  This way, we can just call metric(self.preds, self.yb) to calculate the output of each metric\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class AvgStats():\n",
    "    def __init__(self, metrics, in_train):\n",
    "        self.metrics, self.in_train = metrics, in_train\n",
    "        self.reset_stats()\n",
    "        \n",
    "    def reset_stats(self):\n",
    "        self.tot_loss, self.count = 0., 0\n",
    "        self.tot_metrics = [0.]*len(self.metrics)\n",
    "        \n",
    "    @property\n",
    "    def all_stats(self): return [self.tot_loss.item()] + self.tot_metrics\n",
    "    \n",
    "    @property\n",
    "    def avg_stats(self): return [o/self.count for o in self.all_stats]\n",
    "        \n",
    "    def accumulate(self, learn):\n",
    "        batch_size = learn.preds.shape[0]\n",
    "        self.tot_loss += learn.loss * batch_size\n",
    "        self.count += batch_size\n",
    "        for i, m in enumerate(self.metrics):\n",
    "            self.tot_metrics[i] += m(learn.preds, learn.yb)*batch_size\n",
    "        \n",
    "    \n",
    "    def __repr__(self):\n",
    "        if not self.count: return \"\"\n",
    "        return f\"{'train' if self.in_train else 'valid'}: {self.avg_stats}\"\n",
    "    \n",
    "    \n",
    "class AvgStatsCallback(Callback):\n",
    "    def __init__(self, train_metrics, valid_metrics):\n",
    "        self.train_stats, self.valid_stats = AvgStats(train_metrics, True), AvgStats(valid_metrics, False)\n",
    "        \n",
    "    def begin_fit(self):\n",
    "        self.train_stats.reset_stats()\n",
    "        self.valid_stats.reset_stats()\n",
    "        \n",
    "    def after_batch(self):\n",
    "        current_stats = self.train_stats if self.in_train else self.valid_stats\n",
    "        with torch.no_grad(): current_stats.accumulate(self.learn)\n",
    "            \n",
    "    def after_epoch(self):\n",
    "        print(self.train_stats)\n",
    "        print(self.valid_stats)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: [nan]\n",
      "valid: [nan]\n",
      "train: [nan]\n",
      "valid: [nan]\n"
     ]
    }
   ],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "metrics = []\n",
    "\n",
    "\n",
    "callbacks = [#OutputStepsCallback(), \n",
    "             EarlyStoppingCallback(2, 3), \n",
    "             AvgStatsCallback(metrics, metrics)]\n",
    "callback_functions = [MoveBatchGPUCallback]\n",
    "learn = Learner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)\n",
    "\n",
    "learn.fit(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the base for our statistics, lets add some random metrics to show some examples of what is possible:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0625\n",
      "0.053211890161037445\n",
      "1.6072728633880615\n"
     ]
    }
   ],
   "source": [
    "def percent_close(output, target, tolerance = 0.1):\n",
    "    within_tolerance = (abs(output-target))<tolerance\n",
    "    return (within_tolerance*1.).mean().item()\n",
    "\n",
    "def msle(output, target):\n",
    "    logs = torch.log(output+1) - torch.log(target+1)\n",
    "    return (logs**2).mean().item()\n",
    "\n",
    "def mean_pred(output, target):\n",
    "    return output.mean().item()\n",
    "\n",
    "#random function\n",
    "\n",
    "\n",
    "preds, yb = learn.preds, learn.yb\n",
    "preds = yb+torch.rand(yb.shape).cuda() #simulate another preds function\n",
    "\n",
    "for func in [percent_close, msle, mean_pred]:\n",
    "    print(func(preds, yb))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n"
     ]
    }
   ],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "metrics = [percent_close, msle, mean_pred]\n",
    "\n",
    "\n",
    "callbacks = [#OutputStepsCallback(), \n",
    "             EarlyStoppingCallback(2, 3), \n",
    "             AvgStatsCallback(metrics, metrics)]\n",
    "callback_functions = [MoveBatchGPUCallback]\n",
    "learn = Learner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)\n",
    "\n",
    "learn.fit(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameter Scheduler\n",
    "\n",
    "Another useful callback is the parameter scheduler, which allows us to modify specific parameters throughout the training process.  For instance, it is common to gradually decrease the learning rate as training progresses, allowing you to hone in on a local minimum.  Before we get started on the callback, we need a method of accepting an input (pos, or fraction of the training) and outputing our parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def annealer(f):\n",
    "    def _inner(start, stop):\n",
    "        return partial(f, start, stop)\n",
    "    return _inner\n",
    "\n",
    "@annealer\n",
    "def sched_lin(start, stop, pos): return start + (stop-start)*pos\n",
    "@annealer\n",
    "def sched_cos(start, stop, pos): return start + (1 + math.cos(math.pi*(1-pos))) * (stop-start) / 2\n",
    "@annealer\n",
    "def sched_none(start, stop, pos): return start\n",
    "@annealer\n",
    "def sched_exp(start, stop, pos): return (start)*(stop/start)**pos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at the cosine scheduler with a start of 2 and an end of 0.4,  we can see that our scheduler behaves as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x206f735d248>]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV9Z3/8dcnOwkJCSSEQAgBZVWQ5UIAi8tYR9RW1GqVVZRFtLa17Uzbmc50+mtn7Mx0umhFZRUXFnGpMlaxtWpBSICwyK6yE7YEAmEJW8j398eNDsWEXMxNzl3ez8cjD3Nzvtz7Pia8Ofmec8/XnHOIiEj4i/E6gIiIBIcKXUQkQqjQRUQihApdRCRCqNBFRCJEnFcvnJmZ6fLz8716eRGRsLRy5cqDzrms2rZ5Vuj5+fkUFxd79fIiImHJzHbWtU1TLiIiEUKFLiISIVToIiIRQoUuIhIhVOgiIhGi3kI3s/Zm9r6ZbTKzDWb23VrGmJk9YWZbzGytmfVtnLgiIlKXQC5brAJ+4JxbZWapwEoz+7NzbuN5Y24GOtd8FABP1/xXRESaSL2F7pzbB+yr+fyYmW0C2gHnF/ow4HnnvxdvkZmlm1lOzZ8Nqo/3H+OPa/cSGxNDXKwRF2MkJ8aRkhBLSmIcGckJtExJoFVKAunJ8ZhZsCOIiISkS3pjkZnlA32AZRdsagfsPu9xSc3X/qbQzWwiMBEgLy/v0pLW2FJ6nCfe2xLQ2IS4GHJaJJHTIokOLVPIz0yhY2YKnbObk98qhdgYlb2IRI6AC93MmgOvAo86545euLmWP/KFlTOcc1OBqQA+n+9Lraxxa68cbu11K9XVjrPV1Zw956g8U0Xl6XMcP13F4cozlJ84w8HjZzhw9BR7j5xk75GT/GXzAQ4eP/P58yTGxdA5uzlXtm1Br9x0rmrfgq7ZqcTF6jyxiISngArdzOLxl/ls59xrtQwpAdqf9zgX2NvweHWLiTESY2JJjIPmiXGQWv+fOXrqLDsOnuCTA8fZvO8om/cfY+GG/cxb4f/lIiUhlr4dMuif35KBnVrRJy+deBW8iISJegvd/JPQM4BNzrnf1DFsAfCImc3DfzK0ojHmzxsqLSmeXrnp9MpN//xrzjl2lVeyZvcRVu08zLLt5fz23U9wzv8PxcBOrbi2axY3dGtN2/RmHqYXEbk4q29NUTP7CrAYWAdU13z5n4E8AOfcMzWl/yQwFKgE7nfOXfTOWz6fz4XqzbmOVJ6haNshFn96kEWflrG7/CQAPXLSuLFHNrf2yqFLdgC/EoiIBJmZrXTO+Wrd5tUi0aFc6OdzzrG17ATvbjrAuxsPsHLXYZyDy1s352u9cri9dzvyM1O8jikiUUKFHkSlR0+xcMN+3ly7jxU7ynEO+ualc2ffXG7r3Za0pHivI4pIBFOhN5J9FSd5ffVe/rC6hE8OHCcpPoZbe7ZlREF7+uZl6Bp4EQk6FXojc86xbk8Fc5fvZsGaPZw4c44eOWncN7gDw3q3Iyk+1uuIIhIhVOhN6MTpKl5fs4cXCneyef8x0pPjGVXQgTGDO9A6NcnreCIS5lToHnDOsXx7OTOXbOdPGw8QHxPDHX3a8eC1neiU1dzreCISpi5W6J6tKRrpzIyCTq0o6NSK7QdPMOPDbbxcXML8lbu5tWcO37r+crrnpHkdU0QiiI7Qm1DZsdPM+HA7Lxbt5PjpKm66Ipvv3diFbm1U7CISGE25hJiKyrPMXLKdmR9u5/iZKm7tmcP3b+yiqRgRqZcKPUQdqTzDtMXbeHbJDk5XVXNP//Y8ekNnWqfp5KmI1E6FHuIOHj/Nk+9tYfayncTGGBOGdGLStZeRkqhTHCLyt1ToYWLXoUp+9aeP+d+P9tI6NZF/uKkrd/XNJUb3bReRGhcrdN0bNoTktUrm98P78OpDg2mX0YwfvrKWYZOXsGrXYa+jiUgYUKGHoH4dMnjtocE8fm9vSo+d4s6nlvKD+R9Rduy019FEJISp0EOUmTGsdzve+8F1PHTdZSz4aA9/9+sPeKFoJ9XV3kyTiUhoU6GHuJTEOH40tBsLH72Gnu1a8K+vr+fOp5eyYW+F19FEJMSo0MPEZVnNmT2+gN/d05uSw5Xc9uQS/mvhZk6dPed1NBEJEfUWupnNNLNSM1tfx/YWZva/ZvaRmW0ws/uDH1PAPw1ze592/OX71/GNvu14+oOt3Pz4Yoq2HfI6moiEgECO0GfhX1quLt8CNjrnrgKuA35tZgkNjyZ1aZEcz3/fdRWzxxdwrtpx79Qi/u2N9VSeqfI6moh4qN5Cd84tAsovNgRIrVlXtHnNWDVLE7j68kwWPjqEsYPzea5wJ0N/t5hlOloXiVrBmEN/EugO7MW/kPR3nXPVtQ00s4lmVmxmxWVlZUF4aUlOiONnt13BSxMHAnDvtCL+448bNbcuEoWCUeg3AWuAtkBv4Ekzq/X2gc65qc45n3POl5WVFYSXls8UdGrFwkeHMLIgj2mLtzPsySW6EkYkygSj0O8HXnN+W4DtQLcgPK9couSEOP799p48e39/yivPcPvkJUxbtE3XrYtEiWAU+i7gBgAzywa6AtuC8LzyJV3ftTV/evQaru/amv94axP3Pbuc0qOnvI4lIo0skMsW5wKFQFczKzGzcWY2ycwm1Qz5BTDYzNYBfwF+5Jw72HiRJRAZKQlMGd2Px+7oyYod5Qx9fDHvby71OpaINCLdbTEKbCk9xiNzVrN5/zEmDOnIP97UjYQ4vadMJBzpbotR7vLWqbz+rasZNdB/wvTuKYXsLq/0OpaIBJkKPUokxcfy77f35OmRfdlWepyv/f5D/rLpgNexRCSIVOhR5uaeObz5na+Qm9GMcc8V859vb6bqXK1vGxCRMKNCj0IdWqXw6kODGVGQxzN/3cqoGcs4eFz3WhcJdyr0KJUUH8tjd/Tk13dfxepdR/jaEx9qZSSRMKdCj3Lf6JfLaw8PJj7OuGdKIbOX7fQ6koh8SSp04Yq2LXjzkSEMviyTn/xhPf/02lpOV+leMCLhRoUugP+WvDPH9ufh6y5j7vLdDJ9apHeXioQZFbp8LjbG+OHQbjw1si+b9x/j609+yNqSI17HEpEAqdDlC27pmcMrkwYTFxPD3c8U8saaPV5HEpEAqNClVj3aprHgkau5qn063523hl+9s1l3bRQJcSp0qVOr5om8OK6A4QPaM/n9rXxrzipOntHJUpFQpUKXi0qIi+GxO3ryL7d2Z+GG/XxzSiH7K3SyVCQUqdClXmbG+CGdmHGfj21lx7l98hI27j3qdSwRuYAKXQL2d92yeXnSYMzg7meW6v7qIiFGhS6XpEfbNF7/1tXkZ6Yw7rkVvFC4w+tIIlIjkBWLZppZqZmtv8iY68xsjZltMLO/BjeihJrstCTmPziI67u25l/f2MAv39qkK2BEQkAgR+izgKF1bTSzdOAp4Dbn3BXA3cGJJqEsJTGOKaP7MWpgHlMWbeM781Zz6qyugBHxUlx9A5xzi8ws/yJDRgCvOed21YzXxGqUiIuN4RfDrqR9RjK/fHszpUdPM22MjxbJ8V5HE4lKwZhD7wJkmNkHZrbSzMbUNdDMJppZsZkVl5WVBeGlxWtmxoPXXsYTw/uwZvcR7npmKXuPnPQ6lkhUCkahxwH9gFuBm4B/NbMutQ10zk11zvmcc76srKwgvLSEituuasusB/qzv+IUdz61lM37dVmjSFMLRqGXAAudcyeccweBRcBVQXheCTODL8tk/qRBOBx3P1PI8u3lXkcSiSrBKPQ3gCFmFmdmyUABsCkIzythqHtOGq8+NJis1ERGzVjGOxv2ex1JJGoEctniXKAQ6GpmJWY2zswmmdkkAOfcJmAhsBZYDkx3ztV5iaNEvtyMZF6ZNJgeOWk89OJK5i3f5XUkkahgznlz/bDP53PFxcWevLY0jcozVTw8exUffFzGj4Z2Y9K1nTAzr2OJhDUzW+mc89W2Te8UlUaTnBDHtDE+hvVuy38t3Mxjb23CqwMIkWhQ73XoIg0RHxvDb7/Zm/Rm8UxbvJ3DlWf5zzt7EherYwmRYFOhS6OLiTF+dtsVZKQk8Lt3P+XYqbM8MbwPiXGxXkcTiSg6TJImYWY8+tUu/NvXe/DOhgOMm1XMidNVXscSiSgqdGlS91/dkV/ffRWF2w4xcvoyKirPeh1JJGKo0KXJfaNfLk+N7MvGvUe5Z2ohZcdOex1JJCKo0MUTN13Rhun3+dhx6AT3TCnU/V9EgkCFLp65pksWL4wroOzYae5+ppCdh054HUkkrKnQxVP981syZ8JATpyp4ptTCtlSetzrSCJhS4UunuuZ24KXJg7iXDXcM6VQC1CLfEkqdAkJXdukMv/BgSTExTB8WhEf7T7idSSRsKNCl5DRKas58x8cRFqzOEZNX8bKnbr9rsilUKFLSGnfMpn5Dw4iMzWR0TOWU7j1kNeRRMKGCl1CTk6LZrw0cSDt0ptx/6zlLP5UyxWKBEKFLiGpdVoS8yYOJL9VCuOeK+b9j7X2uEh9AlngYqaZlZrZRRetMLP+ZnbOzO4KXjyJZq2aJzJ3wkA6t27Og8+v5N2NB7yOJBLSAjlCnwUMvdgAM4sF/gt4JwiZRD6XkZLAnPED6Z6TykOzV7JwvZa0E6lLvYXunFsE1He5wbeBVwH9XixB1yI5nhfGF3BluxY8MmcVb6/b53UkkZDU4Dl0M2sH3AE8E8DYiWZWbGbFZWU60SWBS0uK5/kHBnBV+3QembuaP65VqYtcKBgnRX8H/Mg5d66+gc65qc45n3POl5WVFYSXlmiSmhTPcw8MoG9eOt+Zt5r//Wiv15FEQkowCt0HzDOzHcBdwFNmdnsQnlfkC5onxjHr/gH065DBd+etZoFKXeRzDS5051xH51y+cy4feAV42Dn3eoOTidQhJTGOZ8f2p39+Sx6dt5o31uzxOpJISAjkssW5QCHQ1cxKzGycmU0ys0mNH0+kdimJcTx7f38GdGzJ915ao1IXIYBFop1zwwN9Mufc2AalEbkEyQlxzBzbnwdmreB7L60BYFjvdh6nEvGO3ikqYe2zUteRuogKXSKASl3ET4UuEeGzUu+f7y91XdIo0UiFLhEjOcF/otSX35JHX1rDm2tV6hJdVOgSUZIT/Jc09svL4Lvz1vCWbhMgUUSFLhHns0sa+7RP5ztzV7NwvUpdooMKXSJSSmIcsx4YQK/cFjwyZzXvbNBdGiXyqdAlYjVPjOO5BwZ8fpfGP+t+6hLhVOgS0VKT4nl+3AB65KTx8OyVvLdZpS6RS4UuEc9/690CurZJZdILq/jrJ7p1s0QmFbpEhRbJ8bw4roDLWzdnwvPFfPjpQa8jiQSdCl2iRnpyAi+OL6BTZgrjn1/B0q0qdYksKnSJKi1TEpg9voC8lsmMm1XMsm2HvI4kEjQqdIk6rZonMnv8QNqmJ3H/rBUU76hvyVyR8KBCl6iUlZrI3AkDaZOWxNhnV7Bq12GvI4k0mApdolbrtCTmTBhIq+YJ3DdjOWtLjngdSaRBAlmxaKaZlZrZ+jq2jzSztTUfS83squDHFGkcbVr4S71Fcjyjpi9j/Z4KryOJfGmBHKHPAoZeZPt24FrnXC/gF8DUIOQSaTLt0psxd8JAUpPiGT1jGZv3H/U6ksiXUm+hO+cWAXWeNXLOLXXOfTYBWQTkBimbSJNp3zKZuRMGkhQfy8hpy/j0wDGvI4lcsmDPoY8D3q5ro5lNNLNiMysuK9O79SS05LVKZs6EgcTGGMOnLWNL6XGvI4lckqAVupldj7/Qf1TXGOfcVOeczznny8rKCtZLiwRNx8wU5kwYCMCIaUVsP3jC40QigQtKoZtZL2A6MMw5p3dqSFi7vHVz5kwo4Fy1Y/jUInYeUqlLeGhwoZtZHvAaMNo590nDI4l4r0t2Ki+OL+BU1TlGTFvG7vJKryOJ1CuQyxbnAoVAVzMrMbNxZjbJzCbVDPkp0Ap4yszWmFlxI+YVaTLdc9J4cVwBx06dZfi0IvYcOel1JJGLMuecJy/s8/lccbG6X0Lf2pIjjJy+jJYpCcybOJCcFs28jiRRzMxWOud8tW3TO0VF6tErN53nHxjAoeNnGDFtGQeOnvI6kkitVOgiAeiTl8FzD/Sn9Ogphk8rovSYSl1CjwpdJED9OrTk2fsHsO/IKUZOW8bB46e9jiTyN1ToIpdgQMeWzBzbn92HKxk5bRnlJ854HUnkcyp0kUs06LJWzLyvPzsOnWDEtCIOq9QlRKjQRb6EwZdnMv0+H9sOnmDk9GUcqVSpi/dU6CJf0pDOWUwd3Y8tpccZNWMZFZVnvY4kUU6FLtIA13VtzZTR/fhk/3FGz1xGxUmVunhHhS7SQNd3a83To/qyad9RxsxYxtFTKnXxhgpdJAhu6J7N0yP7sXHfUcbMWM4xlbp4QIUuEiRf7ZHN5BF9Wb+ngjEzVerS9FToIkH091e04ckRfVlXUsHYZ1eo1KVJqdBFgmzolW14ckQfPtp9RKUuTUqFLtIIhl6Zo1KXJqdCF2kk55f6fZpTlyagQhdpRP5S78vaEv+JUl3SKI0pkBWLZppZqZmtr2O7mdkTZrbFzNaaWd/gxxQJX/45df+J0jEzVOrSeAI5Qp8FDL3I9puBzjUfE4GnGx5LJLIMvbINk0f6L2kcPWO53lEqjaLeQnfOLQLKLzJkGPC88ysC0s0sJ1gBRSLFTVe04elR/di4t4LRuveLNIJgzKG3A3af97ik5mtfYGYTzazYzIrLysqC8NIi4eXGHtk8M6ofm/cdY8T0It2lUYIqGIVutXyt1pWnnXNTnXM+55wvKysrCC8tEn5u6J7NlDH9+LT0OMO1SIYEUTAKvQRof97jXGBvEJ5XJGJd37U108f42FZ2nOFTi7ScnQRFMAp9ATCm5mqXgUCFc25fEJ5XJKJd0yWLZ8f2Z1d5JfdOLaL0qBaeloYJ5LLFuUAh0NXMSsxsnJlNMrNJNUPeArYBW4BpwMONllYkwgy+PJNZ9/dn75GT3Du1iH0VJ72OJGHMnKt1urvR+Xw+V1xc7Mlri4SalTvLGTtzBekp8cwZP5D2LZO9jiQhysxWOud8tW3TO0VFQkC/Di2ZPaGAoyeruGdKIdsPnvA6koQhFbpIiOiVm87cCQM5VVXNPVMK+fTAMa8jSZhRoYuEkB5t03hp4kAccM/UItbvqfA6koQRFbpIiOmcncr8BweRFBfDiGlFrNp12OtIEiZU6CIhqGNmCvMnDSIjJYFR05exdOtBryNJGFChi4So3IxkXn5wELkZzbj/2RW8t/mA15EkxKnQRUJY67Qk5k0cRJfsVCY+v5I31+pN2FI3FbpIiGuZksDsCQX0yUvnO3NXM2/5Lq8jSYhSoYuEgbSkeJ5/oIAhnbP48WvrmLpoq9eRJASp0EXCRLOEWKaN8XFrrxwee2szv3pnM16901tCU5zXAUQkcAlxMTxxbx/SkuKZ/P5WDlee5RfDriQ2pra7WEu0UaGLhJnYGOOxO64kIzmepz7YSkXlWX5zz1UkxsV6HU08pkIXCUNmxg+HdiMjOYH/eGsTFSfP8szofjRP1F/paKY5dJEwNuGaTvzP3VdRuO2QFsoQFbpIuLurXy5TR/fj09Jj3PX0UnaXV3odSTwSUKGb2VAz+9jMtpjZj2vZnmdm75vZajNba2a3BD+qiNTlhu7ZzB5fwOHKs9z59FLd1CtKBbJiUSwwGbgZ6AEMN7MeFwz7F2C+c64PcC/wVLCDisjF9evQklcmDSI+xrh3ahEffqr7v0SbQI7QBwBbnHPbnHNngHnAsAvGOCCt5vMWaJFoEU90zk7ltYevJjejGWOfXc4fVpd4HUmaUCCF3g7Yfd7jkpqvne9nwCgzK8G/xui3a3siM5toZsVmVlxWVvYl4opIfdq0SGL+pEH0z2/J9176iMnvb9EbkKJEIIVe2zsWLvzpGA7Mcs7lArcAL5jZF57bOTfVOedzzvmysrIuPa2IBCQtKZ5ZD/Tn9t5t+dU7H/PPf1hH1blqr2NJIwvkotUSoP15j3P54pTKOGAogHOu0MySgEygNBghReTSJcbF8tt7etMuoxmT39/KvopTPDmir65Vj2CBHKGvADqbWUczS8B/0nPBBWN2ATcAmFl3IAnQnIqIx8yMf7ypG4/d0ZPFnx7k7mcK2Vdx0utY0kjqLXTnXBXwCPAOsAn/1SwbzOznZnZbzbAfABPM7CNgLjDWadJOJGSMKMhj5tj+7C6v5PbJS3RZY4Qyr3rX5/O54uJiT15bJFpt3n+UB55dwZGTZ3n83j7c2CPb60hyicxspXPOV9s2vVNUJIp0a5PG69+6ms6tmzPxhWKm/HWrroCJICp0kSjz2bJ2t1yZwy/f3swPX1nLmSpdARMJdLpbJAo1S4jl98P7cFlWCk+8t4Udh07w9Kh+ZDZP9DqaNICO0EWiVEyM8f2/78rj9/ZmbUkFw55cwsa9R72OJQ2gQheJcsN6t+PlSYM4V+34xtNLeWvdPq8jyZekQhcReuWms+CRq+mek8rDs1fx3ws3c65aJ0vDjQpdRAD/ydK5EwcyfEB7nvpgK+OeW0FF5VmvY8klUKGLyOcS42L55Z29eOyOnizZcpCvP/mh5tXDiApdRL5gREEe8yYO4nTVOe58eoluwxsmVOgiUqt+HTJ489tD6N0+ne+99BH/8vo6Tled8zqWXIQKXUTqlJWayIvjCnjwmk68WLSLu54uZNchrVkaqlToInJRcbEx/NMt3Zk2xsfOQye49feLeWfDfq9jSS1U6CISkBt7ZPPH7wyhY2YKD76wkp8t2KApmBCjQheRgLVvmczLkwbxwNUdmbV0B3c+tZRtZce9jiU1VOgickkS42L56dd7MH2Mjz1HTvK133/I/BW7ddfGEKBCF5Ev5as9snn7u0O4KjedH766lodnr+JI5RmvY0W1gArdzIaa2cdmtsXMflzHmG+a2UYz22Bmc4IbU0RCUU6LZrw4voAf39yNP288wNDfLWbxp1p90iv1FrqZxQKTgZuBHsBwM+txwZjOwD8BVzvnrgAebYSsIhKCYmOMSddexh8evprmSXGMnrGcny3YwMkzOmHa1AI5Qh8AbHHObXPOnQHmAcMuGDMBmOycOwzgnCsNbkwRCXU9c1vw5re/wv1X5zNr6Q5ufWIxK3ce9jpWVAmk0NsBu897XFLztfN1AbqY2RIzKzKzobU9kZlNNLNiMysuK9OvZSKRJik+ln/7+hXMHl/A6apq7n5mKY+9tYlTZ3W03hQCKXSr5WsXns6OAzoD1wHDgelmlv6FP+TcVOeczznny8rKutSsIhImrr48k4WPDuGe/nlMXbSNWx5fzPLt5V7HiniBFHoJ0P68x7nA3lrGvOGcO+uc2w58jL/gRSRKpSbF88s7e/LCuAGcOVfNN6cU8pM/rOPYKd2St7EEUugrgM5m1tHMEoB7gQUXjHkduB7AzDLxT8FsC2ZQEQlPQzpn8c6j1/DA1R2Zs3wXN/5mEQvX79N1642g3kJ3zlUBjwDvAJuA+c65DWb2czO7rWbYO8AhM9sIvA/8o3PuUGOFFpHwkpIYx0+/3oPXHhpMRkoCk15cxbjnitldrht9BZN59a+kz+dzxcXFnry2iHin6lw1s5bu4Dd//oRz1Y6Hr7ucB6/tRFJ8rNfRwoKZrXTO+WrbpneKikiTiouNYfyQTrz7/Wv5avdsfvvuJ9z427/ypw37NQ3TQCp0EfFE2/RmTB7ZlznjC0iKi2XiCysZNWMZm/ZpybsvS4UuIp4afHkmb313CP/vtivYsPcotz6xmB+/upYDR095HS3saA5dREJGReVZnnjvU54v3EFsjDHuKx158NrLSEuK9zpayLjYHLoKXURCzq5Dlfz6zx/zxpq9pCfH8+A1l3Hf4A4kJ8R5Hc1zKnQRCUvr91TwP3/6mA8+LiOzeQKTrr2MkQUdaJYQvVfEqNBFJKyt3FnOb/78CUu2HKJVSgLjhnRk9MAOpEbhVIwKXUQiwvLt5Tz5/hYWfVJGWlIcowd14L7B+bROTfI6WpNRoYtIRFlbcoSn3t/KOxv3Ex8Twx192vHAVzrStU2q19EanQpdRCLS9oMnmPnhdl5euZtTZ6sp6NiSsYPzubFHNnGxkXlVtgpdRCLa4RNnmF+8m+cLd7LnyElapyZyty+Xe3x55LVK9jpeUKnQRSQqnKt2vLe5lLnLd/HBx6VUOxjUqRV39G3HzVe2iYiTqCp0EYk6+ypO8nJxCa+tKmHHoUqS4mO4oXs2t/bM4fqurcP20kcVuohELeccq3Yd4Q+rS3h73X4OnThDs/hYru+WxVe7Z3N919ZkpCR4HTNgKnQREfy37l2+vZw/rtvHnzceoPTYaWIM+uZlcE2XLIZ0zqRXbjqxMbWtvBkaVOgiIheornas21PBu5sO8NdPyli3pwLnIDUxDl9+Bv07tqR/fkuubNsipKZnGlzoZjYUeByIBaY75/6zjnF3AS8D/Z1zF21rFbqIhJLyE2dYsuUgS7ceYsWOcraUHgcgNsbokp1Kr3Yt6JaTSrc2aXRtk0pLj6ZpGlToZhYLfALciH8x6BXAcOfcxgvGpQJ/BBKAR1ToIhLODh0/zapdR1hbcoSPSipYV3KEw5X/t8B1enI8+a1S6JiZQtv0JNq0aEbbFkm0ap5Iq5QEMlISSEmIxSy40zcXK/RAbl02ANjinNtW82TzgGHAxgvG/QL4b+AfGpBVRCQktGqeyI09srmxRzbgP7laduw0m/cf45MDx9h+8AQ7Dp1g+fZyDhw9RVX1Fw+OzSAlIY7khFgS4mKIizFiY4zhA/IYP6RT0DMHUujtgN3nPS4BCs4fYGZ9gPbOuTfNrM5CN7OJwESAvLy8S08rIuIRM6N1WhKt05K4pkvW32w7V+04ePw0+ypOcej4acpPnKH8xBlOnK7i+OlznDhdxdnqas5VO6qqHZnNExslYyCFXtvvC5//U2RmMcBvgbH1PZFzbiowFfxTLoFFFBEJbbExRnZaEtlp3t4kLJCbHZQA7c97nAvsPe9xKnAl8IGZ7QAGAgvMrNY5HhERaRyBFPoKoLOZdTSzBOBeYMFnG51zFSDwfH8AAAQJSURBVM65TOdcvnMuHygCbqvvpKiIiARXvYXunKsCHgHeATYB851zG8zs52Z2W2MHFBGRwAS0QJ9z7i3grQu+9tM6xl7X8FgiInKpIvOGwSIiUUiFLiISIVToIiIRQoUuIhIhPLvbopmVATu/5B/PBA4GMU440D5HB+1zdGjIPndwzmXVtsGzQm8IMyuu6+Y0kUr7HB20z9GhsfZZUy4iIhFChS4iEiHCtdCneh3AA9rn6KB9jg6Nss9hOYcuIiJfFK5H6CIicgEVuohIhAjpQjezoWb2sZltMbMf17I90cxeqtm+zMzymz5lcAWwz983s41mttbM/mJmHbzIGUz17fN54+4yMxcJ99oPZJ/N7Js13+sNZjanqTMGWwA/23lm9r6Zra75+b7Fi5zBYmYzzazUzNbXsd3M7Ima/x9rzaxvg1/UOReSH0AssBXohH/h6Y+AHheMeRh4pubze4GXvM7dBPt8PZBc8/lD0bDPNeNSgUX477fv8zp3E3yfOwOrgYyax629zt0E+zwVeKjm8x7ADq9zN3CfrwH6Auvr2H4L8Db+VeEGAssa+pqhfIT++eLUzrkzwGeLU59vGPBczeevADdYsJfYblr17rNz7n3nXGXNwyL8K0iFs0C+z/B/i5CfaspwjSSQfZ4ATHbOHQZwzpU2ccZgC2SfHZBW83kL/nZltLDjnFsElF9kyDDgeedXBKSbWU5DXjOUC722xanb1TXG+RfiqABaNUm6xhHIPp9vHP5/4cNZvft8/iLkTRmsEQXyfe4CdDGzJWZWZGZDmyxd4whkn38GjDKzEvzrL3y7aaJ55lL/vtcroAUuPHLRxakvYUw4CXh/zGwU4AOubdREjS9oi5CHkUC+z3H4p12uw/9b2GIzu9I5d6SRszWWQPZ5ODDLOfdrMxsEvFCzz9WNH88TQe+vUD5Cr29x6r8ZY2Zx+H9Nu9ivOKEukH3GzL4K/AT/2q2nmyhbY4nGRcgD/dl+wzl31jm3HfgYf8GHq0D2eRwwH8A5Vwgk4b+JVaQK6O/7pQjlQr/o4tQ1FgD31Xx+F/CeqznbEKbq3eea6Ycp+Ms83OdVIToXIQ/kZ/t1/CfAMbNM/FMw25o0ZXAFss+7gBsAzKw7/kIva9KUTWsBMKbmapeBQIVzbl+DntHrM8H1nCW+BfgE/9nxn9R87ef4/0KD/xv+MrAFWA508jpzE+zzu8ABYE3NxwKvMzf2Pl8w9gPC/CqXAL/PBvwG2AisA+71OnMT7HMPYAn+K2DWAH/vdeYG7u9cYB9wFv/R+DhgEjDpvO/x5Jr/H+uC8XOtt/6LiESIUJ5yERGRS6BCFxGJECp0EZEIoUIXEYkQKnQRkQihQhcRiRAqdBGRCPH/ARG1/iG8i7q2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(0, 1, 100)\n",
    "f = sched_cos(2, 0.4)\n",
    "y = [f(x) for x in x]\n",
    "\n",
    "plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also combine different schedulers to make new ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "def sched_combiner(pcts, scheds):\n",
    "    assert sum(pcts) == 1\n",
    "    pcts = torch.tensor([0.] + pcts)\n",
    "    pcts_cum = torch.cumsum(pcts, 0)\n",
    "    def _inner(pos):\n",
    "        if pos>=1: pos = 0.99999 #Error on the unlikely chance it hits 1.0 exactly\n",
    "        idx = (pos>=pcts_cum).nonzero().max()\n",
    "        new_pos = (pos - pcts_cum[idx])/(pcts_cum[idx+1] - pcts_cum[idx])\n",
    "        return scheds[idx](new_pos)\n",
    "    return _inner\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x206f7319a88>]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUVf7/8dcnPYEklCSUENIIvQlRkCIgomDDrii6KIpYWLHs6upX19VtroqK2FCxrC6KuxasSBNEQAhSQ0hIA0KAJJQkpJfz+yPRH2JIBpiZO+XzfDx4bJK5mXlfA+89OXPvOWKMQSmllPvzsTqAUkop+9BCV0opD6GFrpRSHkILXSmlPIQWulJKeQg/q144IiLCxMXFWfXySinlljZs2FBkjIls6jHLCj0uLo6UlBSrXl4ppdySiOw60WM65aKUUh5CC10ppTyEFrpSSnkILXSllPIQWuhKKeUhWix0EYkRkeUikiYiqSJyTxPHiIjMFpFMEdkiIoMcE1cppdSJ2HLZYi1wvzHmJxEJBTaIyGJjzPZjjpkAJDX+GQK80vi/SimlnKTFQjfG7AP2NX5cKiJpQDRwbKFPBN41DWvxrhWRNiLSqfF7lXJZlTV1ZBeWsedwOXsOlVNSUfPLY8EBfnRpG0xMuxDiI1oRHuxvYVKlWnZSNxaJSBxwBvDjcQ9FA3uO+Tyv8Wu/KnQRmQZMA+jatevJJVXKDowxbMkr5rv0QlZnFbFx9xGq6+p/dYzIz8f++mt9OocxLDGCkUkRDEuMwNdHnJhcqZbZXOgi0hr4HzDTGFNy/MNNfMtvds4wxswF5gIkJyfrzhrKaQ6VVfPxT3ksSNlDxoGjvxT0zcPj6N+lDV3bhRDTLpjwYH+ksdHLqmobR+4VbM8vYXVWEW//kMvcldl0DAviysHRXJvcla7tQyw+O6UaiC07FomIP/AFsMgYM6uJx18DvjPGzG/8PB0Y3dyUS3JystFb/5WjHSqr5rUVWbyzJpfKmnoGxLTh2uQYJvTtSNtWASf9fBXVdXyXXsCClD2syCgEYOLAaO4Zm0RcRCs7p1fqt0RkgzEmuanHWhyhS8Nw5U0grakyb7QQuFtEPqDhzdBinT9XVqqsqeOV77J44/tsymvqmDigM9NHJ9KzY9hpPW9wgC8T+nViQr9O7Cuu4O0fcnlnTS4LN+dz1aAu/GF8DyJaB9rnJJQ6SS2O0EVkBPA9sBX4ebLxYaArgDHm1cbSnwOMB8qBm40xzQ6/dYSuHGV1ZhEPf7KV3IPlXNivI/ee152kDqEOe72C0kpe+S6L99fuJjjAl0cu6sXVg7v8MnWjlD01N0K3acrFEbTQlb2VVdXyl89TWZCSR2z7EP5+eT+Gd4tw2utnFpTy0P+2krLrMMMS2/PM1QPo3CbYaa+vvIMWuvJ46ftLufP9DeQUlXHbOQnce153gvx9nZ6jvt4wf/1u/v5lGgF+Psy6diBjekQ5PYfyXM0Vut76r9zefzfkMfGlVRRX1PLe1CH8aUIvS8ocwMdHuGFILJ/PGEGHsCBufms9Ty/aQe1xl0Yq5Qha6Mpt1dcb/vF1Gg98tJmBMW346p4RDHPiFEtzEiJb8+ldw7k2OYaXlmcx/b0NlFfXWh1LeTgtdOWWqmrruOfDTby2IpvJQ7vy3tQhRIUGWR3rV4L8fXnqqv48ObEPy3YUMGnuWgpLq6yOpTyYFrpyOyWVNdz05jo+35zPQxN68uTEvvj5uu5f5RvPjuO1G5NJP1DKFa/8QG5RmdWRlIdy3X8FSjWhuKKGG99cx0+7D/PCdQOZPirRLS4PHNe7Ax9MO5uyqjqunbuG7MKjVkdSHkgLXbmN4vIabnzzR7bnF/PyDYOZODDa6kgnZWBMG+bfNpTaOsN1c9eSpaWu7EwLXbmF4ooaJr/5I2n7SnjlhsGM693B6kinpEfHUOZPG0q90VJX9qeFrlxeZU0dt72Two79Jbw6eTDnuWmZ/6x7h1Dm3zYUYww3vvEj+4orrI6kPIQWunJptXX1/H7+RtbvOsSsawYytpd7l/nPkjqE8vbNZ1FSWcvv5q2juLym5W9SqgVa6MplGWN49LNtfLv9AH++uDeXDOhsdSS76hsdztwbB5NbVM7Ud9ZTUV1ndSTl5rTQlct6+bss5q/bw11jEpkyPN7qOA4xrFsEs64dwIbdh7n3w03U1+s2AerUaaErl/TNtv08vSidiQM788D5PayO41AX9+/MwxN68U3qfp5futPqOMqNndQWdEo5Q9q+Eu5bsIkBMW146sr+bnGd+em6dWQ86QdKmb10J907tObi/p41vaScQ0foyqUUHa3i1ndSCAvy5/UbB1u2yJaziQh/u7wvg2Pb8sBHm9maV2x1JOWGtNCVy6irN/x+/kaKjlYx96bBRIW51tosjhbo58urkwfTLiSA6e9t4Eh5tdWRlJtpsdBFZJ6IFIjIthM8Hi4in4vIZhFJFZGb7R9TeYMXlmSwOusgT07sS/8ubayOY4nI0EBenjyYgtJK7l+wWd8kVSfFlhH62zRsLXcidwHbjTEDgNHAsyJy8rvvKq+2IqOQF5dncvXgLlxzZozVcSw1MKYN/3dRb5buKOC1ldlWx1FupMVCN8asBA41dwgQ2rivaOvGY3XhZ2Wz/CMVzPxgIz06hPLExL5Wx3EJN50dy0X9O/HMt+mszT5odRzlJuwxhz4H6AXk07CR9D3GmCa3ZxGRaSKSIiIphYWFdnhp5e7q6g0zP9hEdW09L98wiOAA73gTtCUiwlNX9ie2XQgzP9jE4TKdT1cts0ehXwBsAjoDA4E5IhLW1IHGmLnGmGRjTHJkZKQdXlq5u1dXZLEu9xBPXtaXhMjWVsdxKa0D/Zg96QwOllXx8CdbsWr/X+U+7FHoNwMfmwaZQA7Q0w7Pqzzc5j1HeG5xBhf378TlZ7jXUrjO0jc6nPvP78HX2/bz0YY8q+MoF2ePQt8NjAUQkQ5AD0DfyVHNKquqZeaHm4gKDeRvl/XzipuHTtW0kQmcndCexxem6m5Hqlm2XLY4H1gD9BCRPBGZKiLTRWR64yFPAsNEZCuwFHjQGFPkuMjKE/z1yzRyD5Yx69qBhIf4Wx3Hpfn4CM9eMwA/H2Hmh5uorWvyLSqlWr713xgzqYXH84Hz7ZZIebzvdxYyf91upp2TwNCE9lbHcQud2wTz5GV9ueeDTbyxKofpoxKtjqRckN4pqpyqtLKGB/+7hYTIVtw3rrvVcdzKpQM6c0GfDsxanEFmQanVcZQL0kJXTvX3r9LYX1LJ01cN8Jp1WuxFRPjrZf1oFeDL/R9t0akX9Rta6MppGqZa9nDryAQGx7a1Oo5bigwN5C8T+7J5zxFe/z7H6jjKxWihK6cor67lof9t1akWO7ikfyfG9+nIc0syyNGrXtQxtNCVUzy/ZCd7j1Twj8v76VTLaRIRnpjYh0A/Hx7RG47UMbTQlcNt21vMm6tymHRWDEP0qha7iAoL4sHxPVmddZD//bTX6jjKRWihK4eqqzf86eOttA0J4KHxvayO41GuP6srg2Pb8rcvt3PwaJXVcZQL0EJXDvX26ly27i3mz5f01huI7MzHR/jHFf04WlXL375MszqOcgFa6MphDpRUMuvbdEb3iOTi/p2sjuORuncI5fZzEvl4415dZldpoSvH+duXadTUG/5yaR9dq8WB7hrTjeg2wTz22TZq9Np0r6aFrhxidVYRCzfnM31UIrHtW1kdx6MFB/jy50t6k3HgKO+szrU6jrKQFrqyu5q6eh77LJUubYO5c7SuOeIM43p3YEyPSJ5bnMGBkkqr4yiLaKEru3vrhxwyC47y+CV99JpzJxERHr+0DzX1hr/qG6ReSwtd2VVBaSUvLNnJuT2jOK93B6vjeJXY9q2YPiqRzzfnsy6nuW2AlafSQld29fQ36VTX1fPoxb2tjuKV7hiVSKfwIP7yeSp19XoHqbexZYOLeSJSICLbmjlmtIhsEpFUEVlh34jKXWzJO8JHG/K4ZXg88RH6RqgVggN8+dOFvUjNL+GjlD1Wx1FOZssI/W1g/IkeFJE2wMvApcaYPsDV9omm3Ikxhr98vp2I1gHcfW43q+N4tUv6dyI5ti3PfJtOSWWN1XGUE7VY6MaYlUBzE3LX07BJ9O7G4wvslE25kYWb89mw6zB/vKAnoUF6R6iVRIQ/X9KHg2XVzFmWaXUc5UT2mEPvDrQVke9EZIOI3GSH51RupKK6jn9+vYO+0WFcNbiL1XEU0K9LONcMjuGtH3J0Y2kvYo9C9wMGAxcBFwCPikiTC16LyDQRSRGRlMLCQju8tHIF837IYV9xJY9e1BsfH70j1FXcf0F3/H19eOqbHVZHUU5ij0LPA74xxpQZY4qAlcCApg40xsw1xiQbY5IjIyPt8NLKaoWlVby8PJPze3fQpXFdTFRoENNHJfL1tv2k5OpljN7AHoX+GTBSRPxEJAQYAuidDV7ihaUZVNXW89CEnlZHUU24dWQ8HcIC+euXaboRhhew5bLF+cAaoIeI5InIVBGZLiLTAYwxacA3wBZgHfCGMeaElzgqz5FZUMr8dXu4YUhXEiJbWx1HNSEkwI/7z+/Bpj1H+GLLPqvjKAfza+kAY8wkG455GnjaLomU2/jHVzsI8ffl92OTrI6imnHloC7MW5XDU9/s4Pw+HQj00+UYPJXeKapOydrsgyzdUcAdYxJp3zrQ6jiqGb4+wsMX9iLvcAXvrd1tdRzlQFro6qQZY/jn1zvoGBbELcPjrY6jbHBO90hGdItgzrKderORB9NCVydtUep+Nu05wr3jknQ1RTfy4PieHC6vYe6KbKujKAfRQlcnpbaunn99k063qNZcOUhvInIn/bqEc8mAzryxKpsCXTPdI2mhq5OyICWP7KIy/nhBD/x89a+Pu3ng/O7U1hmeX7rT6ijKAfRfpLJZRXUdzy/JYHBsW8bpWuduKbZ9K24Y0pUP1+8hu/Co1XGUnWmhK5u9syaXgtIqHhzfUzd9dmMzxiYR6OfDrMUZVkdRdqaFrmxSUlnDK99lMbpHJGfFt7M6jjoNEa0DuWV4PF9s2UdqfrHVcZQdaaErm7y+MpviihoeOL+H1VGUHdx2TgLhwf48syjd6ijKjrTQVYuKjlbx5qocLurfib7R4VbHUXYQHuzP9FGJLE8vZL0u3OUxtNBVi15enkVlTR33jWtyVWTlpn43LJbI0ECe/iZdF+7yEFroqln5Ryp4b+0urhrchURdgMujhAT4MePcbqzLPcSKDN2fwBNooatmvbgsE4PRBbg81HVndiW6TTCzFmfoKN0DaKGrE9p9sJyPUvYw6ayudGkbYnUc5QABfj7cMzaJLXnFLEnT7YDdnRa6OqHZy3bi6yPcNaab1VGUA10xKJq49iHMWpxBfb2O0t2ZFrpqUlbhUT7+KY8bh8bSISzI6jjKgfx8fZh5XnfS9pXw9bb9VsdRp8GWHYvmiUiBiDS7C5GInCkidSJylf3iKau8sGQnQf6+TB+daHUU5QSXDOhMUlRrnluSQZ2O0t2WLSP0t4HxzR0gIr7AU8AiO2RSFkvfX8rnW/KZMiyOCN28wiv4+gj3jutOZsFRFm7ea3UcdYpaLHRjzEqgpTsPZgD/A/RdFQ/wwtIMWgX4cdvIBKujKCca36cjvTqFMXtpJrV19VbHUafgtOfQRSQauBx41YZjp4lIioikFBbqda+uKG1fCV9t3c/Nw+No2yrA6jjKiXx8hHvGJpFTVMZnm/KtjqNOgT3eFH0eeNAYU9fSgcaYucaYZGNMcmRkpB1eWtnbC0t2Ehrox60jdHTujS7o04HencJ4cdlOHaW7IXsUejLwgYjkAlcBL4vIZXZ4XuVkqfnFfJO6n5tHxBMe4m91HGUBEWHmeUnkHiznk406l+5uTrvQjTHxxpg4Y0wc8F/gTmPMp6edTDndC0t2Ehrkx9QRuvGzNxvXuwN9o8N4cVkmNTpKdyu2XLY4H1gD9BCRPBGZKiLTRWS64+MpZ0nNL+bb7QeYOiKe8GAdnXszEWHm2O7sPlTOxz/lWR1HnQS/lg4wxkyy9cmMMVNOK42yzAtLdhIW5MfNw3V0rmBsryj6RYczZ3kmVwzqgr/uH+sW9Kekfhmd36Kjc9VIpOGKlz2HKnQu3Y1ooStmL22YO9fRuTrW2F5R9I0O46Xlel26u9BC93Lb80tYlHqAW4br6Fz9WsMovTu79IoXt6GF7uVmL2247vwWHZ2rJpzXK4o+ncOYo6N0t6CF7sXS9pXodeeqWT/Ppe86WM6neveoy9NC92IvLttJ60A/puroXDVjXO+Gu0d1Lt31aaF7qfT9pXy1dT9ThsXp6Fw1S0T4/dhu5BSV8cWWfVbHUc3QQvdSc5Zn0irAV+8KVTY5v3dHenYM5cVlO3W9dBemhe6FMguO8sWWfG4apisqKtv4+Agzzk0iq7CMr7bqKN1VaaF7oZeWZxLk58utOjpXJ2FC344kRbXmxWU7de9RF6WF7mUa1rrey01nx9JedyNSJ8HHR5gxNomMA0f5JlX3HnVFWuhe5qXlmQT4+XCr7kakTsFF/TqRENmK2Ut1lO6KtNC9yO7GO/5uGBJLZKiOztXJ8/URZpzbjR37S1mSdsDqOOo4Wuhe5JUVmfj6CLefo6Nzdeou6d+Z2PYhzF62E2N0lO5KtNC9RN7hcv67IY9JZ8YQFRZkdRzlxvx8fbhrTDe27S1hebruC+9KtNC9xKsrsgC4fVSixUmUJ7j8jGi6tA1m9tJMHaW7EFt2LJonIgUisu0Ej98gIlsa/6wWkQH2j6lOx/7iShasz+Pq5Bg6twm2Oo7yAP6+Ptw5uhub9hzh+51FVsdRjWwZob8NjG/m8RxglDGmP/AkMNcOuZQdvboii3pjuENH58qOrhwcTefwIGYv1bl0V9FioRtjVgKHmnl8tTHmcOOna4Eudsqm7KCgtJL563ZzxaBoYtqFWB1HeZBAP1+mj04kZddh1mQftDqOwv5z6FOBr0/0oIhME5EUEUkpLCy080urpry+MpuaunruHN3N6ijKA12THENUaCCzl+60OorCjoUuImNoKPQHT3SMMWauMSbZGJMcGRlpr5dWJ3DwaBXvrd3NZQOjiYtoZXUc5YGC/H25fVQia7MPsS7nhL/IKyexS6GLSH/gDWCiMUZ/93IRb6zKobK2jjvH6OhcOc71Z3UlonUALy7TUbrVTrvQRaQr8DFwozEm4/QjKXs4Ul7Nu6tzubh/Z7pFtbY6jvJgwQG+3DYyge93FvHT7sMtf4NyGFsuW5wPrAF6iEieiEwVkekiMr3xkMeA9sDLIrJJRFIcmFfZaN6qHMqq67hbR+fKCSYPjaVtiD8v6ly6pfxaOsAYM6mFx28FbrVbInXaiitqeGt1LuP7dKRHx1Cr4ygv0CrQj1tHJvD0onS25hXTr0u41ZG8kt4p6oHeWZ1LaWUtM8bq6Fw5z01nxxIW5MdsnUu3jBa6hymtrOHNVTmM692BPp11lKScJzTIn6kjEli8/QCp+cVWx/FKWuge5t01uyiuqOH35yZZHUV5oSnD4wgN9GPOskyro3glLXQPUlZVyxvfZzOmR6TOYSpLhAf7M2V4HF9v20/6/lKr43gdLXQP8t7aXRwur2HGWB2dK+vcMjyeVgG+el26BbTQPUR5dS1zV2YzMimCQV3bWh1HebG2rQK4aVgcX27dR2aBjtKdSQvdQ7y/djcHy6qZeZ6OzpX1bhuZQLC/Ly/qXLpTaaF7gIrqOl5bmc3wbu0ZHNvO6jhK0a5VADcOjeXzzflkFR61Oo7X0EL3AP9Zt5uio1XcM7a71VGU+sVt5yQQ6OfLSzpKdxotdDdXWVPHqyuyODuhPWfF6+hcuY6I1oFMHtqVTzftJaeozOo4XkEL3c3NX7ebwtIq7tG5c+WCpp2TiL+vj16X7iRa6G6ssqaOV77L4qz4dgxNaG91HKV+IzI0kMlDY/l0015ydZTucFrobuyDdbspKK3i3vN07ly5rttHJeDnI3rFixNoobupypo6Xv4uiyHx7Tg7UUfnynVFhQbpKN1JtNDd1M+j85k6OlduQEfpzqGF7oZ0dK7cjY7SncOWHYvmiUiBiGw7weMiIrNFJFNEtojIIPvHVMfS0blyR7ePSsDfV0fpjmTLCP1tYHwzj08Akhr/TANeOf1Y6kR0dK7cVVRoEJOHxPLJxjyy9e5Rh2ix0I0xK4FDzRwyEXjXNFgLtBGRTvYKqH7tvbW7KCit4r5xOjpX7uf2UYkE+vkyW/cedQh7zKFHA3uO+Tyv8Wu/ISLTRCRFRFIKCwvt8NLepby6lldXZDGiWwRD9Lpz5YYiQwO5aVgsn23O15UYHcAehS5NfM00daAxZq4xJtkYkxwZGWmHl/Yu/16zi6Kj1dw7Tu8KVe7r9nMSCfH35fklOkq3N3sUeh4Qc8znXYB8OzyvOsbRqobR+ajukbqionJr7VoFMGV4w3rpO/aXWB3Ho9ij0BcCNzVe7TIUKDbG7LPD86pjvLM6l8PlNdyrc+fKA9w2MoHWAX68oKN0u7LlssX5wBqgh4jkichUEZkuItMbD/kKyAYygdeBOx2W1ksVV9Qwd2U25/aMYmBMG6vjKHXa2oQEcMuIeL7etp9te4utjuMx/Fo6wBgzqYXHDXCX3RKp33jz+2yKK2r0yhblUaaOjOft1bnMWpzBvClnWh3HI+idoi7u4NEq3lyVw4X9OtI3OtzqOErZTViQP7ePSmDZjgI27DpsdRyPoIXu4l5dkUVFTZ2OzpVHmjIsjojWATyzKN3qKB5BC92FHSip5N01u7jsjGi6RYVaHUcpuwsJ8OPO0d1Yk32Q1ZlFVsdxe1roLmzOskzq6g0zda9Q5cGuH9KVTuFBPP1tOg1vyalTpYXuonYfLOeD9bu55swYurYPsTqOUg4T5O/LjHOT2Lj7CEvTCqyO49a00F3Uc0sy8BHhnrF6V6jyfFcndyE+ohVPL0qnrl5H6adKC90Fpe0r4dNNe7l5eDwdwoKsjqOUw/n7+nD/+d1JP1DKZ5v2Wh3HbWmhu6BnFqUTGujHHaMSrY6ilNNc2LcTfaPDmLU4g6raOqvjuCUtdBezPvcQS3cUMH10IuEh/lbHUcppfHyEP17Qk7zDFcz/cbfVcdySFroLMcbwr292EBUayM3D4q2Oo5TTjUyK4OyE9ry4LJOjVbVWx3E7WuguZElaAetzD/P7sUkEB/haHUcppxMR/ji+BwfLqnl9ZbbVcdyOFrqLqK2r559fp5EQ2Yprz4xp+RuU8lBndG3LRf068fr32RSUVFodx61oobuIBSl5ZBWW8eD4nvj76o9Febc/XNCDmrp6ntet6k6KNocLKKuq5bklGSTHtuX83h2sjqOU5eIiWnHDkFg+XL9Ht6o7CVroLuCN73MoLK3iTxf2QqSpHf2U8j4zzu1GiL8vT32jC3fZyqZCF5HxIpIuIpki8lATj3cVkeUislFEtojIhfaP6pkKSip5bWUWE/p2ZHBsW6vjKOUy2rcOZProRBZvP8Da7INWx3ELtuxY5Au8BEwAegOTRKT3cYf9H7DAGHMGcB3wsr2Deqpnvk2npq6eP47vaXUUpVzO1BHxdA4P4q9fbqdelwRokS0j9LOATGNMtjGmGvgAmHjcMQYIa/w4HN0k2iap+cV8tCGP350dR3xEK6vjKOVygvx9+eP4nmzbW8LHG3VJgJbYUujRwJ5jPs9r/NqxHgcmi0geDXuMzmjqiURkmoikiEhKYWHhKcT1HMYY/vpFGm2C/ZmhC3ApdUKXDujMgJg2PL1oB+XVerNRc2wp9KbepTv+d59JwNvGmC7AhcC/ReQ3z22MmWuMSTbGJEdGRp58Wg+yePsB1mQf5N5x3QkP1lv8lToRHx/hsYt7caCkildX6M1GzbGl0POAY+906cJvp1SmAgsAjDFrgCAgwh4BPVF1bT1//yqNblGtuf6srlbHUcrlDY5tx8X9OzF3ZRb5RyqsjuOybCn09UCSiMSLSAANb3ouPO6Y3cBYABHpRUOhe/ecSjPe+iGH3IPlPHJRL/z0JiKlbPLg+J4YA//4eofVUVxWi21ijKkF7gYWAWk0XM2SKiJPiMiljYfdD9wmIpuB+cAUo3tJNelASSWzl+5kbM8oxvSIsjqOUm4jpl0It49K5PPN+XoZ4wmIVb2bnJxsUlJSLHltK9334Sa+2LKPb+89hzi9skWpk1JRXcd5s1YQGuTHFzNGeOVvuCKywRiT3NRj3vdfw0Ibdh3i4417uXVkvJa5UqcgOMCXRy7qxY79pfxnna6ZfjwtdCepqzc89lkqHcOCuGtMN6vjKOW2JvTtyLDE9jz7bQaHyqqtjuNStNCd5D8/7iI1v4SHL+pFq0A/q+Mo5bZEhMcv7cPRqlqe0jdIf0UL3QkKSiv516J0hndrzyX9O1kdRym3171DKFNHxPNhyh5Scg9ZHcdlaKE7wd++TKOqpp4nJvbV1RSVspN7xibROTyIRz7ZRk1dvdVxXIIWuoP9kFnEZ5vymT46kcTI1lbHUcpjtAr04/FL+5B+oJR5q3KsjuMStNAdqKq2jkc/3UZs+xDuHJ1odRylPM75fTpyXq8onl+yk7zD5VbHsZwWugO9tCyT7KIynpjYlyB/3fRZKUd4/NI+ADz2WSrefj+jFrqDpO0r4eXvsrjijGhGdffuhciUcqQubUN44IIeLNtRwMLN3r1ytxa6A9TW1fPg/7YQHuzPoxcfvxeIUsrepgyLY2BMGx5fmMrBo1VWx7GMFroDvPVDLlvyivnLxD60bRVgdRylPJ6vj/Cvq/pztKqWJ77YbnUcy2ih21luURnPLk5nXO8OXNRPrzlXylm6dwjl7jFJfLYpnyXbD1gdxxJa6HZUV2+4/6PNBPj68KRec66U090xOpGeHUP50ydbOeyFywJoodvR3JXZbNh1mCcv60vH8CCr4yjldQL8fJh1zUCOlFfzf59u87qrXrTQ7WR7fgmzFqdzUb9OXDqgs9VxlPJavTuHce+47ny5dZ/XXfWihW4HVbV13LdgE21CAnjyMp1qUcpqt5+TyODYtjz66Tb2FXvPlnU2FbqIjBeRdLgBMtMAAAusSURBVBHJFJGHTnDMNSKyXURSReQ/9o3p2p5ZlM6O/aU8dWU/2ulVLUpZztdHePbqAdTUGe5fsJm6eu+Yemmx0EXEF3gJmAD0BiaJSO/jjkkC/gQMN8b0AWY6IKtLWr6jgNe/z2Hy0K6c27OD1XGUUo3iIlrx+KW9WZ11kFdXZFkdxylsGaGfBWQaY7KNMdXAB8DE4465DXjJGHMYwBhTYN+YrulASSX3f7SZnh1D+b+L9AYipVzNNckxXDKgM7MWZ3jFMru2FHo0sOeYz/Mav3as7kB3EflBRNaKyPimnkhEpolIioikFBYWnlpiF1FXb5j5wSYqquuYc/0gXatFKRckIvz98r5Etwnmng82caTcsy9ltKXQm3qH7/gJKT8gCRgNTALeEJE2v/kmY+YaY5KNMcmRke69vsnspTtZk32QJyb2oVuULourlKsKDfJnzvVnUFBayQMfbaHeg+fTbSn0PCDmmM+7AMdfC5QHfGaMqTHG5ADpNBS8R1qadoAXlu7kikHRXDW4i9VxlFIt6N+lDQ9f2IslaQd4aXmm1XEcxpZCXw8kiUi8iAQA1wELjzvmU2AMgIhE0DAFk23PoK4ip6iMmR9uom90GH+/vJ9eoqiUm5gyLI7LBnZm1pIMlqd75tt8LRa6MaYWuBtYBKQBC4wxqSLyhIhc2njYIuCgiGwHlgN/MMYcdFRoq5RV1XL7v1Pw8xFenTxY582VciMiwj+u6E+vjmHcM38juUVlVkeyO7Hq1tjk5GSTkpJiyWufirp6w53vb2Dx9gO8e8sQRiRFWB1JKXUK9hwq55I5q4hsHch/7xhGeLC/1ZFOiohsMMYkN/WY3ilqo398lcai1AM8enFvLXOl3FhMuxBevmEQOUVl3PX+Tx61wbQWug3eXZPLG6tymDIsjpuHx1sdRyl1moYlRvDPK/uzKrOIRz7Z6jGLePlZHcDVLdl+gMcXpnJerw66+5BSHuSqwV3Yfaic2Ut3EtM2hBlj3f/CPC30ZqzOKuKu//xEn87hzJ40EF8fvaJFKU9y73lJ5B0q59nFGYSH+HPT2XFWRzotWugnsGHXYW59J4XY9iG8c8tZhATofyqlPI2I8NRV/SmtquWxz1IJ9vfl6uSYlr/RRekcehO27S1mylvriAoN5L2pQ3QFRaU8mL+vD3OuP4ORSRE8+L8tfO7Ga6hroR9nw67DXP/6WkID/Xjv1iFEhenOQ0p5ukA/X+bemExybDtmfriJj3/KszrSKdFCP8aqnUVMfuNH2rYK4MPbz6ZL2xCrIymlnCQ4wJe3bj6TIfHtuG/BZt5ZnWt1pJOmhd7om237uOXt9cS2D+Gj6WcT007LXClv0yrQj3lTzmRc7w78eWEqs5fudKtLGr2+0I0xzFm2kzve/4k+0WF8OO1sokJ1mkUpbxXk78srNwziikHRzFqcwX0LNlNZU2d1LJt49aUbFdV1/OG/m/liyz4mDuzMU1f21/VZlFL4+frw7NUDiG/fimcXZ5BdeJTXbkymY7hrD/a8doSecaCUK15ZzZdb9/HQhJ48f+1ALXOl1C9EhBljk3jtxsFkFhzlkjmrWJHh2hvzeF2h19cb5q3K4eIXV1FQUsm8353J9FGJugyuUqpJF/TpyMd3Dic82J/fzVvH4wtTXXYKxqumXHYeKOXPC1NZnXWQsT2j+OeV/YkMDbQ6llLKxfXoGMoXM0bw1Dc7eOuHXL7fWcgTE/syvJtrLdTnFcvnHimv5rnFGbz3425aBfjy0IReTDorRkflSqmTtmpnEX/6ZAt7DlVwfu8OPHxhL+IiWjnt9ZtbPtejC31/cSVvr87l/R93UVZVy/VDunLfuB5656dS6rRU1tTx5qocXlqeSU1dPRMHRnPbyAR6dAx1+GufdqGLyHjgBcAXeMMY888THHcV8BFwpjGm2bZ2VKFX1dbxQ2YRCzfl88WWfdQbw/i+Hfn92CR6dgyz++sppbxXQUklc5Zn8lFKHhU1dYxMiuDKQV04t1cUYUGO2TjjtApdRHyBDGAcDZtBrwcmGWO2H3dcKPAlEADc7YxCN8ZQWlVLWn4J2/JL2Lj7MN+lF3K0qpbQQD+uHNyFqSPi9SYhpZRDHSmv5v0fd/PvNbvYX1KJv68wLDGCs+Lb0adzGH06hxPROsAu07zNFbotb4qeBWQaY7Ibn+wDYCKw/bjjngT+BTxwGllbtDy9gCe/2E5JRQ0lFbVUH7PbSIewQC7q14nx/ToyPDGCAD+vu4hHKWWBNiEB3DWmG3eMSmTjniMsSt3Pku0HfnWZo6+PEBbkR3iwP5OHxnLryAS757Cl0KOBPcd8ngcMOfYAETkDiDHGfCEiJyx0EZkGTAPo2rXryacFwoP96dUxjLBgf8KD/Wkb4k/3jqH06Rymd3gqpSzl4yMMjm3L4Ni2PHxhL0oqa9ieX0LavhKKjlZR3DgQddTVdbYUelO/I/wyTyMiPsBzwJSWnsgYMxeYCw1TLrZF/LVBXdsy6Ia2p/KtSinlVGFB/gxNaM/QhPZOeT1b5iTygGNXfO8CHLtgcCjQF/hORHKBocBCEWlyjkcppZRj2FLo64EkEYkXkQDgOmDhzw8aY4qNMRHGmDhjTBywFri0pTdFlVJK2VeLhW6MqQXuBhYBacACY0yqiDwhIpc6OqBSSinb2HTrvzHmK+Cr47722AmOHX36sZRSSp0sva5PKaU8hBa6Ukp5CC10pZTyEFroSinlISxbbVFECoFdp/jtEUCRHeO4Az1n76Dn7B1O55xjjTGRTT1gWaGfDhFJOdHiNJ5Kz9k76Dl7B0eds065KKWUh9BCV0opD+GuhT7X6gAW0HP2DnrO3sEh5+yWc+hKKaV+y11H6EoppY6jha6UUh7CpQtdRMaLSLqIZIrIQ008HigiHzY+/qOIxDk/pX3ZcM73ich2EdkiIktFJNaKnPbU0jkfc9xVImI8Ya19W85ZRK5p/Fmnish/nJ3R3mz4u91VRJaLyMbGv98XWpHTXkRknogUiMi2EzwuIjK78b/HFhEZdNovaoxxyT+AL5AFJNCw8fRmoPdxx9wJvNr48XXAh1bndsI5jwFCGj++wxvOufG4UGAlDevtJ1ud2wk/5yRgI9C28fMoq3M74ZznAnc0ftwbyLU692me8znAIGDbCR6/EPiahl3hhgI/nu5ruvII/ZfNqY0x1cDPm1MfayLwTuPH/wXGij221bZOi+dsjFlujClv/HQtDTtIuTNbfs7w/zchr3RmOAex5ZxvA14yxhwGMMYUODmjvdlyzgYIa/w4nF/vjOZ2jDErgUPNHDIReNc0WAu0EZFOp/OarlzoTW1OHX2iY0zDRhzFgHM273MMW875WFNp+H94d9biOR+7CbkzgzmQLT/n7kB3EflBRNaKyHinpXMMW875cWCyiOTRsP/CDOdEs8zJ/ntvkU0bXFik2c2pT+IYd2Lz+YjIZCAZGOXQRI5nt03I3YgtP2c/GqZdRtPwW9j3ItLXGHPEwdkcxZZzngS8bYx5VkTOBv7deM71jo9nCbv3lyuP0FvanPpXx4iIHw2/pjX3K46rs+WcEZHzgEdo2Lu1yknZHMUbNyG39e/2Z8aYGmNMDpBOQ8G7K1vOeSqwAMAYswYIomERK09l07/3k+HKhd7s5tSNFgK/a/z4KmCZaXy3wU21eM6N0w+v0VDm7j6vCt65Cbktf7c/peENcEQkgoYpmGynprQvW855NzAWQER60VDohU5N6VwLgZsar3YZChQbY/ad1jNa/U5wC+8SXwhk0PDu+CONX3uChn/Q0PAD/wjIBNYBCVZndsI5LwEOAJsa/yy0OrOjz/m4Y7/Dza9ysfHnLMAsYDuwFbjO6sxOOOfewA80XAGzCTjf6syneb7zgX1ADQ2j8anAdGD6MT/jlxr/e2y1x99rvfVfKaU8hCtPuSillDoJWuhKKeUhtNCVUspDaKErpZSH0EJXSikPoYWulFIeQgtdKaU8xP8DE33ySQjoOw8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pcts = [0.5, 0.5]\n",
    "scheds = [sched_cos(0.4, 2), sched_cos(2, 0.4)]\n",
    "sched_new = sched_combiner(pcts, scheds)\n",
    "\n",
    "\n",
    "x = np.linspace(0., 1., 100)\n",
    "f = sched_new\n",
    "y = [f(x) for x in x]\n",
    "plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last step is to create a callback that will be able to change our parameter of choice according to the progress through the epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class ParamSchedulerCallback(Callback):\n",
    "    _order = 1\n",
    "    def __init__(self, pname, scheduler):\n",
    "        self.pname, self.scheduler = pname, scheduler\n",
    "        \n",
    "    def begin_fit(self):\n",
    "        setattr(self.opt, self.pname, self.scheduler(0.))\n",
    "        \n",
    "    def begin_batch(self):\n",
    "        pos = self.epoch/self.num_epochs\n",
    "        if hasattr(self.learn, 'n_batches'): #if you've included a way to measure the dataset length\n",
    "            pos += self.iter/self.learn.n_batches/self.num_epochs\n",
    "        setattr(self.opt, self.pname, self.scheduler(pos))\n",
    "           \n",
    "class BatchNumberCalculator(Callback):\n",
    "    def begin_fit(self):\n",
    "        ds_len = self.learn.data.train_dl.sampler.n\n",
    "        bs = self.learn.data.train_dl.sampler.bs\n",
    "        self.learn.n_batches = ds_len//bs\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n"
     ]
    }
   ],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "metrics = [percent_close, msle, mean_pred]\n",
    "\n",
    "\n",
    "callbacks = [ParamSchedulerCallback('lr', sched_new), \n",
    "             EarlyStoppingCallback(2, 3), \n",
    "             AvgStatsCallback(metrics, metrics)]\n",
    "callback_functions = [MoveBatchGPUCallback, BatchNumberCalculator]\n",
    "learn = Learner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)\n",
    "\n",
    "learn.fit(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In theory, we now have a method of changing parameters like the learning rate on the fly.  Let's create an easy way of testing this through a new callback: the Recorder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recorder\n",
    "\n",
    "We can also give callbacks class methods that are useful beyond what they do in the actual training cycle.  For instance, we can build a recorder class to track the loss and learning rates over time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export\n",
    "class RecorderCallback(Callback):\n",
    "    def begin_fit(self):\n",
    "        self.losses = [[],[]]\n",
    "        self.lrs = [[],[]]\n",
    "        \n",
    "    def after_loss(self):\n",
    "        train_valid_selector = 0 if self.in_train else 1\n",
    "        self.losses[train_valid_selector].append(self.loss)\n",
    "        self.lrs[train_valid_selector].append(self.opt.lr)\n",
    "    def plot_lrs(self, train = True):\n",
    "        train_valid_selector = 0 if train else 1\n",
    "        plt.plot(self.lrs[train_valid_selector])\n",
    "        \n",
    "    def plot_losses(self, train = True):\n",
    "        plt.plot(self.losses[train_valid_selector])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test the recorder callback, we'll make a few changes to our early stopper and call to .fit so that we train for 10 epochs, and do the first two batches of each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n",
      "train: [nan, 0.0, nan, nan]\n",
      "valid: [nan, 0.0, nan, nan]\n"
     ]
    }
   ],
   "source": [
    "data = Datablock(dl_train, dl_valid)\n",
    "metrics = [percent_close, msle, mean_pred]\n",
    "\n",
    "\n",
    "callbacks = [ParamSchedulerCallback('lr', sched_new), \n",
    "             EarlyStoppingCallback(10, 1), \n",
    "             AvgStatsCallback(metrics, metrics)]\n",
    "callback_functions = [MoveBatchGPUCallback, BatchNumberCalculator, RecorderCallback]\n",
    "learn = Learner(model = model, \n",
    "                data = data, \n",
    "                loss_func = loss_function, \n",
    "                opt = opt, cbs = callbacks, cbfs = callback_functions)\n",
    "\n",
    "learn.fit(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3hcd33n8fd3RrLutmV7lNiWbMeS0yTQXEXiLLdAaEnSkkALlJCEy0ObwnIt7Rbo7gILu38Unl6eboGsodmUBBJCCE2aTQk8NJACcYIdcg8hkh1Hih1rbMkX3S/z3T/mjK3I0mgsnZlzZvR5PY8eaWaO5nyfE+uT3/zO95yfuTsiIlL+ElEXICIi4VCgi4hUCAW6iEiFUKCLiFQIBbqISIWoimrHa9as8U2bNkW1exGRsrRz584D7p6a7bXIAn3Tpk3s2LEjqt2LiJQlM9sz12uachERqRAKdBGRCqFAFxGpEAp0EZEKoUAXEakQ8wa6mbWZ2f1m9oyZPWVmH59lGzOzfzCzLjN73MzOL065IiIyl0LaFieBP3f3R8ysCdhpZj9y96enbXM5sCX4ugj4WvBdRERKZN5Ad/d9wL7g56Nm9gywHpge6FcB3/TsvXi3m9lKM1sb/K5I2RqdmOLGn+9mdHyqaPt401mncHbryqK9vywdJ3VhkZltAs4DHprx0nqgZ9rj3uC5lwW6mV0PXA+wYcOGk6tUJAK37+jhSz94FgCz8N/fHf7tyZf44Z+9DivGDmRJKTjQzawR+B7wCXc/MvPlWX7lhJUz3H0bsA2gs7NTK2tIrLk7Nz+4h7NbV3D3R15TlH3cvqOHv7zjcR7a3c/WzauLsg9ZOgrqcjGzarJh/i13v3OWTXqBtmmPW4G9iy9PJDrbd/XzXN8g127dWLR9vOXsdayoq+bm7XNezS1SsEK6XAz4J+AZd//bOTa7G3hP0O2yFTis+XMpd7ds38OKumquPGdd0fZRtyzJOy5o5b4nX6Lv6GjR9iNLQyEj9FcD1wFvNLNHg68rzOyDZvbBYJt7gV1AF/B14D8Xp1yR0th/ZJT7nnqJd3a2UludLOq+rtm6kcmM852He+bfWCSPQrpcfsbsc+TTt3Hgw2EVJRK12x7uYTLjXHNR8aZbck5b08Brt6zh2w+/wIcuaacqqev9ZGH0L0dkhompDN9+eA+vOz3FpjUNJdnntVs3su/wKD/+dV9J9ieVSYEuMsOPn9nP/iNjXFfEk6EzXXpGC2tX1HKLTo7KIijQRWa4efse1q+s441ntJRsn1XJBO++cAP/8dwBdh8YKtl+pbIo0EWm6eob5OddB3n3RRtIJkp7oc8fXdhGVcL4lkbpskAKdJFpbtm+h+qk8Uevapt/45C1NNVy2StP5bs7exkp4q0GpHIp0EUCw+OTfG9nL1f89lrWNNZEUsN1WzdyeGSCf31c1+XJyVOgiwTuenQvR8cmS3oydKYLT1vF6ac06uSoLIgCXYTj920549QmLtjYHFkdZsa1WzfyeO9hHus5FFkdUp4U6CLAIy8c4ul9R7ju4o2R3/Xwbeetp35ZUqN0OWkKdBHg5gefp7Gmireeuz7qUmiqreZt563n7sf2cmh4POpypIwo0GXJOzA4xr1PvMQfnr+ehpqTWiKgaK7dupGxyQx37OyNuhQpIwp0WfJu39HD+FSG6y6O7mToTGeuXc6rNjXzrYdeIJPR0gFSmHgMR0Ty6Oob5P/8tJspzxNsDhl3MsF3d3CcTCb7OJ8dewa4ePNqOlqaQq58ca7dupGP3/YoH7xlJ421c/+pGkbCsisqJcwws+Dn7Gv5Tgmct2ElbzuvtQjVSxQU6BJ7393Rwx2P9LJ+ZV3e7ZIJCwIte3vQhB1/nM/aFbV89NKO8AoOyWWvPJWLTlvF0/tmLhD2cu7ZLp1M7n9iwWP3/P8zGx6f4vuPvMhbz10f+YlgCYcCXWKvOz3I6S1N3Pdnr4u6lJKqqUrynT+9uGjv/80Hn+ezdz1F+ugYLctri7YfKR3NoUvsdfUN0t5SmtvYLiXtqUYge3ylMhSyBN2NZtZnZk/O8foKM/tXM3vMzJ4ys/eHX6YsVWOTU7zQP0xHED4Sno6W7DHtTivQK0UhI/SbgMvyvP5h4Gl3Pwe4BPgbM1u2+NJE4PkDw2Qc2lsU6GFraaqhsaZKI/QKMm+gu/sDQH++TYCmYDHpxmDbyXDKk6UuN3ps1wg9dGZGe0sj3Wndf71ShDGH/o/AmcBe4Ang4+6emW1DM7vezHaY2Y50Oh3CrqXS5UaPm1OaQy+G9lSDRugVJIxAfzPwKLAOOBf4RzNbPtuG7r7N3TvdvTOVSoWwa6l03elB1q+so36ZGrKKoT3VyEtHRhkc04fqShBGoL8fuNOzuoDdwBkhvK9I0OGi6ZZiOXZiVKP0ihBGoL8AXApgZqcAvwXsCuF9ZYnLZJxd6SF1uBRR7tyEOl0qw7yfY83sVrLdK2vMrBf4HFAN4O43AF8EbjKzJ8heoPcpdz9QtIplydh7eISRiSn1oBfRxtX1VCVM8+gVYt5Ad/er53l9L/C7oVUkEsh1X2iEXjzVyQQbV9drhF4hdKWoxFZu1Kg59OLqaGnUCL1CKNAltrrTg6ysr2Z1g65TK6b2VCN7Dg4zMTVrt7GUEQW6xFZX3yDtqUbdCbDIOloamcw4ew4OR12KLJICXWJrV3pQ8+cloE6XyqFAl1g6NDzOgcFxdbiUQO4chebRy58CXWIpN1rs0AnRomusqeLU5bUaoVcABbrE0rEOF025lESHbtJVERToEkvd6SGWVSVoba6PupQloT3VQHffID7P+qsSbwp0iaWuvkE2r2kgmVCHSym0tzQyODZJ39GxqEuRRVCgSyx1p3VTrlLq0HJ0FUGBLrEzOjFFT/+w5s9LqF3L0VUEBbrEzvMHh8i4OlxKqaWphiYtR1f2FOgSO8c7XNSDXipmxuaWRo3Qy5wCXWKnu28IM9i8RiP0UupI6SZd5U6BLrHTFSw7V7csGXUpS0p7SwP7j4xxdHQi6lJkgRToEjvdfYOaP49Ax7F7uugCo3I1b6Cb2Y1m1mdmT+bZ5hIze9TMnjKzn4ZboiwlmYyz68CgOlwi0K71RcteISP0m4DL5nrRzFYCXwWudPdXAO8IpzRZil48NMLoREYj9AhsWFVPddLo0onRsjVvoLv7A0B/nk3eDdzp7i8E2/eFVJssQbkw0Qi99LLL0TVohF7GwphDPx1oNrOfmNlOM3vPXBua2fVmtsPMdqTT6RB2LZUmFyYaoUejI9WoEXoZCyPQq4ALgN8D3gz8dzM7fbYN3X2bu3e6e2cqlQph11JputODNNdXs0rLzkWivaWBF7QcXdkKI9B7gR+4+5C7HwAeAM4J4X1lCeruG9LoPELHl6NTp0s5CiPQ7wJea2ZVZlYPXAQ8E8L7yhLUlVaHS5Taj92kS4Fejqrm28DMbgUuAdaYWS/wOaAawN1vcPdnzOwHwONABviGu8/Z4igyl/6hcfqHxhXoEdqs9UXL2ryB7u5XF7DNl4Evh1KRLFladi56jTVVrF1Rq06XMqUrRSU2urXsXCy0q9OlbCnQJTa6+gapqUqwvrku6lKWtI6WRi1HV6YU6BIb3elBTtOyc5FrTzUwND7FS0dGoy5FTpICXWKjK62bcsXB8Xu6qNOl3CjQJRZGJ6boHRjR/HkMHF9f9GjElcjJUqBLLOxKD+Fadi4WUk01NNVW6Ta6ZUiBLrHQrZtyxYaZZTtd1LpYdhToEgtdfYPZZee0jmgsdGh90bKkQJdY6E4P0tpcR221lp2Lg/ZUI31Hxzii5ejKigJdYqGrb/DYyTiJXodWLypLCnSJ3FTG2X1gSPPnMdIeTH1pHr28zHsvFxGAobFJnsvzx+3uZBwy7mQyzpQ7mQzBdyeT56rDgeEJxia17Fyc5Jaje2h3P1tOaZpzOwOSCSNhRjJhJBPZk6rJ4HE+y2urWVFfHXLlS5sCXQrymTuf4O7H9hZ1H69Yt6Ko7y+Fq0om2NLSxB07e7ljZ29R9lFTleDhv3qTQj1ECnQpyG/2H+W8DSv52Bu3zLmN2fHR2vQRW+6x5Rmw1S+r0gg9Zra95wKe259/yiXjzlTwCWwqE3xCC56bysz9qew3+4/y9f/YTfeBQc7f0Bx26UuWAl3m5e68ODDCH5y/njec0RJ1OVIirc31tDbXF+W9n30pG+i9AyMK9BDNe1LUzG40sz4zy7tohZm9ysymzOzt4ZUncXB4ZIKjY5O0rSrOH7csPa3BHTV7+ocjrqSyFNLlchNwWb4NzCwJ/DVwXwg1Scz09I8Ax/8IRRaroaaK1Q3L6B1QoIdp3kB39weA/nk2+yjwPaAvjKIkXnqCP7piffyWpam1ue7YYEHCseg+dDNbD7wNuKGAba83sx1mtiOdTi9211IiuVGUplwkTK2r6jVCD1kYFxb9PfApd5+ab0N33+bune7emUqlQti1lEJP/wjLa6tYUaf2MglPW3M9Lx4aydsNIycnjC6XTuA2y/akrQGuMLNJd/+XEN5bYqBnYFjTLRK6tlV1TEw5+4+Msm6lzs+EYdGB7u6n5X42s5uAexTmlaWnf1g94hK63CChp39YgR6SQtoWbwUeBH7LzHrN7ANm9kEz+2Dxy5OouTu9AyO0aYQuIWsLuqZ6B3RiNCzzjtDd/epC38zd37eoaiR20oNjjE1mdEJUQre+uQ6z411Usni626LkpR50KZaaqiSnNNWqdTFECnTJSy2LUkytzXUaoYdIgS555eY3NUKXYmhbVc+LmkMPjQJd8urpH2ZN4zLql+k+bhK+tuY69h0eYWIqE3UpFUGBLnn1DAyzXh0uUiStq+rJOOw9pFF6GBTokle2ZVHTLVIcx++6qEAPgwJd5jSVcfYeGtEJUSma3PUNuqdLOBToMqeXjowyMeW6qEiKZu2KWpIJU6dLSBToMqfc4gNtqzTlIsVRlUywbqV60cOiQJc5HW9Z1Ahdiqd1Zb1G6CFRoMucevqHMYN1K2ujLkUqWNuqOt3PJSQKdJlTz8Awpy6vpaYqGXUpUsHamutJHx1jdGLeJRVkHgp0mVNvv+6yKMWX66JSp8viKdBlTr0Dw7rkX4oud9JdJ0YXT4EusxqfzLDvyCit6kGXImtVL3poFOgyq72HRnBHV4lK0aUaa1hWlaBHJ0YXrZAVi240sz4ze3KO168xs8eDr1+Y2Tnhlyml1qPb5kqJJBKWvY1uv0boi1XICP0m4LI8r+8GXu/uZwNfBLaFUJdETLfNlVJqa1YvehjmDXR3fwDoz/P6L9x9IHi4HWgNqTaJUE//MFUJY+0KBboUX2uzetHDEPYc+geAf5vrRTO73sx2mNmOdDod8q4lTD0DI6xbWUcyYVGXIktA26p6Dg1PcHR0IupSylpogW5mbyAb6J+aaxt33+bune7emUqlwtq1FEFP/7Du4SIlk7veQa2LixNKoJvZ2cA3gKvc/WAY7ynR6h0YoXWlTohKaRzrRdc8+qIsOtDNbANwJ3Cdu/9m8SVJ1EbGpzgwOKYRupTM8V50jdAXY96FIs3sVuASYI2Z9QKfA6oB3P0G4LPAauCrZgYw6e6dxSpYiq9XLYtSYs311TQsS6p1cZHmDXR3v3qe1/8Y+OPQKpLI5T726ra5UipmRtuqel0tuki6UlROkPvYq6tEpZRam+t1UnSRFOhygp7+YWqqEqSaaqIuRZaQbC/6MO4edSllS4EuJ+jpH6G1uY7gnIhISbStqmdofIqBYfWiL5QCXU7QMzCsE6JScrkpPp0YXTgFupygd0ALW0jp5QYR6kVfOAW6vMyR0QkOj0zoplxScrl/c+pFXzgFurxM7uOuplyk1Jpqq1lZX60pl0VQoMvL5NrGNOUiUcjeRlcj9IVSoMvLHL9KVFMuUnptq+ro1Qh9wRTo8jK9AyM01lSxoq466lJkCWprrqf30AiZjHrRF0KBLi/T0z+sHnSJTGtzHeOTGdKDY1GXUpYU6PIy6kGXKLXmWhc17bIgCnQ5xt3Vgy6ROrbQhXrRF0SBLsf0D40zPD6lHnSJzLFedN2ka0EU6HJMrl1MUy4SldrqJKmmGo3QF0iBLsccv6hII3SJTltznW6ju0DzBrqZ3WhmfWb25Byvm5n9g5l1mdnjZnZ++GVKKeQuudbCFhKltlX1GqEvUCEj9JuAy/K8fjmwJfi6Hvja4suSKPQMDNNcX01jzbwLWYkUTVtzPfsOjzI5lYm6lLIzb6C7+wNAf55NrgK+6VnbgZVmtjasAqV0evrVsijRa22uYyrj7Ds8GnUpZSeMOfT1QM+0x73Bcycws+vNbIeZ7Uin0yHsWsKklkWJA91Gd+HCCPTZLimc9bpdd9/m7p3u3plKpULYtYQlk3FeHBihVSdEJWK5QYVuo3vywgj0XqBt2uNWYG8I7ysl1Hd0jPGpjE6ISuTWrqwlYegmXQsQRqDfDbwn6HbZChx2930hvK+UUO7jbZsuKpKIVScTrF1Rp9voLsC87QxmditwCbDGzHqBzwHVAO5+A3AvcAXQBQwD7y9WsUvd8PgkBwfH53zdHTLuTLmTyWS/T2Ucd5jKOJk8q6k/2H0Q0EVFEg+tzXXsSg/Oe0+XRMJImpFIQNKMZMJIJIyEGYk895dLJoyaqmTIVUdv3kB396vned2BD4dWkczpLf/7Z3Snh4r2/suqEqxfqRG6RO+0NQ3c9sseXvul+4vy/smEcfufXswFG5uL8v5RUcNxmTg6OkF3eoi3nLOO158+9wnlhGX/sSZyo5Vj3yFhNvsp7MDaFbXUVlfeqEXKzyd/53RetWnV7N0VAffsp86pDMc/lQafRDOe/WQ6m8mM8+X7nuWRPQMKdIlGbmT++2ev5c2vODXiakSKq2V5LX94QWvR3v/Gn+2mOz1YtPePiu7lUia6+7L/+NpTjRFXIlL+2lsa6epToEtEutODVCWMjat10lJksdpTjRqhS3S6+gbZuLqe6qT+k4ksVkdLIwPDExyssKXulA5lojs9SEeLpltEwtCeagAoatdYFBToZWBiKsOeg8OaPxcJSW5wVGnz6Ar0MrDn4DCTGdcIXSQk61bUUVedrLh5dAV6GehSh4tIqBIJY3OqQSN0Kb3cKKJdI3SR0FRip4sCvQx09w1y6vJarSQkEqKOlkZePDTCyPhU1KWERoFeBtThIhK+9lQj7rDrQOWM0hXoMefudKeHjrVZiUg4KrHTRYEec/uPjDE4NqkRukjINq2pJ2GV1YuuQI85dbiIFEdNVZINq+qP3SepEhQU6GZ2mZk9a2ZdZvbpWV7fYGb3m9mvzOxxM7si/FKXJnW4iBRPpXW6zBvoZpYEvgJcDpwFXG1mZ83Y7L8Bt7v7ecC7gK+GXehS1Z0epKmmipammqhLEak4HS2N7DowxFQm353Xy0chI/QLgS533+Xu48BtwFUztnFgefDzCrRIdGi6+gbZ3NKIWZ6VKURkQdpTjYxPZugdqIwFqQsJ9PVAz7THvcFz030euDZYc/Re4KOzvZGZXW9mO8xsRzqdXkC5S093epAOzZ+LFEV7hXW6FBLosw0NZ34+uRq4yd1byS4YfbOZnfDe7r7N3TvdvTOVmnsZNck6MjrB/iNjtLeoZVGkGHKDpUqZRy8k0HuBtmmPWzlxSuUDwO0A7v4gUAusCaPApWxX0E6lEbpIcayor2ZNY82SGqH/EthiZqeZ2TKyJz3vnrHNC8ClAGZ2JtlA15zKIh1rWVSHi0jRtKcaKqYXfd5Ad/dJ4CPAfcAzZLtZnjKzL5jZlcFmfw78iZk9BtwKvM99rjW3pVDd6UGqk8aGVVp2TqRYOoL1RSshsgq625O730v2ZOf05z477eengVeHW5pkl51r0LJzIkXUnmrk8MgEB4fGWdNY3u3BSooYU4eLSPFV0j1dFOgxNT4ZLDunDheRosqdo6qEThcFeky90J+9ek035RIprrXLa6lfltQIXYpHN+USKY1KWo5OgR5TuTaqzQp0kaLrSDUeu+6jnCnQY6qrb5C1K7TsnEgptKeyy9ENjU1GXcqiKNBjqjs9qOkWkRLJnavafaC8R+kK9Bhyd7r7tI6oSKlUyk26FOgx9NKRUYbGp7SOqEiJbFxdTzJhZd+6qECPoe6+7Mc+3cNFpDRyy9FphC6h6+o7CuguiyKlVAnL0SnQY6g7PURTbRUpLTsnUjLtLQ3sPjDE5FQm6lIWTIEeQ1192Q4XLTsnUjodqUYmppyegZGoS1kwBXoMdafV4SJSapXQ6aJAj5kjoxP0HR1TD7pIibVXwHJ0CvSY6Q5GBxqhi5TWirpqUk3lvRxdQYFuZpeZ2bNm1mVmn55jm3ea2dNm9pSZfTvcMpeO4zflUg+6SKl1lHmny7yBbmZJ4CvA5cBZwNVmdtaMbbYAnwFe7e6vAD5RhFqXhO70kJadE4lIe0tDWS9HV8gI/UKgy913ufs4cBtw1Yxt/gT4irsPALh7X7hlLh1dfYNsWt1AlZadEym5jlQjR0cnSQ+ORV3KghSSGuuBnmmPe4PnpjsdON3Mfm5m283sstneyMyuN7MdZrYjnU4vrOIKt0s35RKJTLl3uhQS6LM1Q8/8PFIFbAEuAa4GvmFmK0/4Jfdt7t7p7p2pVOpka61445MZ9vQP64SoSEQ6ji1HV553XSwk0HuBtmmPW4G9s2xzl7tPuPtu4FmyAS8nYc/B7LJzWkdUJBqnLq+lYVnyWLdZuSkk0H8JbDGz08xsGfAu4O4Z2/wL8AYAM1tDdgpmV5iFLgW5s+sdqaaIKxFZmsyM9pby7XSZN9DdfRL4CHAf8Axwu7s/ZWZfMLMrg83uAw6a2dPA/cB/cfeDxSq6UuXm7TarZVEkMu2pxrKdQy9ofTN3vxe4d8Zzn532swOfDL4q1ujEFH/1/SfoHxqfcxv37AkGdyfjjjtk3Mk44OAnnH447vmDw6xbUUuDlp0TiUxHSyPf/9WLvPfGh8l3O6WEGQkDyH5PmJFIZEf5892F6XfOOoWrzp3ZW7J4So6TcM/j+7jzkRc5a+1yqpNz/yez4D/09O9GdnXxfDfc2tLSyKVnnlKEykWkUJee2cL9v+7j0HCegRu8bLDm0wZvUwX0sJ/TekLPSCgU6Cfhlu17aE818P8+9hrdCVGkQp1x6nLu+NB/irqMBdHVKwV6ovcwj/Yc4rqtGxXmIhJLCvQC3bJ9D3XVSf7ggtaoSxERmZUCvQCHhye467EXeet561heWx11OSIis1KgF+COR3oZnchw7daNUZciIjInBfo83J1vbd/D+RtW8op1K6IuR0RkTgr0efyi+yC7Dgxx3cUanYtIvCnQ53Hzg3tY1bCMy1+5NupSRETyUqDnse/wCD96Zj/v6GyltjoZdTkiInkp0PO49eEeMu5cc6GmW0Qk/hToc5iYynDrwy9wyekpNqzWcnAiEn8K9Dn88Kn9pI+O6WSoiJQNBfocbt7+PK3Ndbz+9JaoSxERKYgCfRZdfUfZvqufay7aSDKh+7aISHlQoM/ilu0vsCyZ4J2dum+LiJSPggLdzC4zs2fNrMvMPp1nu7ebmZtZZ3glltbQ2CTf29nLFb99Kqsba6IuR0SkYPMGupklga8AlwNnAVeb2VmzbNcEfAx4KOwiS+muR/dydGxSJ0NFpOwUssDFhUCXu+8CMLPbgKuAp2ds90XgS8BfhFrhDD/9TZr/ec/MXYfnpcOjnLl2OedvaC7aPkREiqGQQF8P9Ex73AtcNH0DMzsPaHP3e8xszkA3s+uB6wE2bNhw8tUCjTVVbDmlcUG/W4jTT2nimq0btIiFiJSdQgJ9tmQ7tmiemSWAvwPeN98bufs2YBtAZ2fn/AvvzeKCjc1csPGChfyqiEhFK+SkaC/QNu1xK7B32uMm4JXAT8zseWArcHc5nxgVESlHhQT6L4EtZnaamS0D3gXcnXvR3Q+7+xp33+Tum4DtwJXuvqMoFYuIyKzmDXR3nwQ+AtwHPAPc7u5PmdkXzOzKYhcoIiKFKWQOHXe/F7h3xnOfnWPbSxZfloiInCxdKSoiUiEU6CIiFUKBLiJSIRToIiIVwtwXdH3P4ndslgb2LPDX1wAHQiwnTKptYeJcG8S7PtW2MOVa20Z3T832QmSBvhhmtsPdY3nhkmpbmDjXBvGuT7UtTCXWpikXEZEKoUAXEakQ5Rro26IuIA/VtjBxrg3iXZ9qW5iKq60s59BFRORE5TpCFxGRGRToIiIVouwCvdAFq6NgZs+b2RNm9qiZRXr7YDO70cz6zOzJac+tMrMfmdlzwfdI1tmbo7bPm9mLwbF71MyuiKi2NjO738yeMbOnzOzjwfORH7s8tUV+7Mys1sweNrPHgtr+R/D8aWb2UHDcvhPcgjsutd1kZrunHbdzS13btBqTZvYrM7sneLyw4+buZfMFJIFuYDOwDHgMOCvquqbV9zywJuo6glpeB5wPPDntuS8Bnw5+/jTw1zGq7fPAX8TguK0Fzg9+bgJ+Q3Zx9MiPXZ7aIj92ZFc2awx+ria7WPxW4HbgXcHzNwAfilFtNwFvj/rfXFDXJ4FvA/cEjxd03MpthH5swWp3HwdyC1bLDO7+ANA/4+mrgH8Ofv5n4K0lLSowR22x4O773P2R4OejZNcAWE8Mjl2e2iLnWYPBw+rgy4E3AncEz0d13OaqLRbMrBX4PeAbwWNjgcet3AJ9tgWrY/EPOuDAD81sZ7Agdtyc4u77IBsOQEvE9cz0ETN7PJiSiWQ6aDoz2wScR3ZEF6tjN6M2iMGxC6YNHgX6gB+R/TR9yLOL5ECEf68za3P33HH7X8Fx+zszq4miNuDvgb8EMsHj1SzwuJVboOddsDoGXu3u5wOXAx82s9dFXVAZ+RrQDpwL7AP+JspizKwR+B7wCXc/EmUtM81SWyyOnbtPufu5ZNcdvhA4c7bNSltVsNMZtZnZK4HPAGcArwJWAZ8qdV1m9vtAn7vvnP70LJsWdNzKLdDnW7A6Uu6+N/jeB3yf7D/qONlvZmsBgu99EddzjLvvD/7oMsDXifDYmVk12cD8lrvfGTwdi2M3W21xOnZBPYeAn5Cdp15pZrmV0SL/e51W22XBFFUSHwwAAAFDSURBVJa7+xjwf4nmuL0auNLMnic7hfxGsiP2BR23cgv0vAtWR8nMGsysKfcz8LvAk/l/q+TuBt4b/Pxe4K4Ia3mZXFgG3kZExy6Yv/wn4Bl3/9tpL0V+7OaqLQ7HzsxSZrYy+LkOeBPZOf77gbcHm0V13Gar7dfT/gdtZOeoS37c3P0z7t7q7pvI5tm/u/s1LPS4RX12dwFng68ge3a/G/ivUdczra7NZLtuHgOeiro24FayH78nyH6y+QDZubkfA88F31fFqLabgSeAx8mG59qIansN2Y+3jwOPBl9XxOHY5akt8mMHnA38KqjhSeCzwfObgYeBLuC7QE2Mavv34Lg9CdxC0AkT1RdwCce7XBZ03HTpv4hIhSi3KRcREZmDAl1EpEIo0EVEKoQCXUSkQijQRUQqhAJdRKRCKNBFRCrE/weba5+TLwcl/gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.recorder.plot_lrs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, it's a bit blocky but has the general shape of our scheduler.  Had we trained will all the batches, the values in between would likely have smoothed out the curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Notes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Converted Part 4 - Callbacks.ipynb to scripts\\callbacks.py\n"
     ]
    }
   ],
   "source": [
    "!python scripts/notebook2script.py \"Part 4 - Callbacks.ipynb\" 'callbacks.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}