Music gerenation with Keras and TF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Music gereration firth Keras and TensorFlow backend\n",
    "\n",
    "Plan was simple:\n",
    "1. Read midi file, convert it to matrix of features\n",
    "2. Create simple model with Keras and LSTM to learn the pattern\n",
    "3. Use subsample of initial midi file as a input for model to generate pure art\n",
    "4. Save prediction from model to midi file\n",
    ".\n",
    ".\n",
    ".\n",
    "5. PROFIT\n",
    "\n",
    "<i> For disclamer: I've been using my macbook with no GPU support</i>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mYou are using pip version 18.1, however version 21.3.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "!{sys.executable} -m pip install -q mido tqdm pandas tensorflow scikit-learn\n",
    "# on my macOS I couldn't import keras properly, so I reinstall it with this command\n",
    "# maybe you will need it too 🤷‍♂️\n",
    "# !{sys.executable} -m  pip uninstall -y tensorflow keras tf-nightly keras-nightly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mido\n",
    "import numpy as np\n",
    "from mido import Message, MidiFile, MidiTrack\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.callbacks import ModelCheckpoint\n",
    "from tensorflow.keras.layers import LSTM, Activation, Dense, Dropout, Flatten\n",
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.optimizers import Adam\n",
    "from tensorflow.keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Read midi file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "mid = MidiFile(\n",
    "    \"Samples/Nintendo_-_Pokemon_Fire_Red_Route_1_Piano_Cover_Hard_Version.mid\"\n",
    ")\n",
    "notes = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Extract notes sequence\n",
    "for the sake of simplicity I will use only 1 channel, it's like playining on piano with 1 hand 🙃"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "notes = []\n",
    "for msg in mid:\n",
    "    if not msg.is_meta and msg.channel == 0 and msg.type == \"note_on\":\n",
    "        data = msg.bytes()\n",
    "        notes.append(data[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Apply min-max scalling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = MinMaxScaler(feature_range=(0, 1))\n",
    "scaler.fit(np.array(notes).reshape(-1, 1))\n",
    "notes = list(scaler.transform(np.array(notes).reshape(-1, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Prepare features for training and data subsample for prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LSTM layers requires that data must have a certain shape\n",
    "# create list of lists fist\n",
    "notes = [list(note) for note in notes]\n",
    "\n",
    "# subsample data for training and prediction\n",
    "X = []\n",
    "y = []\n",
    "# number of notes in a batch\n",
    "n_prev = 30\n",
    "for i in range(len(notes) - n_prev):\n",
    "    X.append(notes[i : i + n_prev])\n",
    "    y.append(notes[i + n_prev])\n",
    "# save a seed to do prediction later\n",
    "X_test = X[-300:]\n",
    "X = X[:-300]\n",
    "y = y[:-300]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Made sequential model with several layers, use LSTM as it time dependent data\n",
    "\n",
    "I also whant to save checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(LSTM(256, input_shape=(n_prev, 1), return_sequences=True))\n",
    "model.add(Dropout(0.6))\n",
    "model.add(LSTM(128, input_shape=(n_prev, 1), return_sequences=True))\n",
    "model.add(Dropout(0.6))\n",
    "model.add(LSTM(64, input_shape=(n_prev, 1), return_sequences=False))\n",
    "model.add(Dropout(0.6))\n",
    "model.add(Dense(1))\n",
    "model.add(Activation(\"linear\"))\n",
    "optimizer = Adam(learning_rate=0.001)\n",
    "model.compile(loss=\"mse\", optimizer=optimizer)\n",
    "filepath = \"./Checkpoints/checkpoint_model_{epoch:02d}.hdf5\"\n",
    "model_save_callback = ModelCheckpoint(\n",
    "    filepath,\n",
    "    monitor=\"val_acc\",\n",
    "    verbose=1,\n",
    "    save_best_only=False,\n",
    "    mode=\"auto\",\n",
    "    save_freq=5,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train your model.\n",
    "It might take a while, I was waiting for 1 hour with just 5 epoch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "  5/400 [..............................] - ETA: 30s - loss: 0.1279\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 10/400 [..............................] - ETA: 28s - loss: 0.0929\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 15/400 [>.............................] - ETA: 27s - loss: 0.0858\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 20/400 [>.............................] - ETA: 25s - loss: 0.0783\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 25/400 [>.............................] - ETA: 25s - loss: 0.0730\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 30/400 [=>............................] - ETA: 24s - loss: 0.0689\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 35/400 [=>............................] - ETA: 23s - loss: 0.0679\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 40/400 [==>...........................] - ETA: 23s - loss: 0.0669\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 45/400 [==>...........................] - ETA: 23s - loss: 0.0647\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 50/400 [==>...........................] - ETA: 22s - loss: 0.0633\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 55/400 [===>..........................] - ETA: 22s - loss: 0.0622\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 60/400 [===>..........................] - ETA: 21s - loss: 0.0608\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 65/400 [===>..........................] - ETA: 21s - loss: 0.0599\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 70/400 [====>.........................] - ETA: 21s - loss: 0.0591\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 75/400 [====>.........................] - ETA: 20s - loss: 0.0589\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 80/400 [=====>........................] - ETA: 20s - loss: 0.0583\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 85/400 [=====>........................] - ETA: 20s - loss: 0.0582\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 90/400 [=====>........................] - ETA: 19s - loss: 0.0584\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      " 95/400 [======>.......................] - ETA: 19s - loss: 0.0582\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "100/400 [======>.......................] - ETA: 19s - loss: 0.0577\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "105/400 [======>.......................] - ETA: 18s - loss: 0.0573\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "110/400 [=======>......................] - ETA: 18s - loss: 0.0570\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "115/400 [=======>......................] - ETA: 18s - loss: 0.0567\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "120/400 [========>.....................] - ETA: 17s - loss: 0.0563\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "125/400 [========>.....................] - ETA: 17s - loss: 0.0562\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "130/400 [========>.....................] - ETA: 17s - loss: 0.0558\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "135/400 [=========>....................] - ETA: 16s - loss: 0.0555\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "140/400 [=========>....................] - ETA: 16s - loss: 0.0558\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "145/400 [=========>....................] - ETA: 16s - loss: 0.0555\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "150/400 [==========>...................] - ETA: 15s - loss: 0.0553\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "155/400 [==========>...................] - ETA: 15s - loss: 0.0551\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "160/400 [===========>..................] - ETA: 15s - loss: 0.0548\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "165/400 [===========>..................] - ETA: 14s - loss: 0.0545\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "170/400 [===========>..................] - ETA: 14s - loss: 0.0546\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "175/400 [============>.................] - ETA: 14s - loss: 0.0544\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "180/400 [============>.................] - ETA: 13s - loss: 0.0542\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "185/400 [============>.................] - ETA: 13s - loss: 0.0541\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "190/400 [=============>................] - ETA: 13s - loss: 0.0538\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "195/400 [=============>................] - ETA: 12s - loss: 0.0536\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "200/400 [==============>...............] - ETA: 12s - loss: 0.0535\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "205/400 [==============>...............] - ETA: 12s - loss: 0.0534\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "210/400 [==============>...............] - ETA: 12s - loss: 0.0532\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "215/400 [===============>..............] - ETA: 11s - loss: 0.0532\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "219/400 [===============>..............] - ETA: 11s - loss: 0.0531\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "225/400 [===============>..............] - ETA: 11s - loss: 0.0530\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "230/400 [================>.............] - ETA: 10s - loss: 0.0529\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "235/400 [================>.............] - ETA: 10s - loss: 0.0527\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "240/400 [=================>............] - ETA: 10s - loss: 0.0527\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "245/400 [=================>............] - ETA: 9s - loss: 0.0526\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "250/400 [=================>............] - ETA: 9s - loss: 0.0525\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "255/400 [==================>...........] - ETA: 9s - loss: 0.0525\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "259/400 [==================>...........] - ETA: 8s - loss: 0.0525\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "265/400 [==================>...........] - ETA: 8s - loss: 0.0525\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "270/400 [===================>..........] - ETA: 8s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "275/400 [===================>..........] - ETA: 7s - loss: 0.0525\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "280/400 [====================>.........] - ETA: 7s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "285/400 [====================>.........] - ETA: 7s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "290/400 [====================>.........] - ETA: 6s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "295/400 [=====================>........] - ETA: 6s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "300/400 [=====================>........] - ETA: 6s - loss: 0.0524\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "305/400 [=====================>........] - ETA: 5s - loss: 0.0523\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "310/400 [======================>.......] - ETA: 5s - loss: 0.0523\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "315/400 [======================>.......] - ETA: 5s - loss: 0.0523\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "320/400 [=======================>......] - ETA: 4s - loss: 0.0521\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "325/400 [=======================>......] - ETA: 4s - loss: 0.0520\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "330/400 [=======================>......] - ETA: 4s - loss: 0.0521\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "335/400 [========================>.....] - ETA: 4s - loss: 0.0520\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "340/400 [========================>.....] - ETA: 3s - loss: 0.0519\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "345/400 [========================>.....] - ETA: 3s - loss: 0.0518\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "350/400 [=========================>....] - ETA: 3s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "355/400 [=========================>....] - ETA: 2s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "360/400 [==========================>...] - ETA: 2s - loss: 0.0518\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "365/400 [==========================>...] - ETA: 2s - loss: 0.0518\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "370/400 [==========================>...] - ETA: 1s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "375/400 [===========================>..] - ETA: 1s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "380/400 [===========================>..] - ETA: 1s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "385/400 [===========================>..] - ETA: 0s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "390/400 [============================>.] - ETA: 0s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "395/400 [============================>.] - ETA: 0s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "399/400 [============================>.] - ETA: 0s - loss: 0.0517\n",
      "Epoch 00001: saving model to ./Checkpoints/checkpoint_model_01.hdf5\n",
      "400/400 [==============================] - 29s 63ms/step - loss: 0.0517\n",
      "Epoch 2/5\n",
      "  5/400 [..............................] - ETA: 21s - loss: 0.0494\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 10/400 [..............................] - ETA: 22s - loss: 0.0456\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 15/400 [>.............................] - ETA: 22s - loss: 0.0468\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 20/400 [>.............................] - ETA: 22s - loss: 0.0488\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 25/400 [>.............................] - ETA: 22s - loss: 0.0485\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 30/400 [=>............................] - ETA: 22s - loss: 0.0476\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 35/400 [=>............................] - ETA: 22s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 40/400 [==>...........................] - ETA: 21s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 45/400 [==>...........................] - ETA: 21s - loss: 0.0481\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 50/400 [==>...........................] - ETA: 21s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 55/400 [===>..........................] - ETA: 20s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 60/400 [===>..........................] - ETA: 20s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 65/400 [===>..........................] - ETA: 20s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 70/400 [====>.........................] - ETA: 20s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 75/400 [====>.........................] - ETA: 19s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 80/400 [=====>........................] - ETA: 19s - loss: 0.0473\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 85/400 [=====>........................] - ETA: 19s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 90/400 [=====>........................] - ETA: 18s - loss: 0.0471\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      " 95/400 [======>.......................] - ETA: 18s - loss: 0.0471\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "100/400 [======>.......................] - ETA: 18s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "105/400 [======>.......................] - ETA: 18s - loss: 0.0471\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "110/400 [=======>......................] - ETA: 17s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "115/400 [=======>......................] - ETA: 17s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "120/400 [========>.....................] - ETA: 17s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "125/400 [========>.....................] - ETA: 17s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "130/400 [========>.....................] - ETA: 16s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "135/400 [=========>....................] - ETA: 16s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "140/400 [=========>....................] - ETA: 16s - loss: 0.0480\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "145/400 [=========>....................] - ETA: 15s - loss: 0.0482\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "150/400 [==========>...................] - ETA: 15s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "155/400 [==========>...................] - ETA: 15s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "160/400 [===========>..................] - ETA: 14s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "165/400 [===========>..................] - ETA: 14s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "170/400 [===========>..................] - ETA: 14s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "175/400 [============>.................] - ETA: 13s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "180/400 [============>.................] - ETA: 13s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "185/400 [============>.................] - ETA: 13s - loss: 0.0480\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "190/400 [=============>................] - ETA: 12s - loss: 0.0480\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "195/400 [=============>................] - ETA: 12s - loss: 0.0479\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "200/400 [==============>...............] - ETA: 12s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "205/400 [==============>...............] - ETA: 12s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "210/400 [==============>...............] - ETA: 11s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "215/400 [===============>..............] - ETA: 11s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "220/400 [===============>..............] - ETA: 11s - loss: 0.0478\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "225/400 [===============>..............] - ETA: 10s - loss: 0.0476\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "230/400 [================>.............] - ETA: 10s - loss: 0.0477\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "234/400 [================>.............] - ETA: 10s - loss: 0.0476\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "240/400 [=================>............] - ETA: 9s - loss: 0.0475\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "245/400 [=================>............] - ETA: 9s - loss: 0.0476\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "250/400 [=================>............] - ETA: 9s - loss: 0.0475\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "255/400 [==================>...........] - ETA: 8s - loss: 0.0476\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "260/400 [==================>...........] - ETA: 8s - loss: 0.0475\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "265/400 [==================>...........] - ETA: 8s - loss: 0.0475\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "270/400 [===================>..........] - ETA: 7s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "275/400 [===================>..........] - ETA: 7s - loss: 0.0473\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "280/400 [====================>.........] - ETA: 7s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "285/400 [====================>.........] - ETA: 7s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "290/400 [====================>.........] - ETA: 6s - loss: 0.0474\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "295/400 [=====================>........] - ETA: 6s - loss: 0.0473\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "300/400 [=====================>........] - ETA: 6s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "305/400 [=====================>........] - ETA: 5s - loss: 0.0472\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "310/400 [======================>.......] - ETA: 5s - loss: 0.0471\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "315/400 [======================>.......] - ETA: 5s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "320/400 [=======================>......] - ETA: 4s - loss: 0.0469\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "325/400 [=======================>......] - ETA: 4s - loss: 0.0470\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "330/400 [=======================>......] - ETA: 4s - loss: 0.0469\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "335/400 [========================>.....] - ETA: 3s - loss: 0.0468\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "340/400 [========================>.....] - ETA: 3s - loss: 0.0468\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "345/400 [========================>.....] - ETA: 3s - loss: 0.0468\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "350/400 [=========================>....] - ETA: 3s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "355/400 [=========================>....] - ETA: 2s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "360/400 [==========================>...] - ETA: 2s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "365/400 [==========================>...] - ETA: 2s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "370/400 [==========================>...] - ETA: 1s - loss: 0.0468\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "375/400 [===========================>..] - ETA: 1s - loss: 0.0469\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "380/400 [===========================>..] - ETA: 1s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "385/400 [===========================>..] - ETA: 0s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "390/400 [============================>.] - ETA: 0s - loss: 0.0466\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "395/400 [============================>.] - ETA: 0s - loss: 0.0466\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "399/400 [============================>.] - ETA: 0s - loss: 0.0467\n",
      "Epoch 00002: saving model to ./Checkpoints/checkpoint_model_02.hdf5\n",
      "400/400 [==============================] - 25s 62ms/step - loss: 0.0467\n",
      "Epoch 3/5\n",
      "  5/400 [..............................] - ETA: 24s - loss: 0.0453\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 10/400 [..............................] - ETA: 23s - loss: 0.0527\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 15/400 [>.............................] - ETA: 23s - loss: 0.0491\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 20/400 [>.............................] - ETA: 23s - loss: 0.0486\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 25/400 [>.............................] - ETA: 23s - loss: 0.0475\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 30/400 [=>............................] - ETA: 22s - loss: 0.0476\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 35/400 [=>............................] - ETA: 22s - loss: 0.0472\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 40/400 [==>...........................] - ETA: 21s - loss: 0.0477\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 45/400 [==>...........................] - ETA: 21s - loss: 0.0474\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 50/400 [==>...........................] - ETA: 21s - loss: 0.0472\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 55/400 [===>..........................] - ETA: 20s - loss: 0.0472\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 60/400 [===>..........................] - ETA: 20s - loss: 0.0471\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 65/400 [===>..........................] - ETA: 20s - loss: 0.0464\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 70/400 [====>.........................] - ETA: 19s - loss: 0.0463\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 75/400 [====>.........................] - ETA: 19s - loss: 0.0466\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 80/400 [=====>........................] - ETA: 19s - loss: 0.0471\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 85/400 [=====>........................] - ETA: 19s - loss: 0.0468\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 90/400 [=====>........................] - ETA: 18s - loss: 0.0468\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      " 95/400 [======>.......................] - ETA: 18s - loss: 0.0469\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "100/400 [======>.......................] - ETA: 18s - loss: 0.0469\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "105/400 [======>.......................] - ETA: 18s - loss: 0.0465\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "110/400 [=======>......................] - ETA: 18s - loss: 0.0463\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "115/400 [=======>......................] - ETA: 17s - loss: 0.0462\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "120/400 [========>.....................] - ETA: 17s - loss: 0.0462\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "125/400 [========>.....................] - ETA: 17s - loss: 0.0461\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "130/400 [========>.....................] - ETA: 16s - loss: 0.0461\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "135/400 [=========>....................] - ETA: 16s - loss: 0.0458\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "140/400 [=========>....................] - ETA: 16s - loss: 0.0459\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "145/400 [=========>....................] - ETA: 16s - loss: 0.0459\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "150/400 [==========>...................] - ETA: 15s - loss: 0.0458\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "155/400 [==========>...................] - ETA: 15s - loss: 0.0458\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "160/400 [===========>..................] - ETA: 15s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "165/400 [===========>..................] - ETA: 14s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "170/400 [===========>..................] - ETA: 14s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "175/400 [============>.................] - ETA: 14s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "180/400 [============>.................] - ETA: 13s - loss: 0.0458\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "185/400 [============>.................] - ETA: 13s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "190/400 [=============>................] - ETA: 13s - loss: 0.0457\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "194/400 [=============>................] - ETA: 12s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "200/400 [==============>...............] - ETA: 12s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "205/400 [==============>...............] - ETA: 12s - loss: 0.0454\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "210/400 [==============>...............] - ETA: 11s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "215/400 [===============>..............] - ETA: 11s - loss: 0.0456\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "220/400 [===============>..............] - ETA: 11s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "225/400 [===============>..............] - ETA: 11s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "230/400 [================>.............] - ETA: 10s - loss: 0.0454\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "235/400 [================>.............] - ETA: 10s - loss: 0.0455\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "240/400 [=================>............] - ETA: 10s - loss: 0.0454\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "245/400 [=================>............] - ETA: 9s - loss: 0.0453\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "250/400 [=================>............] - ETA: 9s - loss: 0.0453\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "255/400 [==================>...........] - ETA: 9s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "260/400 [==================>...........] - ETA: 9s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "265/400 [==================>...........] - ETA: 8s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "270/400 [===================>..........] - ETA: 8s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "275/400 [===================>..........] - ETA: 8s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "280/400 [====================>.........] - ETA: 7s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "285/400 [====================>.........] - ETA: 7s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "290/400 [====================>.........] - ETA: 7s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "295/400 [=====================>........] - ETA: 6s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "300/400 [=====================>........] - ETA: 6s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "305/400 [=====================>........] - ETA: 6s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "310/400 [======================>.......] - ETA: 5s - loss: 0.0453\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "315/400 [======================>.......] - ETA: 5s - loss: 0.0453\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "320/400 [=======================>......] - ETA: 5s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "325/400 [=======================>......] - ETA: 4s - loss: 0.0452\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "330/400 [=======================>......] - ETA: 4s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "335/400 [========================>.....] - ETA: 4s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "339/400 [========================>.....] - ETA: 3s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "345/400 [========================>.....] - ETA: 3s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "350/400 [=========================>....] - ETA: 3s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "355/400 [=========================>....] - ETA: 2s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "360/400 [==========================>...] - ETA: 2s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "365/400 [==========================>...] - ETA: 2s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "370/400 [==========================>...] - ETA: 1s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "375/400 [===========================>..] - ETA: 1s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "380/400 [===========================>..] - ETA: 1s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "385/400 [===========================>..] - ETA: 0s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "390/400 [============================>.] - ETA: 0s - loss: 0.0451\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "395/400 [============================>.] - ETA: 0s - loss: 0.0450\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "399/400 [============================>.] - ETA: 0s - loss: 0.0449\n",
      "Epoch 00003: saving model to ./Checkpoints/checkpoint_model_03.hdf5\n",
      "400/400 [==============================] - 26s 64ms/step - loss: 0.0449\n",
      "Epoch 4/5\n",
      "  5/400 [..............................] - ETA: 20s - loss: 0.0435\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 10/400 [..............................] - ETA: 23s - loss: 0.0414\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 15/400 [>.............................] - ETA: 22s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 20/400 [>.............................] - ETA: 22s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 25/400 [>.............................] - ETA: 22s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 30/400 [=>............................] - ETA: 22s - loss: 0.0427\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 35/400 [=>............................] - ETA: 22s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 40/400 [==>...........................] - ETA: 21s - loss: 0.0422\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 45/400 [==>...........................] - ETA: 21s - loss: 0.0420\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 50/400 [==>...........................] - ETA: 21s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 55/400 [===>..........................] - ETA: 20s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 60/400 [===>..........................] - ETA: 20s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 65/400 [===>..........................] - ETA: 20s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 70/400 [====>.........................] - ETA: 20s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 75/400 [====>.........................] - ETA: 19s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 80/400 [=====>........................] - ETA: 19s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 85/400 [=====>........................] - ETA: 19s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 90/400 [=====>........................] - ETA: 18s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      " 95/400 [======>.......................] - ETA: 18s - loss: 0.0433\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "100/400 [======>.......................] - ETA: 18s - loss: 0.0432\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "105/400 [======>.......................] - ETA: 18s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "110/400 [=======>......................] - ETA: 17s - loss: 0.0427\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "115/400 [=======>......................] - ETA: 17s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "120/400 [========>.....................] - ETA: 17s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "125/400 [========>.....................] - ETA: 17s - loss: 0.0433\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "130/400 [========>.....................] - ETA: 16s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "135/400 [=========>....................] - ETA: 16s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "140/400 [=========>....................] - ETA: 16s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "145/400 [=========>....................] - ETA: 15s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "150/400 [==========>...................] - ETA: 15s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "155/400 [==========>...................] - ETA: 15s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "160/400 [===========>..................] - ETA: 14s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "165/400 [===========>..................] - ETA: 14s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "170/400 [===========>..................] - ETA: 14s - loss: 0.0431\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "175/400 [============>.................] - ETA: 13s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "180/400 [============>.................] - ETA: 13s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "185/400 [============>.................] - ETA: 13s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "190/400 [=============>................] - ETA: 13s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "195/400 [=============>................] - ETA: 12s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "200/400 [==============>...............] - ETA: 12s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "205/400 [==============>...............] - ETA: 12s - loss: 0.0430\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "210/400 [==============>...............] - ETA: 11s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "215/400 [===============>..............] - ETA: 11s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "220/400 [===============>..............] - ETA: 11s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "225/400 [===============>..............] - ETA: 10s - loss: 0.0429\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "230/400 [================>.............] - ETA: 10s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "235/400 [================>.............] - ETA: 10s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "240/400 [=================>............] - ETA: 9s - loss: 0.0427 \n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "245/400 [=================>............] - ETA: 9s - loss: 0.0428\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "250/400 [=================>............] - ETA: 9s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "255/400 [==================>...........] - ETA: 9s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "260/400 [==================>...........] - ETA: 8s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "265/400 [==================>...........] - ETA: 8s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "270/400 [===================>..........] - ETA: 8s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "275/400 [===================>..........] - ETA: 7s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "280/400 [====================>.........] - ETA: 7s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "285/400 [====================>.........] - ETA: 7s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "290/400 [====================>.........] - ETA: 6s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "295/400 [=====================>........] - ETA: 6s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "300/400 [=====================>........] - ETA: 6s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "305/400 [=====================>........] - ETA: 5s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "310/400 [======================>.......] - ETA: 5s - loss: 0.0423\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "315/400 [======================>.......] - ETA: 5s - loss: 0.0423\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "320/400 [=======================>......] - ETA: 4s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "325/400 [=======================>......] - ETA: 4s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "330/400 [=======================>......] - ETA: 4s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "335/400 [========================>.....] - ETA: 4s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "340/400 [========================>.....] - ETA: 3s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "345/400 [========================>.....] - ETA: 3s - loss: 0.0427\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "350/400 [=========================>....] - ETA: 3s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "355/400 [=========================>....] - ETA: 2s - loss: 0.0427\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "360/400 [==========================>...] - ETA: 2s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "365/400 [==========================>...] - ETA: 2s - loss: 0.0427\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "370/400 [==========================>...] - ETA: 1s - loss: 0.0426\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "375/400 [===========================>..] - ETA: 1s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "380/400 [===========================>..] - ETA: 1s - loss: 0.0425\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "385/400 [===========================>..] - ETA: 0s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "390/400 [============================>.] - ETA: 0s - loss: 0.0423\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "395/400 [============================>.] - ETA: 0s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "399/400 [============================>.] - ETA: 0s - loss: 0.0424\n",
      "Epoch 00004: saving model to ./Checkpoints/checkpoint_model_04.hdf5\n",
      "400/400 [==============================] - 25s 62ms/step - loss: 0.0424\n",
      "Epoch 5/5\n",
      "  5/400 [..............................] - ETA: 20s - loss: 0.0395\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 10/400 [..............................] - ETA: 21s - loss: 0.0367\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 15/400 [>.............................] - ETA: 22s - loss: 0.0361\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 20/400 [>.............................] - ETA: 22s - loss: 0.0367\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 25/400 [>.............................] - ETA: 21s - loss: 0.0383\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 30/400 [=>............................] - ETA: 21s - loss: 0.0378\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 35/400 [=>............................] - ETA: 21s - loss: 0.0381\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 40/400 [==>...........................] - ETA: 21s - loss: 0.0381\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 45/400 [==>...........................] - ETA: 21s - loss: 0.0374\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 50/400 [==>...........................] - ETA: 20s - loss: 0.0378\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 55/400 [===>..........................] - ETA: 20s - loss: 0.0375\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 60/400 [===>..........................] - ETA: 20s - loss: 0.0377\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 65/400 [===>..........................] - ETA: 20s - loss: 0.0378\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 70/400 [====>.........................] - ETA: 19s - loss: 0.0381\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 75/400 [====>.........................] - ETA: 19s - loss: 0.0378\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 80/400 [=====>........................] - ETA: 19s - loss: 0.0376\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 85/400 [=====>........................] - ETA: 18s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 90/400 [=====>........................] - ETA: 18s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      " 95/400 [======>.......................] - ETA: 18s - loss: 0.0368\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "100/400 [======>.......................] - ETA: 18s - loss: 0.0367\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "105/400 [======>.......................] - ETA: 17s - loss: 0.0368\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "110/400 [=======>......................] - ETA: 17s - loss: 0.0368\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "115/400 [=======>......................] - ETA: 17s - loss: 0.0366\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "120/400 [========>.....................] - ETA: 16s - loss: 0.0368\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "125/400 [========>.....................] - ETA: 16s - loss: 0.0370\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "130/400 [========>.....................] - ETA: 16s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "135/400 [=========>....................] - ETA: 15s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "140/400 [=========>....................] - ETA: 15s - loss: 0.0372\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "145/400 [=========>....................] - ETA: 15s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "150/400 [==========>...................] - ETA: 15s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "155/400 [==========>...................] - ETA: 14s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "160/400 [===========>..................] - ETA: 14s - loss: 0.0370\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "165/400 [===========>..................] - ETA: 14s - loss: 0.0370\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "170/400 [===========>..................] - ETA: 13s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "175/400 [============>.................] - ETA: 13s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "180/400 [============>.................] - ETA: 13s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "185/400 [============>.................] - ETA: 13s - loss: 0.0371\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "190/400 [=============>................] - ETA: 12s - loss: 0.0370\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "195/400 [=============>................] - ETA: 12s - loss: 0.0368\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "200/400 [==============>...............] - ETA: 12s - loss: 0.0367\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "205/400 [==============>...............] - ETA: 11s - loss: 0.0365\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "210/400 [==============>...............] - ETA: 11s - loss: 0.0365\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "215/400 [===============>..............] - ETA: 11s - loss: 0.0365\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "220/400 [===============>..............] - ETA: 10s - loss: 0.0363\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "225/400 [===============>..............] - ETA: 10s - loss: 0.0363\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "230/400 [================>.............] - ETA: 10s - loss: 0.0362\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "235/400 [================>.............] - ETA: 9s - loss: 0.0361 \n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "240/400 [=================>............] - ETA: 9s - loss: 0.0361\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "245/400 [=================>............] - ETA: 9s - loss: 0.0361\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "250/400 [=================>............] - ETA: 9s - loss: 0.0361\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "255/400 [==================>...........] - ETA: 8s - loss: 0.0360\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "260/400 [==================>...........] - ETA: 8s - loss: 0.0359\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "265/400 [==================>...........] - ETA: 8s - loss: 0.0359\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "270/400 [===================>..........] - ETA: 7s - loss: 0.0359\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "275/400 [===================>..........] - ETA: 7s - loss: 0.0359\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "280/400 [====================>.........] - ETA: 7s - loss: 0.0358\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "285/400 [====================>.........] - ETA: 6s - loss: 0.0359\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "290/400 [====================>.........] - ETA: 6s - loss: 0.0358\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "295/400 [=====================>........] - ETA: 6s - loss: 0.0356\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "300/400 [=====================>........] - ETA: 6s - loss: 0.0356\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "305/400 [=====================>........] - ETA: 5s - loss: 0.0356\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "310/400 [======================>.......] - ETA: 5s - loss: 0.0356\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "315/400 [======================>.......] - ETA: 5s - loss: 0.0355\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "320/400 [=======================>......] - ETA: 4s - loss: 0.0353\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "325/400 [=======================>......] - ETA: 4s - loss: 0.0353\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "330/400 [=======================>......] - ETA: 4s - loss: 0.0353\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "335/400 [========================>.....] - ETA: 3s - loss: 0.0353\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "340/400 [========================>.....] - ETA: 3s - loss: 0.0352\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "345/400 [========================>.....] - ETA: 3s - loss: 0.0352\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "350/400 [=========================>....] - ETA: 3s - loss: 0.0351\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "355/400 [=========================>....] - ETA: 2s - loss: 0.0352\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "360/400 [==========================>...] - ETA: 2s - loss: 0.0351\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "365/400 [==========================>...] - ETA: 2s - loss: 0.0351\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "370/400 [==========================>...] - ETA: 1s - loss: 0.0350\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "375/400 [===========================>..] - ETA: 1s - loss: 0.0350\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "380/400 [===========================>..] - ETA: 1s - loss: 0.0349\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "385/400 [===========================>..] - ETA: 0s - loss: 0.0348\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "390/400 [============================>.] - ETA: 0s - loss: 0.0348\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "395/400 [============================>.] - ETA: 0s - loss: 0.0347\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "399/400 [============================>.] - ETA: 0s - loss: 0.0347\n",
      "Epoch 00005: saving model to ./Checkpoints/checkpoint_model_05.hdf5\n",
      "400/400 [==============================] - 25s 62ms/step - loss: 0.0347\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x14d7caac8>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(np.array(X), np.array(y), 32, 5, verbose=1, callbacks=[model_save_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make a prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model.predict(np.array(X_test))\n",
    "prediction = np.squeeze(prediction)\n",
    "prediction = np.squeeze(scaler.inverse_transform(prediction.reshape(-1, 1)))\n",
    "prediction = [int(i) for i in prediction]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save your result to new midi file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "mid = MidiFile()\n",
    "track = MidiTrack()\n",
    "t = 0\n",
    "for note in prediction:\n",
    "    msg_on = Message.from_dict({'type': 'note_on', 'channel': 0, 'note': note, 'velocity': 67, 'time':0})\n",
    "    # you need to add some pauses \"note_off\"\n",
    "    msg_off = Message.from_dict({'type': 'note_off', 'channel': 0, 'note': note, 'velocity': 67, 'time':64})\n",
    "    track.append(msg_on)\n",
    "    track.append(msg_off)\n",
    "    track.append(msg_off)\n",
    "mid.tracks.append(track)\n",
    "mid.save(\"LSTM_music.mid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Just listen to it. The result is surreal!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}