exercise12_MusicGenreClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bdece447-b974-4cd6-bb10-b67977170f4c",
   "metadata": {},
   "source": [
    "Sascha Spors,\n",
    "Professorship Signal Theory and Digital Signal Processing,\n",
    "Institute of Communications Engineering (INT),\n",
    "Faculty of Computer Science and Electrical Engineering (IEF),\n",
    "University of Rostock,\n",
    "Germany\n",
    "\n",
    "# Data Driven Audio Signal Processing - A Tutorial with Computational Examples\n",
    "\n",
    "Winter Semester 2023/24 (Master Course #24512)\n",
    "\n",
    "- lecture: https://github.com/spatialaudio/data-driven-audio-signal-processing-lecture\n",
    "- tutorial: https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise\n",
    "\n",
    "Feel free to contact lecturer frank.schultz@uni-rostock.de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46602eb-0171-412d-990c-a0a63f4d47f9",
   "metadata": {},
   "source": [
    "# Music Genre Classification with Fully Connected Layers\n",
    "\n",
    "In this toy example we have a special look at\n",
    "- feature design (loudness, crest, peak, rms, spectral weight)\n",
    "- feature inspection / avoiding NaNs\n",
    "- feature normalization\n",
    "- balancing data set wrt class occurrence\n",
    "- one hot encoding\n",
    "- hyper parameter tuning using train / val data set\n",
    "- training of best model with re-set weights using train / val data set\n",
    "- final prediction on unseen test data set compared to predictions on train / val data sets\n",
    "- confusion matrix and visualization of predictions\n",
    "for a music genre classification application.\n",
    "\n",
    "We put some mp3 into `audiofolder = './audio_ex12/'`. The end of the mp3 filename encodes the label `_i.mp3` where\n",
    "- `i=0` Metal\n",
    "- `i=1` EDM\n",
    "- `i=2` Classical,\n",
    "\n",
    "or we use other nice genres. We make sure that we have about the same playing length of all genres/labels such that we can fairly train the model.\n",
    "\n",
    "Instead of feature extraction of raw audio data, we might also use `_raw_data_large.npz` or `_raw_data_small.npz` data , then we do not run the `feature extraction` cell.\n",
    "`_raw_data_large.npz` originates from reading 66 mp3 files with 7h 8 min playtime, with balanced occurrence of the 3 genres.\n",
    "`_raw_data_small.npz` originates from reading 4 mp3 files with about 22 min playtime, with balanced occurrence of the 3 genres.\n",
    "Therefore, the amount of data and sampling of music should be considered as toy data, but we can reasonably play around with the machine learning procedure achieving about 80% (small) / 84% (large) accuracy.\n",
    "To achieve very high accuracies >95% much more data and increased computational power as well as more sophisticated feature design is needed.\n",
    "We should try this at home and read corresponding scientific literature, how people solved this for professional applications.\n",
    "\n",
    "We make sure that in section `Data Handling / Inspection / Selection / Visualization` the intended `*.npz` data file is loaded by `with np.load(audiofolder+'/_raw_data_small.npz') as data:`. Default uses the small data set for reasonable computing time. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53c8f4c-22f0-46a4-ae4e-168cfc1e6615",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21feb9ba-458b-40b8-bc33-a0b677008efe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras_tuner as kt\n",
    "import librosa\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import pyloudnorm\n",
    "from scipy.signal import resample\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import OneHotEncoder, LabelBinarizer\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import time\n",
    "\n",
    "\n",
    "print(\n",
    "    \"TF version\",\n",
    "    tf.__version__,\n",
    "    \"\\nKeras Tuner version\",\n",
    "    kt.__version__,\n",
    ")\n",
    "verbose = 1  # plot training status\n",
    "\n",
    "CI_flag = True  # use toy parameters to check if this notebooks runs in CI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41669170-f69b-43e4-acc8-8ab001d90404",
   "metadata": {},
   "source": [
    "## Folder Structure for Log Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a249164f-10e7-480a-a748-61eab77e7896",
   "metadata": {},
   "outputs": [],
   "source": [
    "audiofolder = \"./audio_ex12/\"\n",
    "ex_str = \"mgc_\"\n",
    "time_str = \"%Y_%m_%d_%H_%M_\"\n",
    "\n",
    "\n",
    "def get_kt_logdir():\n",
    "    run_id = time.strftime(time_str + ex_str + \"kt\")\n",
    "    return os.path.join(root_logdir, run_id)\n",
    "\n",
    "\n",
    "def get_tf_kt_logdir():\n",
    "    run_id = time.strftime(time_str + ex_str + \"tf_kt\")\n",
    "    return os.path.join(root_logdir, run_id)\n",
    "\n",
    "\n",
    "def get_tf_logdir():\n",
    "    run_id = time.strftime(time_str + ex_str + \"tf\")\n",
    "    return os.path.join(root_logdir, run_id)\n",
    "\n",
    "\n",
    "root_logdir = os.path.join(os.curdir, \"tf_keras_logs\")\n",
    "kt_logdir = get_kt_logdir()\n",
    "tf_kt_logdir = get_tf_kt_logdir()\n",
    "tf_logdir = get_tf_logdir()\n",
    "print(root_logdir)\n",
    "print(kt_logdir)  # folder for keras tuner results\n",
    "print(tf_kt_logdir)  # folder for TF checkpoints while keras tuning\n",
    "print(tf_logdir)  # folder for TF checkpoint for best model training\n",
    "\n",
    "os.makedirs(tf_logdir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22c7522a-50c2-449c-b900-0677a98ce8f4",
   "metadata": {},
   "source": [
    "## Feature Extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "ed5b4e1e-c6fe-4363-80bc-919558d6e06b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Labels:\n",
    "# 0 Metal\n",
    "# 1 EDM\n",
    "# 2 Classical\n",
    "\n",
    "N_F = 12  # number of features, must match dim of np array 'features'\n",
    "t = 0.5  # s  # 1s long blocks\n",
    "\n",
    "N_OS = 4  # oversampling factor for true peak\n",
    "# final dimension is (data entry, features):\n",
    "Xdata = np.zeros((1, N_F))\n",
    "# final dimension is (data entry, 1), we encode labels as integers >=0:\n",
    "Ydata = np.zeros((1, 1), dtype=int)\n",
    "files = sorted(os.listdir(audiofolder))  # get file names in this folder\n",
    "for i in files:\n",
    "    if i[-4:] == \".mp3\":  # consider only mp3 files\n",
    "        # make sure to check defaults, they might be not convenient for us:\n",
    "        x, fs = librosa.load(\n",
    "            audiofolder + i, mono=False, sr=None, dtype=\"double\"\n",
    "        )\n",
    "        x = x.T  # we use more convenient dim: (samples, channels)\n",
    "        label = i[-6:]  # get '_i.mp3' with i={0,1,2,...} as genre label\n",
    "        label = int(label[1])\n",
    "        print(\"song\", i, \"label\", label)\n",
    "        meter = pyloudnorm.Meter(fs)  # init 1770 meter\n",
    "        N = int(t * fs)  # in samples\n",
    "        df = fs / N  # fft frequency resolution\n",
    "        fsplit = 1000  # Hz, split frequency between low / high frequency band\n",
    "        Nsplit = int(fsplit / df) + 1\n",
    "        Nb = x.shape[0] // N  # number of blocks in the file\n",
    "        print(Nb)\n",
    "        for ch in range(x.shape[1]):  # use all channels\n",
    "            k = 0  # clr accu, move through samples\n",
    "            # print(k, k+N)\n",
    "            for n in range(Nb):  # go through blocks\n",
    "                tmp = x[k : k + N, ch]  # get signal block\n",
    "                k += N  # hop\n",
    "                # print(tmp.shape[0])\n",
    "                if (\n",
    "                    np.mean(tmp**2) > (10 ** (-80 / 20)) ** 2\n",
    "                ):  # use simple silence detection\n",
    "                    tmp_os = resample(tmp, N_OS * tmp.shape[0])\n",
    "\n",
    "                    # calc potential (here rather simple) features\n",
    "                    true_peak_lin = np.max(np.abs(tmp_os))\n",
    "                    true_peak_lin2 = true_peak_lin**2\n",
    "                    true_peak_db = 10 * np.log10(true_peak_lin2)\n",
    "\n",
    "                    rms_lin2 = np.mean(tmp**2)\n",
    "                    rms_lin = np.sqrt(rms_lin2)\n",
    "                    rms_db = 10 * np.log10(rms_lin2)\n",
    "\n",
    "                    lufs_db = meter.integrated_loudness(tmp)\n",
    "                    lufs_lin2 = 10 ** (lufs_db / 10)\n",
    "                    lufs_lin = np.sqrt(lufs_lin2)\n",
    "\n",
    "                    crest_lin = true_peak_lin / rms_lin\n",
    "                    crest_db = 20 * np.log10(crest_lin)\n",
    "\n",
    "                    ffttmp = np.fft.fft(tmp)\n",
    "                    # sum squared entries in low frequency band\n",
    "                    Xe_low = np.sum(np.abs(ffttmp[1:Nsplit]) ** 2)  # without DC\n",
    "                    # sum squared entries in high frequency band\n",
    "                    Xe_high = np.sum(\n",
    "                        np.abs(ffttmp[Nsplit : N // 2]) ** 2\n",
    "                    )  # without DC\n",
    "                    low_high_ratio = Xe_low / Xe_high\n",
    "\n",
    "                    if lufs_db < -70:  # avoid NaN\n",
    "                        lufs_db = -70\n",
    "                    # put all features into np:\n",
    "                    features = np.array(\n",
    "                        [\n",
    "                            true_peak_lin,\n",
    "                            true_peak_lin2,\n",
    "                            true_peak_db,\n",
    "                            rms_lin2,\n",
    "                            rms_lin,\n",
    "                            rms_db,\n",
    "                            lufs_lin,\n",
    "                            lufs_lin2,\n",
    "                            lufs_db,\n",
    "                            crest_lin,\n",
    "                            crest_db,\n",
    "                            low_high_ratio,\n",
    "                        ]\n",
    "                    )\n",
    "                    # store features and according label\n",
    "                    Xdata = np.vstack((Xdata, features))\n",
    "                    Ydata = np.vstack((Ydata, label))\n",
    "\n",
    "# del very first entries since these are not valid data\n",
    "# but rather stems from allocating by np.zeros(())\n",
    "Xdata = Xdata[1:, :]\n",
    "Ydata = Ydata[1:, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf4a47f4-0c36-4bdc-a293-6d6d75c15383",
   "metadata": {},
   "source": [
    "## Data Handling / Inspection / Selection / Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "23424375-717e-4ca1-bcf7-9bd4a4fe8af8",
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(tf_logdir):\n",
    "    os.makedirs(tf_logdir)\n",
    "np.savez(audiofolder + \"/_raw_data.npz\", Xdata=Xdata, Ydata=Ydata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "4a58498f-2445-4837-a9c0-95b940fd9086",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we use this option when features were extracted in cell [5] above\n",
    "# model calc time depends on how much data is preprocessed to features\n",
    "with np.load(audiofolder + \"/_raw_data.npz\") as data:\n",
    "    Xdata = data[\"Xdata\"]\n",
    "    Ydata = data[\"Ydata\"]\n",
    "\n",
    "# we use this when we want to use 'large' toy example data set\n",
    "# this might take time, especially for a large hyper param search\n",
    "with np.load(audiofolder + \"/_raw_data_large.npz\") as data:\n",
    "    Xdata = data[\"Xdata\"]\n",
    "    Ydata = data[\"Ydata\"]\n",
    "\n",
    "# we use this when we want to use 'small' toy example data set\n",
    "# this takes <10 min on a Mac Book Pro M1\n",
    "with np.load(audiofolder + \"/_raw_data_small.npz\") as data:\n",
    "    Xdata = data[\"Xdata\"]\n",
    "    Ydata = data[\"Ydata\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "39305f54-e211-4564-853d-28dfc52119c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# permute to make data entries random\n",
    "p = np.random.permutation(Ydata.shape[0])\n",
    "Xdata = Xdata[p, :]\n",
    "Ydata = Ydata[p]\n",
    "# minimal example\n",
    "# Ydata = np.array([[0,1,2,3,4,5,6,7,8,9]]).T\n",
    "# Xdata = np.random.rand(10, 4)\n",
    "# print(Ydata, Ydata.shape)\n",
    "# print(Xdata)\n",
    "# p = np.random.permutation(Ydata.shape[0])\n",
    "# print(p)\n",
    "# print(Ydata[p])\n",
    "# print(Xdata[p,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e25b38e7-e6e7-4a63-8af3-12ce5b0dd47c",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Xdata.shape, Ydata.shape\")\n",
    "print(Xdata.shape, Ydata.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7df85f14-8bfd-4e28-9122-ac93ae25d78f",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlabels = np.max(Ydata) + 1\n",
    "# we encode as integers\n",
    "labels = np.arange(nlabels)\n",
    "print(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2afd247f-c7cb-494e-b835-a6aa1a145798",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we should very carefully inspect our data !\n",
    "# at least check for NaN, Inf\n",
    "print(np.sum(np.isfinite(Xdata)) == Xdata.shape[0] * Xdata.shape[1])\n",
    "print(np.sum(np.isnan(Xdata)))\n",
    "print(np.sum(np.isinf(Xdata)))\n",
    "# we should visualize as much as possible..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e9e812b-621b-4d4e-a095-6a5be6953a9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# check if label occurence is balanced\n",
    "# if not then balance it for fair learning\n",
    "m = Ydata.shape[0]\n",
    "for n in labels:\n",
    "    print(\"label\", n, \"occurence\", np.sum(Ydata == n) / m * 100, \"%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc903325-778e-40d9-b9e4-ce9104a4396e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# normalize all features columns to mean=0, std=1\n",
    "Xdata_norm = Xdata - np.mean(Xdata, axis=0)\n",
    "Xdata_norm = Xdata_norm / np.std(Xdata_norm, ddof=1, axis=0)\n",
    "print(\"\\nmean\\n\", np.mean(Xdata_norm, axis=0))\n",
    "print(\"\\nstd\\n\", np.std(Xdata_norm, axis=0, ddof=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d8d343b-c04c-401d-8267-9675917c2109",
   "metadata": {},
   "outputs": [],
   "source": [
    "#  0 true_peak_lin\n",
    "#  1 true_peak_lin2\n",
    "#  2 true_peak_db\n",
    "#  3 rms_lin2\n",
    "#  4 rms_lin\n",
    "#  5 rms_db\n",
    "#  6 lufs_lin\n",
    "#  7 lufs_lin2\n",
    "#  8 lufs_db\n",
    "#  9 crest_lin\n",
    "# 10 crest_db\n",
    "# 11 low_high_ratio\n",
    "# the seven features [2, 3, 5, 7, 8, 10, 11] might be useful:\n",
    "which_features = [2, 3, 5, 7, 8, 10, 11]\n",
    "X = np.copy(Xdata_norm[:, which_features])\n",
    "Y = np.copy(Ydata)\n",
    "m = X.shape[0]  # number data examples\n",
    "nx = X.shape[1]  # number of features\n",
    "print(\"X.shape, Y.shape\")\n",
    "print(X.shape, Y.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43ffc210-756a-4bec-8358-f6c2f46943ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 4))\n",
    "for f in range(nx):\n",
    "    plt.plot(X[:, f], label=\"feature \" + str(f))\n",
    "plt.xlabel(\"data entry index\")\n",
    "plt.ylabel(\"feature extent\")\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "286c31e3-128a-4c87-ab0b-70adae6a8fdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(12, 9))\n",
    "for sp in range(nlabels):\n",
    "    plt.subplot(nlabels, 1, sp + 1)\n",
    "    for f in range(nx):\n",
    "        plt.plot(X[Y[:, 0] == sp, f])\n",
    "        plt.ylabel(\"feature extent for label \" + str(sp))\n",
    "plt.xlabel(\"data entry index\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e65e1f55-f334-4623-9469-2d4380e422fe",
   "metadata": {},
   "source": [
    "## Data Preparation / Splitting "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "49cb1e2f-da96-4b43-8490-f5b96d0c60cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = OneHotEncoder(sparse_output=False)\n",
    "# we encode as one-hot for TF model\n",
    "Y = encoder.fit_transform(Y.reshape(-1, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5d24d3a-c5cd-47eb-9444-4b0008b1cfc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_size = 5 / 10\n",
    "validate_size = 5 / 10 * 1 / 2\n",
    "test_size = 1 - train_size - validate_size  # remaining data, must be > 0\n",
    "\n",
    "# split into train, val, test data:\n",
    "X_train, X_tmp, Y_train, Y_tmp = train_test_split(\n",
    "    X, Y, train_size=train_size, random_state=None\n",
    ")\n",
    "val_size = (validate_size * m) / ((1 - train_size) * m)\n",
    "X_val, X_test, Y_val, Y_test = train_test_split(\n",
    "    X_tmp, Y_tmp, train_size=val_size, random_state=None\n",
    ")\n",
    "\n",
    "print(train_size, validate_size, test_size)\n",
    "print(train_size * m, validate_size * m, test_size * m)\n",
    "print(X_train.shape, X_val.shape, X_test.shape)\n",
    "print(Y_train.shape, Y_val.shape, Y_test.shape)\n",
    "\n",
    "m_train = X_train.shape[0]\n",
    "m_val = X_val.shape[0]\n",
    "m_test = X_test.shape[0]\n",
    "print(m_train, m_val, m_test, m_train + m_val + m_test == m)\n",
    "\n",
    "# we should get balanced occurence\n",
    "print(\"occurence of labels in train\")\n",
    "for n in range(nlabels):\n",
    "    print(n, np.sum(Y_train[:, n]))\n",
    "print(\"occurence of labels in val\")\n",
    "for n in range(nlabels):\n",
    "    print(n, np.sum(Y_val[:, n]))\n",
    "print(\"occurence of labels in test\")\n",
    "for n in range(nlabels):\n",
    "    print(n, np.sum(Y_test[:, n]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16945e0c-91f6-4d72-93c3-7c3394814fdf",
   "metadata": {},
   "source": [
    "## Model Preparation / Hyper Parameter Range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "dbf6b6d1-dfb7-47ed-a723-967f2d1f52dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "earlystopping_cb = keras.callbacks.EarlyStopping(\n",
    "    monitor=\"val_loss\", patience=2, restore_best_weights=True  # on val data!\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "dabce6d1-009d-470f-8407-788b7478ab70",
   "metadata": {},
   "outputs": [],
   "source": [
    "# as homework we might also consider dropout / regularization in the model\n",
    "def build_model(hp):  # with hyper parameter ranges\n",
    "    model = keras.Sequential()\n",
    "    # input layer\n",
    "    model.add(keras.Input(shape=(nx,)))\n",
    "    # hidden layers\n",
    "    for layer in range(hp.Int(\"no_layers\", 1, 5)):\n",
    "        model.add(\n",
    "            keras.layers.Dense(\n",
    "                units=hp.Int(\n",
    "                    f\"no_perceptrons_{layer}\", min_value=2, max_value=16, step=2\n",
    "                ),\n",
    "                activation=hp.Choice(\"activation\", [\"tanh\"]),\n",
    "            )\n",
    "        )\n",
    "    # softmax output layer\n",
    "    model.add(keras.layers.Dense(nlabels, activation=\"softmax\"))\n",
    "    # learning_rate = hp.Float('learning_rate', min_value=1e-5, max_value=1e-1,\n",
    "    #                         sampling='log')\n",
    "    model.compile(\n",
    "        optimizer=keras.optimizers.Adam(),  # learning_rate=learning_rate\n",
    "        loss=keras.losses.CategoricalCrossentropy(\n",
    "            from_logits=False, label_smoothing=0\n",
    "        ),\n",
    "        metrics=[\"CategoricalCrossentropy\", \"CategoricalAccuracy\"],\n",
    "    )\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f708fe9-abb1-4343-9c71-3c2ae10599c9",
   "metadata": {},
   "source": [
    "## Hyper Parameter Tuner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "281a2b25-36a1-4c1e-81e2-6bce058dabea",
   "metadata": {},
   "outputs": [],
   "source": [
    "if CI_flag:\n",
    "    max_trials = 5  # very low for CI version\n",
    "else:\n",
    "    max_trials = 50\n",
    "executions_per_trial = 2\n",
    "model = build_model(kt.HyperParameters())\n",
    "hptuner = kt.RandomSearch(\n",
    "    hypermodel=build_model,\n",
    "    objective=\"val_loss\",  # check performance on val data!\n",
    "    max_trials=max_trials,\n",
    "    executions_per_trial=executions_per_trial,\n",
    "    overwrite=True,\n",
    "    directory=kt_logdir,\n",
    "    project_name=None,\n",
    ")\n",
    "print(hptuner.search_space_summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44d8fb98-7797-4673-b849-8074cd95c530",
   "metadata": {},
   "source": [
    "## Training of Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40265786-0952-470e-9d55-c4721e791d23",
   "metadata": {},
   "outputs": [],
   "source": [
    "if CI_flag:\n",
    "    epochs = 20  # very low for CI version\n",
    "else:\n",
    "    epochs = 250\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(tf_kt_logdir)\n",
    "hptuner.search(\n",
    "    X_train,\n",
    "    Y_train,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    epochs=epochs,\n",
    "    callbacks=[earlystopping_cb, tensorboard_cb],\n",
    "    verbose=verbose,\n",
    ")\n",
    "print(hptuner.results_summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8081f7ac-bb3e-4d06-8165-80cd1cbae963",
   "metadata": {},
   "source": [
    "## Best Model Selection / Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d003b28a-595c-41fb-9909-f709bed5a812",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we might check (train) the best XX models in detail\n",
    "# for didactical purpose we choose only the very best one, located in [0]:\n",
    "model = hptuner.get_best_models(num_models=1)[0]\n",
    "model.save(tf_logdir + \"/best_model.keras\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "4f27ff18-5dd4-42fa-86a1-60926c9b5993",
   "metadata": {},
   "outputs": [],
   "source": [
    "# taken from https://github.com/keras-team/keras/issues/341\n",
    "# 183amir commented on 7 Oct 2019:\n",
    "# \"If you are using tensorflow 2, you can use this:\"\n",
    "def reset_weights(model):\n",
    "    for layer in model.layers:\n",
    "        if isinstance(layer, tf.keras.Model):\n",
    "            reset_weights(layer)\n",
    "            continue\n",
    "        for k, initializer in layer.__dict__.items():\n",
    "            if \"initializer\" not in k:\n",
    "                continue\n",
    "            # find the corresponding variable\n",
    "            var = getattr(layer, k.replace(\"_initializer\", \"\"))\n",
    "            var.assign(initializer(var.shape, var.dtype))\n",
    "\n",
    "\n",
    "# 183amir: \"I am not sure if it works in all cases, I have only tested the Dense and Conv2D layers.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "958db756-ae6a-446e-bfd7-933ccab79d30",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load best model and reset weights\n",
    "model = keras.models.load_model(tf_logdir + \"/best_model.keras\")\n",
    "reset_weights(model)  # start training from scratch\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de203e3d-bf01-4f1b-8115-8f7c5f0e8c2c",
   "metadata": {},
   "source": [
    "## Training of Best Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fd86f4b-f780-4c4c-a202-4f6f5af4a108",
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 16\n",
    "if CI_flag:\n",
    "    epochs = 20  # very low for CI version\n",
    "else:\n",
    "    epochs = 250\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(tf_logdir)\n",
    "history = model.fit(\n",
    "    X_train,\n",
    "    Y_train,\n",
    "    epochs=epochs,\n",
    "    batch_size=batch_size,\n",
    "    validation_data=(X_val, Y_val),\n",
    "    callbacks=[earlystopping_cb, tensorboard_cb],\n",
    "    verbose=1,\n",
    ")\n",
    "model.save(tf_logdir + \"/trained_best_model.keras\")\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7c926e7-c181-4111-9635-c6c4a1ca2709",
   "metadata": {},
   "source": [
    "## Evaluation of Best Model on Unseen Test Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dfb48e2c-f068-4b35-afdd-5d9654a5efb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_results(X, Y):\n",
    "    # https://stackoverflow.com/questions/48908641/how-to-get-a-single-value-from-softmax-instead-of-probability-get-confusion-ma:\n",
    "    lb = LabelBinarizer()\n",
    "    lb.fit(labels)\n",
    "\n",
    "    m = X.shape[0]\n",
    "    results = model.evaluate(X, Y, batch_size=m, verbose=verbose)\n",
    "    Y_pred = model.predict(X)\n",
    "    cm = tf.math.confusion_matrix(\n",
    "        labels=lb.inverse_transform(Y),\n",
    "        predictions=lb.inverse_transform(Y_pred),\n",
    "        num_classes=nlabels,\n",
    "    )\n",
    "    print(\"data entries\", m)\n",
    "    print(\n",
    "        \"Cost\",\n",
    "        results[0],\n",
    "        \"\\nCategoricalCrossentropy\",\n",
    "        results[1],\n",
    "        \"\\nCategoricalAccuracy\",\n",
    "        results[2],\n",
    "    )\n",
    "    print(\n",
    "        \"nCategoricalAccuracy from Confusion Matrix = \",\n",
    "        np.sum(np.diag(cm.numpy())) / m,\n",
    "    )\n",
    "    print(\"Confusion Matrix in %\\n\", cm / m * 100)\n",
    "\n",
    "\n",
    "print(\"\\n\\nmetrics on train data:\")\n",
    "print_results(X_train, Y_train)\n",
    "\n",
    "print(\"\\n\\nmetrics on val data:\")\n",
    "print_results(X_val, Y_val)\n",
    "\n",
    "print(\"\\n\\nmetrics on never seen test data:\")\n",
    "print_results(X_test, Y_test)\n",
    "# we never used X_test, Y_test in an above training steps!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cad6d07-4ce0-4fb8-bcdd-93b7d14a8fc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "Y_pred = model.predict(X_test)\n",
    "\n",
    "k0 = 0\n",
    "kN = k0 + 100\n",
    "\n",
    "plt.figure(figsize=(12, 4))\n",
    "plt.plot(Y_test[k0:kN, 0], \"C0o\", label=\"True 0...Metal\")\n",
    "plt.plot(Y_test[k0:kN, 1], \"C1o\", label=\"True 1...EDM\")\n",
    "plt.plot(Y_test[k0:kN, 2], \"C2o\", label=\"True 2...Classical\")\n",
    "\n",
    "plt.plot(Y_pred[k0:kN, 0], \"C0\", lw=1)\n",
    "plt.plot(Y_pred[k0:kN, 1], \"C1\", lw=1)\n",
    "plt.plot(Y_pred[k0:kN, 2], \"C2\", lw=1)\n",
    "\n",
    "tmp = np.argmax(Y_pred, axis=1)\n",
    "tmp = tmp[k0:kN]\n",
    "plt.plot((tmp == 0) + 0.03, \"C0v\", label=\"Predicted 0...Metal\")\n",
    "plt.plot((tmp == 1) + 0.03, \"C1v\", label=\"Predicted 1...EDM\")\n",
    "plt.plot((tmp == 2) + 0.03, \"C2v\", label=\"Predicted 2...Classical\")\n",
    "\n",
    "plt.ylim(0.06, 1.075)\n",
    "plt.xlabel(\"index for chosen data entries\")\n",
    "plt.ylabel(\"predicted via softmax\")\n",
    "plt.legend(loc=\"center\")\n",
    "plt.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adc4b816-4c61-42e4-8f46-7d0259e78c02",
   "metadata": {},
   "source": [
    "## Copyright\n",
    "\n",
    "- the notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources)\n",
    "- the text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/)\n",
    "- the code of the IPython examples is licensed under the [MIT license](https://opensource.org/licenses/MIT)\n",
    "- feel free to use the notebooks for your own purposes\n",
    "- please attribute the work as follows: *Frank Schultz, Data Driven Audio Signal Processing - A Tutorial Featuring Computational Examples, University of Rostock* ideally with relevant file(s), github URL https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise, commit number and/or version tag, year."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myddasp",
   "language": "python",
   "name": "myddasp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}