binary_logistic_regression_tf_with_hidden_layers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ff5888c",
   "metadata": {},
   "source": [
    "Sascha Spors,\n",
    "Professorship Signal Theory and Digital Signal Processing,\n",
    "Institute of Communications Engineering (INT),\n",
    "Faculty of Computer Science and Electrical Engineering (IEF),\n",
    "University of Rostock,\n",
    "Germany\n",
    "\n",
    "# Data Driven Audio Signal Processing - A Tutorial with Computational Examples\n",
    "\n",
    "Winter Semester 2023/24 (Master Course #24512)\n",
    "\n",
    "- lecture: https://github.com/spatialaudio/data-driven-audio-signal-processing-lecture\n",
    "- tutorial: https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise\n",
    "\n",
    "Feel free to contact lecturer frank.schultz@uni-rostock.de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a199b3df",
   "metadata": {},
   "source": [
    "# Binary logistic regression model with hidden layers and a sigmoid output layer\n",
    "\n",
    "- we use TensorFlow & Keras API and scikit learn function to create and handle data sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "80a46046",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "\n",
    "print(\n",
    "    \"TF version\",\n",
    "    tf.__version__,\n",
    ")\n",
    "\n",
    "# tf.keras.backend.set_floatx('float64')  # we could use double precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bfaa826",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_class(y):\n",
    "    y[y < 0.5], y[y >= 0.5] = 0, 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d77047c",
   "metadata": {},
   "outputs": [],
   "source": [
    "verbose = 1  # plot training status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e8b4800",
   "metadata": {},
   "outputs": [],
   "source": [
    "# DATA\n",
    "M = int(5 / 4 * 80000)  # number of samples per feature\n",
    "N = 2  # number of features\n",
    "\n",
    "train_size = 4 / 5  # 80% of data are used for training\n",
    "\n",
    "# these seeds produce 'nice' two classes each with\n",
    "# two clusters for chosen M, N and train_size\n",
    "random_state_idx = 0\n",
    "random_state = np.array([7, 21, 24, 25, 29, 33, 38])\n",
    "X, Y = make_classification(\n",
    "    n_samples=M,\n",
    "    n_features=N,\n",
    "    n_informative=N,\n",
    "    n_redundant=0,\n",
    "    n_classes=2,\n",
    "    n_clusters_per_class=2,\n",
    "    class_sep=1.5,\n",
    "    flip_y=1e-2,\n",
    "    random_state=random_state[random_state_idx],\n",
    ")\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(\n",
    "    X, Y, train_size=train_size, random_state=None\n",
    ")\n",
    "M_train = X_train.shape[0]\n",
    "M_test = X_test.shape[0]\n",
    "print(\"\\nM_train\", M_train)\n",
    "print(\"X train dim\", X_train.shape, \"Y train dim\", Y_train.shape)\n",
    "print(\"\\nM_test\", M_test)\n",
    "print(\"X test dim\", X_test.shape, \"Y test dim\", Y_test.shape, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b23f6d44",
   "metadata": {},
   "source": [
    "### Setup of Tensor Flow Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e73e768",
   "metadata": {},
   "source": [
    "- hyper parameters\n",
    "- in practice we do hyper parameter tuning, see upcoming exercises"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c792c68",
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 2**2\n",
    "batch_size = 2**3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d32b7d1a",
   "metadata": {},
   "source": [
    "- define model architecture based on fully connected layers\n",
    "- in practice number and dimension of hidden layers should be hyper parameters to be learned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95161d97",
   "metadata": {},
   "outputs": [],
   "source": [
    "# some made up models, trainable params for N=2\n",
    "# feel free to invent a new, optimum model for this classification example\n",
    "\n",
    "# models too complex?! :\n",
    "# no_perceptron_in_hl = np.array([64, 64])  # trainable params 4417\n",
    "# no_perceptron_in_hl = np.array([64, 32, 16, 8, 4, 2])  # trainable params 2985\n",
    "# no_perceptron_in_hl = np.array([64, 16, 4, 16, 64])  # trainable params 2533\n",
    "# no_perceptron_in_hl = np.array([64, 4, 2, 4, 64])  # trainable params 859\n",
    "# no_perceptron_in_hl = np.array([32, 16, 8, 4, 2])  # trainable params 809\n",
    "# no_perceptron_in_hl = np.array([16, 16, 4, 2])  # trainable params 401\n",
    "# no_perceptron_in_hl = np.array([16, 8, 4, 2])  # trainable params 233\n",
    "# no_perceptron_in_hl = np.array([8, 8, 4, 2])  # trainable params 145\n",
    "# no_perceptron_in_hl = np.array([5, 5, 5])  # trainable params 81\n",
    "# model complexity reasonable?! :\n",
    "# no_perceptron_in_hl = np.array([5, 3, 2])  # trainable params 44\n",
    "# no_perceptron_in_hl = np.array([5, 4])  # trainable params 44\n",
    "# no_perceptron_in_hl = np.array([5, 3])  # trainable params 37\n",
    "# no_perceptron_in_hl = np.array([8])  # trainable params 33\n",
    "# no_perceptron_in_hl = np.array([5, 2])  # trainable params 30\n",
    "# no_perceptron_in_hl = np.array([5])  # trainable params 21\n",
    "# no_perceptron_in_hl = np.array([3, 2])  # trainable params 20\n",
    "no_perceptron_in_hl = np.array([2, 2])  # trainable params 15\n",
    "# model too simple?! :\n",
    "# no_perceptron_in_hl = np.array([2])  # trainable params 9->train this longer. i.e. with more epochs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6e3f633",
   "metadata": {},
   "source": [
    "- define and compile model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "842503d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = keras.optimizers.SGD()\n",
    "\n",
    "loss = keras.losses.BinaryCrossentropy(from_logits=False, label_smoothing=0)\n",
    "\n",
    "metrics = [\n",
    "    keras.metrics.BinaryCrossentropy(),\n",
    "    keras.metrics.BinaryAccuracy(),\n",
    "    keras.metrics.Precision(),  # PPV (FP related)\n",
    "    keras.metrics.Recall(),  # TPR (FN related)\n",
    "]\n",
    "\n",
    "model = keras.Sequential()\n",
    "\n",
    "model.add(keras.Input(shape=(N,)))  # input layer\n",
    "\n",
    "for n in no_perceptron_in_hl:  # hidden layers (fully connected=dense)\n",
    "    model.add(\n",
    "        keras.layers.Dense(n, activation=\"tanh\")\n",
    "    )  # relu vs tanh makes a big difference\n",
    "\n",
    "model.add(\n",
    "    keras.layers.Dense(\n",
    "        1,  # output layer, sigmoid for binary classification\n",
    "        activation=\"sigmoid\",\n",
    "    )\n",
    ")\n",
    "\n",
    "model.compile(optimizer=optimizer, loss=loss, metrics=metrics)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f4e4f44",
   "metadata": {},
   "source": [
    "### Train / Fit the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d50b98",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(\n",
    "    X_train, Y_train[:, None], epochs=epochs, batch_size=batch_size, verbose=verbose\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd901f4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(model.summary())\n",
    "print(\"model weights\\n\", model.get_weights())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9073cba9",
   "metadata": {},
   "source": [
    "### Performance Measures: Fitted Model on Training Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a59cc9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.evaluate(X_train, Y_train[:, None], batch_size=M_train, verbose=False)\n",
    "Y_train_pred = model.predict(X_train)\n",
    "predict_class(Y_train_pred[:, None])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8234f199",
   "metadata": {},
   "outputs": [],
   "source": [
    "cost = results[0]\n",
    "accuracy = results[2]\n",
    "precision = results[3]\n",
    "recall = results[4]\n",
    "F1_score = 2 / (1 / precision + 1 / recall)  # harmonic mean\n",
    "cm = tf.math.confusion_matrix(\n",
    "    labels=Y_train, predictions=Y_train_pred, num_classes=2\n",
    ")\n",
    "\n",
    "print(\"binary_crossentropy cost \", cost)\n",
    "print(\"precision / PPV (FP related)\", precision)\n",
    "print(\"recall / TPR (FN related)\", recall)\n",
    "print(\"accuracy\", accuracy)\n",
    "print(\n",
    "    \"our accuracy:\",\n",
    "    (cm.numpy()[0, 0] + cm.numpy()[1, 1]) / M_train * 100,\n",
    "    \"% are correct predictions\",\n",
    ")\n",
    "print(\"F1\", F1_score)\n",
    "print(\n",
    "    \"\\nconfusion matrix\\nreal0,pred0  real0,pred1\\nreal1,pred0  real1,pred1\\nin % on train data:\"\n",
    ")\n",
    "print(cm / M_train * 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21166f87",
   "metadata": {},
   "source": [
    "### Test the Model\n",
    "\n",
    "- we check model performance on **unseen** test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55cf758a",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = model.evaluate(X_test, Y_test[:,None], batch_size=M_test, verbose=False)\n",
    "Y_test_pred = model.predict(X_test)\n",
    "predict_class(Y_test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e63a6abe",
   "metadata": {},
   "source": [
    "### Performance Measures: Fitted Model on Test Data Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c834a8b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "cost = results[0]\n",
    "accuracy = results[2]\n",
    "precision = results[3]\n",
    "recall = results[4]\n",
    "F1_score = 2 / (1 / precision + 1 / recall)  # harmonic mean\n",
    "cm = tf.math.confusion_matrix(\n",
    "    labels=Y_test, predictions=Y_test_pred, num_classes=2\n",
    ")\n",
    "\n",
    "print(\"binary_crossentropy cost \", cost)\n",
    "print(\"precision / PPV (FP related)\", precision)\n",
    "print(\"recall / TPR (FN related)\", recall)\n",
    "print(\"accuracy\", accuracy)\n",
    "print(\n",
    "    \"our accuracy:\",\n",
    "    (cm.numpy()[0, 0] + cm.numpy()[1, 1]) / M_test * 100,\n",
    "    \"% are correct predictions\",\n",
    ")\n",
    "print(\"F1\", F1_score)\n",
    "print(\n",
    "    \"\\nconfusion matrix\\nreal0,pred0  real0,pred1\\nreal1,pred0  real1,pred1\\nin % on test data:\"\n",
    ")\n",
    "print(cm / M_test * 100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14fa82ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "if N == 2:  # 2D plot of data and classification (curved) line\n",
    "    f1, f2 = np.arange(-6, 6, 0.05), np.arange(-6, 6, 0.05)\n",
    "    xv, yv = np.meshgrid(f1, f2)\n",
    "    # create data such that TF can handle it in model.predict():\n",
    "    Xgrid = np.concatenate(\n",
    "        (np.reshape(xv, (1, -1)), np.reshape(yv, (1, -1))), axis=0\n",
    "    ).T\n",
    "\n",
    "    ygrid = model.predict(Xgrid)\n",
    "    predict_class(ygrid)\n",
    "    ygrid = np.reshape(ygrid, (xv.shape[0], xv.shape[1]))\n",
    "\n",
    "    plt.figure(figsize=(12, 5))\n",
    "\n",
    "    plt.subplot(1, 2, 1)  # left plot for training data set\n",
    "    plt.plot(X_train[Y_train == 0, 0], X_train[Y_train == 0, 1], \"C0o\", ms=1)\n",
    "    plt.plot(X_train[Y_train == 1, 0], X_train[Y_train == 1, 1], \"C1o\", ms=1)\n",
    "    plt.contourf(f1, f2, ygrid, cmap=\"RdBu_r\")\n",
    "    plt.colorbar()\n",
    "    plt.axis(\"equal\")\n",
    "    plt.xlim(-6, 6)\n",
    "    plt.ylim(-6, 6)\n",
    "    plt.title(\"training: \" + str(X_train.shape))\n",
    "    plt.xlabel(\"feature 1\")\n",
    "    plt.ylabel(\"feature 2\")\n",
    "\n",
    "    plt.subplot(1, 2, 2)  # right plot for test data set\n",
    "    plt.plot(X_test[Y_test == 0, 0], X_test[Y_test == 0, 1], \"C0o\", ms=1)\n",
    "    plt.plot(X_test[Y_test == 1, 0], X_test[Y_test == 1, 1], \"C1o\", ms=1)\n",
    "    plt.contourf(f1, f2, ygrid, cmap=\"RdBu_r\")\n",
    "    plt.colorbar()\n",
    "    plt.axis(\"equal\")\n",
    "    plt.xlim(-6, 6)\n",
    "    plt.ylim(-6, 6)\n",
    "    plt.title(\"test: \" + str(X_test.shape))\n",
    "    plt.xlabel(\"feature 1\")\n",
    "    plt.ylabel(\"feature 2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0dac86a2",
   "metadata": {},
   "source": [
    "Nice to do at home:\n",
    "- instead of using `activation='tanh'` in the dense layers, we could experience that `activation='relu'` yields a more piece-wise linear classification boundary line.\n",
    "- have a look at the classification boundary line and guess how many coefficients a polynomial or spline curve would need to create such a curve. A good model should exhibit about same parameter number to create this classification curve...thousands of model parameters for this example is too much. That's why mathematical thinking rather than just playing around helps a lot for such tasks.\n",
    "- for more than two features `N` we cannot conveniently plot the data sets and boundary line anymore. Hence, instead of having visual contact to the data and classification, we heavily rely on the performances measures...we make sure that we fully understand the given numbers.\n",
    "- How to choose the best number of features? How to find the best model? When is a model fully trained?...That are important questions for real applications. We soon learn about hyper parameter tuning, regularization...  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "651b1eff",
   "metadata": {},
   "source": [
    "## Copyright\n",
    "\n",
    "- the notebooks are provided as [Open Educational Resources](https://en.wikipedia.org/wiki/Open_educational_resources)\n",
    "- feel free to use the notebooks for your own purposes\n",
    "- the text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/)\n",
    "- the code of the IPython examples is licensed under the [MIT license](https://opensource.org/licenses/MIT)\n",
    "- please attribute the work as follows: *Frank Schultz, Data Driven Audio Signal Processing - A Tutorial Featuring Computational Examples, University of Rostock* ideally with relevant file(s), github URL https://github.com/spatialaudio/data-driven-audio-signal-processing-exercise, commit number and/or version tag, year."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "myddasp",
   "language": "python",
   "name": "myddasp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}