Merge pull request #15 from DerikVo/readme

add more commentary to notebooks

Author: DerikVo

Created_images/Neural Network confusion matrix.png

@@ -0,0 +1 @@
+This folder is used to store any models being trained. When you fork this project all models will be stored here. Please ensure you have the models(.h5 files) added to your git ignore to avoid any git issues. These models are often above the file limit for github.

Created_images/Neural Network with regularization confusion matrix.png

@@ -30,8 +30,25 @@
   },
   {
    "cell_type": "markdown",
-   "id": "fbc628f5-2cfd-4bf1-9cc5-2430c025d8b6",
+   "id": "90b73eda-7137-4ecd-ae15-8ab4cc998182",
    "metadata": {},
+   "source": [
+    "# Exploratory Data Analysis\n",
+    "\n",
+    "In this notebook we explore our classes by taking the average pixel value of each class and comparing them to each other. By taking the average pixel value of each class we can see if there are any distinguishing features between the classes. \n",
+    "\n",
+    "Additionally, we also take the contrast between each class that contain a tumor and subtracted the average of the no tumor class. This way we can see if there are any unique characteristics between the classes.\n",
+    "\n",
+    "The idea of this EDA process was developed during a [previous project](https://github.com/DerikVo/DSI_project_4_plant_disease/blob/main/notebooks/01_Potato_PlantVillageEDA.ipynb). The has been adapted and converted to a [py file](../modules/eda.py) to maintain a cleaner notebook.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "fbc628f5-2cfd-4bf1-9cc5-2430c025d8b6",
+   "metadata": {
+    "jp-MarkdownHeadingCollapsed": true,
+    "tags": []
+   },
    "source": [
     "# Loading in data\n",
     "This portion of the notebook loads in a random image from each class: 'glioma', 'meningioma', 'notumor', 'pituitary' and lists the number of images within each class with in the raining and testing datasets."

Models/README.md

@@ -26,7 +26,9 @@
    "source": [
     "# Getting a baseline score\n",
     "\n",
-    "This gets the average pixel value of a class"
+    "This gets the average pixel value of a class. The general idea of getting a baseline by comparing its average was developed using ChatGPT with the prompt \"How do I dynamically classify images using the folder they are in as a class. Please use the OS module\" a more detailed explanation of the code can be found in the [model.py](../modules/model.py) file.\n",
+    "\n",
+    "This model simply serves as a baseline for our neural network. At a minimum our model should perform better than 25%, or simply guessing only 1 class. If our baseline model of taking the average pixel value does better than 25% than we will use that score as a baseline instead."
    ]
   },
   {
@@ -159,9 +161,9 @@
     "# Conclusion:\n",
     "\n",
     "\n",
-    "Our baseline model has an accuracy of 46% which is better than simply guessing one class. This model simply compares which class an image belongs to by seeing the avg pixel value of a class and the image and seeing which class the image is closest to.\n",
+    "Our baseline model has an accuracy of 46% which is better than simply guessing one class which should theoretically be 25%. This model simply compares which class an image belongs to by seeing the avg pixel value of a class and the image and seeing which class the image is closest to.\n",
     "\n",
-    "With our baseline our most accurate classification is no tumor and glioma.\n",
+    "With our baseline our most accurate classification, in terms of true positives, is no tumor and glioma. However, we primarily care about the false negatives of our no tumor class. In which case there are 153 mis-classifications compared to 215 correct classification. Because missing a tumor diagnosis can result in the tumor developing further we want to focus on our precision score and reduce the number of false negatives for our no tumor class.\n",
     "\n",
     "We will now move into a more complex mode, a [neural network](../Notebooks/03_Neural_network.ipynb), to classify our images."
    ]

Notebooks/01_EDA.ipynb

@@ -37,8 +37,7 @@
     "from tensorflow.keras.callbacks import EarlyStopping\n",
     "from tensorflow.keras.models import Sequential\n",
     "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n",
-    "from tensorflow.keras import layers\n",
-    "from keras.applications import ConvNeXtTiny"
+    "from tensorflow.keras import layers"
    ]
   },
   {
@@ -49,7 +48,9 @@
     "___________________________________________________________________________________________________________________________________________________________________________________________________\n",
     "# Neural Network\n",
     "\n",
-    "Here we build a basic neural network to classify out images. The below code repurposed many aspects of previous projects I worked on with colleagues. Most of the neural network and metrics takes inspiration from [A plant disease classification project](https://github.com/DerikVo/DSI_project_4_plant_disease) and a single day [Hack-a-thon](https://github.com/DerikVo/NN_hackathon) to classify if an object was a hotdog or not a hotdog."
+    "Here we build a basic neural network to classify out images. The below code repurposed many aspects of previous projects I worked on with colleagues. Most of the neural network and metrics takes inspiration from [A plant disease classification project](https://github.com/DerikVo/DSI_project_4_plant_disease) and a single day [Hack-a-thon](https://github.com/DerikVo/NN_hackathon) to classify if an object was a hotdog or not a hotdog.\n",
+    "\n",
+    "We opted to use a convolutional neural network because of its ability to capture important features by scanning through segments of an image. These features can be shapes and textures that distinguish the uniqueness of a type of tumor. Additionally these models can be used in transfer learning which will allow for more accuracy and less time spent on training the model. Furthermore, because pre-trained models are trained on a diverse set of data our model can be more robust to unseen data. "
    ]
   },
   {
@@ -194,19 +195,19 @@
      "output_type": "stream",
      "text": [
       "Epoch 1/10\n",
-      "125/125 [==============================] - 87s 574ms/step - loss: 6.8223 - accuracy: 0.7800 - val_loss: 1.1457 - val_accuracy: 0.7775\n",
+      "125/125 [==============================] - 43s 321ms/step - loss: 6.8280 - accuracy: 0.7810 - val_loss: 1.0524 - val_accuracy: 0.7558\n",
       "Epoch 2/10\n",
-      "125/125 [==============================] - 50s 402ms/step - loss: 0.2138 - accuracy: 0.9258 - val_loss: 1.0864 - val_accuracy: 0.7079\n",
+      "125/125 [==============================] - 44s 348ms/step - loss: 0.2021 - accuracy: 0.9373 - val_loss: 1.0493 - val_accuracy: 0.7336\n",
       "Epoch 3/10\n",
-      "125/125 [==============================] - 54s 432ms/step - loss: 0.1285 - accuracy: 0.9575 - val_loss: 1.2109 - val_accuracy: 0.7623\n",
+      "125/125 [==============================] - 43s 347ms/step - loss: 0.1155 - accuracy: 0.9605 - val_loss: 1.4395 - val_accuracy: 0.7407\n",
       "Epoch 4/10\n",
-      "125/125 [==============================] - 81s 649ms/step - loss: 0.0572 - accuracy: 0.9820 - val_loss: 1.7071 - val_accuracy: 0.7407\n",
+      "125/125 [==============================] - 44s 355ms/step - loss: 0.0583 - accuracy: 0.9837 - val_loss: 1.6265 - val_accuracy: 0.7342\n",
       "Epoch 5/10\n",
-      "125/125 [==============================] - 55s 441ms/step - loss: 0.0271 - accuracy: 0.9905 - val_loss: 1.7876 - val_accuracy: 0.7815\n",
+      "125/125 [==============================] - 42s 335ms/step - loss: 0.0385 - accuracy: 0.9872 - val_loss: 1.9628 - val_accuracy: 0.7482\n",
       "Epoch 6/10\n",
-      "125/125 [==============================] - 82s 654ms/step - loss: 0.0348 - accuracy: 0.9902 - val_loss: 1.7913 - val_accuracy: 0.7523\n",
+      "125/125 [==============================] - 43s 340ms/step - loss: 0.0331 - accuracy: 0.9887 - val_loss: 1.7622 - val_accuracy: 0.7734\n",
       "Epoch 7/10\n",
-      "125/125 [==============================] - 81s 649ms/step - loss: 0.0084 - accuracy: 0.9985 - val_loss: 1.7633 - val_accuracy: 0.7687\n"
+      "125/125 [==============================] - 43s 346ms/step - loss: 0.0187 - accuracy: 0.9948 - val_loss: 2.5243 - val_accuracy: 0.7418\n"
      ]
     }
    ],
@@ -230,7 +231,7 @@
    "metadata": {},
    "source": [
     "### Interpretation:\n",
-    "Here we see that our training accuracy is about 99% while our validation is 76% which suggest our model is very overfit. We will need to either reduce features or add some regularization. The validation score is higher than our baseline, but the score is lower than [Munir)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) and their team's study accuracy of 87% (N=154). However, this is simply a supportive tool to assist radiologist, and the radiologist response would continue to train the model.\n",
+    "Here we see that our training accuracy is about 99% while our validation is 74% which suggest our model is very overfit. We will need to either reduce features or add some regularization. The validation score is higher than our baseline, but the score is lower than [Munir)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) and their team's study accuracy of 87% (N=154). However, this is simply a supportive tool to assist radiologist, and the radiologist response would continue to train the model.\n",
     "\n",
     "For out next iteration, lets try adding some regularization to see if it can reduce overfitting so our model can be more generalized."
    ]
@@ -293,25 +294,25 @@
      "output_type": "stream",
      "text": [
       "Epoch 1/10\n",
-      "125/125 [==============================] - 60s 458ms/step - loss: 21.2878 - accuracy: 0.6155 - val_loss: 6.0345 - val_accuracy: 0.6232\n",
+      "125/125 [==============================] - 48s 380ms/step - loss: 21.2882 - accuracy: 0.6140 - val_loss: 6.0970 - val_accuracy: 0.5695\n",
       "Epoch 2/10\n",
-      "125/125 [==============================] - 56s 449ms/step - loss: 4.9348 - accuracy: 0.7905 - val_loss: 5.2085 - val_accuracy: 0.5917\n",
+      "125/125 [==============================] - 47s 375ms/step - loss: 4.9106 - accuracy: 0.8052 - val_loss: 5.2855 - val_accuracy: 0.6034\n",
       "Epoch 3/10\n",
-      "125/125 [==============================] - 86s 683ms/step - loss: 3.9861 - accuracy: 0.8510 - val_loss: 4.1504 - val_accuracy: 0.7009\n",
+      "125/125 [==============================] - 46s 370ms/step - loss: 3.9846 - accuracy: 0.8460 - val_loss: 4.1538 - val_accuracy: 0.7272\n",
       "Epoch 4/10\n",
-      "125/125 [==============================] - 55s 441ms/step - loss: 3.2915 - accuracy: 0.8687 - val_loss: 3.8028 - val_accuracy: 0.7062\n",
+      "125/125 [==============================] - 46s 366ms/step - loss: 3.3387 - accuracy: 0.8655 - val_loss: 3.8504 - val_accuracy: 0.7068\n",
       "Epoch 5/10\n",
-      "125/125 [==============================] - 58s 460ms/step - loss: 2.8297 - accuracy: 0.8915 - val_loss: 3.3022 - val_accuracy: 0.7447\n",
+      "125/125 [==============================] - 46s 368ms/step - loss: 2.8415 - accuracy: 0.8882 - val_loss: 3.4656 - val_accuracy: 0.7284\n",
       "Epoch 6/10\n",
-      "125/125 [==============================] - 85s 678ms/step - loss: 2.4836 - accuracy: 0.8950 - val_loss: 2.9780 - val_accuracy: 0.7436\n",
+      "125/125 [==============================] - 46s 367ms/step - loss: 2.4600 - accuracy: 0.9028 - val_loss: 2.9135 - val_accuracy: 0.7634\n",
       "Epoch 7/10\n",
-      "125/125 [==============================] - 56s 445ms/step - loss: 2.2336 - accuracy: 0.9015 - val_loss: 2.7743 - val_accuracy: 0.7471\n",
+      "125/125 [==============================] - 46s 365ms/step - loss: 2.2206 - accuracy: 0.9010 - val_loss: 2.6101 - val_accuracy: 0.7623\n",
       "Epoch 8/10\n",
-      "125/125 [==============================] - 56s 443ms/step - loss: 2.0099 - accuracy: 0.9078 - val_loss: 2.5257 - val_accuracy: 0.7529\n",
+      "125/125 [==============================] - 46s 368ms/step - loss: 1.9791 - accuracy: 0.9128 - val_loss: 2.5702 - val_accuracy: 0.7512\n",
       "Epoch 9/10\n",
-      "125/125 [==============================] - 86s 684ms/step - loss: 1.8300 - accuracy: 0.9137 - val_loss: 2.6962 - val_accuracy: 0.7237\n",
+      "125/125 [==============================] - 46s 366ms/step - loss: 1.8075 - accuracy: 0.9122 - val_loss: 2.3851 - val_accuracy: 0.7629\n",
       "Epoch 10/10\n",
-      "125/125 [==============================] - 59s 473ms/step - loss: 1.7438 - accuracy: 0.9062 - val_loss: 2.4427 - val_accuracy: 0.7348\n"
+      "125/125 [==============================] - 46s 367ms/step - loss: 1.6626 - accuracy: 0.9187 - val_loss: 2.3468 - val_accuracy: 0.7430\n"
      ]
     }
    ],
@@ -337,22 +338,9 @@
    },
    "source": [
     "### Interpretation:\n",
-    "Here we see that our training accuracy is about 90% while our validation is 73% which suggest our model is still overfit, but not as much. The model does better than our baseline model's accuracy of 46% but is less than the accuracy in [Munir's team's](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) study which found the accuracy of two radiologist was 87%.\n",
+    "Here we see that our training accuracy is about 91% while our validation is 74% which suggest our model is still overfit, but not as much. The model does better than our baseline model's accuracy of 46% but is less than the accuracy in [Munir's team's](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) study which found the accuracy of two radiologist was 87%.\n",
     "\n",
-    "Since our validation score was higher without regularization we will stick with the first model. "
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "27c2a852-8f70-4fbe-9d78-0507ea65995e",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import sys\n",
-    "sys.path.append('../modules/')\n",
-    "import model as m\n",
-    "from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix"
+    "Since our validation scores are pretty similar we will have to evaluate the models on other metrics such as precision to see which model suits our needs."
    ]
   },
   {
@@ -365,11 +353,11 @@
     "__________________________________________________________________________________________________________________________________________________________________________________________________________________\n",
     "# Conclusion:\n",
     "\n",
-    "It appears our neural networks have similar scores. These score better than our baseline of 46%, but does less than the accuracy (87%) of the radiologist found in the [Munir et al. (2021)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) study. It should noted their sample size was 154 patients while this data set had over 7000 images; however, we need to keep in mind multiple images could be of the same patient.\n",
+    "It appears our neural networks have similar scores. These score better than our baseline of 46%, but does less than the accuracy (87%) of the radiologists found in the [Munir et al. (2021)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7794124/) study. It should noted their sample size was 154 patients while this data set had over 7000 images; however, we need to keep in mind multiple images could be of the same patient.\n",
     "\n",
-    "A neural network implementing augmentation was attempted, but that was an issue with running out of memory. There was attempts at saving the images instead, but that was causing conflicts as well so augmentation was scrapped.\n",
+    "A neural network implementing augmentation was attempted, but there was an issue with running out of memory. There was an attempt at saving the images instead, but that was causing conflicts as well so augmentation was scrapped. Using a pre-trained model was also tested, specifically MobileNet and NASNetMobile, but those models did not work with greyscale images so that idea was also scrapped. We wanted a lightweight pretrained model for the purposes of this classification problem, so that was the logic behind selecting those two models. In the future more research would need to be conducted on which pre-trained models can be combined with out model to improve accuracy, but due to team constraints that will have to be put on hold.\n",
     "\n",
-    "We will now proceed to our [Modeling Evaluation Notebook](../Notebooks/04_Model_evaluation.ipynb) to evaluate our models even further."
+    "We will now proceed to our [Modeling Evaluation Notebook](../Notebooks/04_Model_evaluation.ipynb) to evaluate our models on other metrics such as their precision scores."
    ]
   }
  ],

Notebooks/02_Baseline_Model.ipynb

@@ -1,4 +1,4 @@
 ,Accuracy,Precision,Recall,F1 Score
 baseline,0.4698703279938978,0.4683084749604378,0.4698703279938978,0.4635669509975083
-Neural Network: No regularization,0.8749046529366895,0.8738431092045584,0.8749046529366895,0.8672969669842382
-Neural Network: With Regularization,0.8421052631578947,0.8403497250779882,0.8421052631578947,0.8293207661173666
+Neural Network: No regularization,0.8871090770404272,0.8865742251752347,0.8871090770404272,0.8822743140636549
+Neural Network: With Regularization,0.8565980167810832,0.8544444062871545,0.8565980167810832,0.8488967210076771