Add images

Author: weidi-chang

CMAC.ipynb

@@ -80,21 +80,20 @@
    "metadata": {},
    "source": [
     "## CMAC vs the human brain\n",
-    "# TODO: Finish\n",
-    "Looking at Figure 1, we can correlate each section loosely to the human brain. However, these are loose correlations and do not accurately model the cerebellum. More importantly the model is fast as it can be viewed as a lookup table.\n",
+    "Looking at Figure 3, we can correlate each section loosely to the human brain. However, these are loose correlations and do not accurately model the cerebellum. More importantly the model is fast as it can be viewed as a lookup table.\n",
     "\n",
-    "#### Feature Detecting Neurons\n",
-    "The first layer is related to sensory feature-detecting neurons.\n",
+    "#### Feature Detecting Neurons (Mossy fibers)\n",
+    "The Mossy fibers are one of the two input fiber systems to the cerebellum. Therefore as the first layer (L1) in Figure 3 is the input to the CMAC architecture, drawing a parallel is fairly natural.\n",
     "\n",
     "#### Granule cells\n",
-    "The second layer in Figure 1 corresponds to granule cells.\n",
+    "These cells are the most numerous cells in the brain ($\\approx3 \\times 10^{10}$ granule cells in the cerebellum\n",
+    "alone). They can be said to be association cells which \"recode inputs from N inputs to at least 100N inputs\" [6].  \n",
+    "We can relate this mechanism to the second layer in Figure 3 since in the CMAC architecture, a single feature links to multiple tilings/AUs.\n",
     "\n",
     "#### Purkinje cells\n",
-    "The third layer in Figure 1 corresponds to Purkinje cells.\n",
-    "\n",
-    "### Differences from the human brain\n",
-    "\n",
-    "There are suggestions that the cerebellum is capable of exclusive disjunction which is not in CMAC [1]. TODO more on this."
+    "Purkinje cells, as shown below, are highly branched neurons which connect to a number of other cells. They exist in a single layer over the cortex. They form a sort of tree structure over other cells. A parallel fiber below it “synapses with virtually every Purkinje dendritic tree it passes” [6]. The axons of these fibers are the only network exit from the cerebellar cortex.   \n",
+    "Note that this structure relates to Layer 3 fairly intuitively. As Layer 3 in our CMAC architecture connects in a tree-like fashion to the AUs, so does the Purkinje fibers connect down to the parallel synapses below it and aggregate these potentials before passing them out of the cerebellar cortex. This is an obvious simplification of the fibers, but the parallels are noticeable.\n",
+    "![cmac](images/purkinje.png) [wikipedia:Purkinje cells]"
    ]
   },
   {
@@ -109,8 +108,8 @@
     "## CNNs vs CMAC\n",
     "\n",
     "Indeed, Tile Coding bears some resemblance to both Convolutional Neural Networks and HMAX (Hierarchical Model and X) in several ways.  \n",
-    "To demonstrate this similarity, we first look at the biological basis for CNNs/HMAX. In HMAX [6], the early layers of the visual cortex are modeled as convolutional filters which are precomputed to match those found in the visual cortex. These convolutional filters match the receptive fields found in the human brain. Receptive fields come from the foundation that each cell is activated by a small area of the visual input. In the early layers of the visual cortex, we find that the receptive fields are simple and circular (on and off depending on stimulus in one area). These cells are called simple cells.   \n",
-    "Moving forward in the visual cortex, we find more complex activation patterns (complex cells) which have receptive fields responsive to lines or curves. In CNNs, these convolutional filters representing receptive fields are learned [7]. These compare to tile coding as the tiles can be thought of as square receptive fields which aggregate the input from a set of sensory inputs. As we can see, this parallels HMAX and CNNS. "
+    "To demonstrate this similarity, we first look at the biological basis for CNNs/HMAX. In HMAX [7], the early layers of the visual cortex are modeled as convolutional filters which are precomputed to match those found in the visual cortex. These convolutional filters match the receptive fields found in the human brain. Receptive fields come from the foundation that each cell is activated by a small area of the visual input. In the early layers of the visual cortex, we find that the receptive fields are simple and circular (on and off depending on stimulus in one area). These cells are called simple cells.   \n",
+    "Moving forward in the visual cortex, we find more complex activation patterns (complex cells) which have receptive fields responsive to lines or curves. In CNNs, these convolutional filters representing receptive fields are learned [8]. These compare to tile coding as the tiles can be thought of as square receptive fields which aggregate the input from a set of sensory inputs. As we can see, this parallels HMAX and CNNS. "
    ]
   },
   {
@@ -140,11 +139,12 @@
    "source": [
     "# Implementation and Experiments\n",
     "Our implementation of CMAC/Tile Coding was done in TensorFlow.  \n",
-    "Note: Our implementation is based off of [8, 9], and the plotting functions off of [9].\n",
+    "Note: Our implementation is based off of [9, 10], and the plotting functions off of [10].\n",
     "\n",
     "### Function approximation\n",
-    "We demonstrate our CMAC implementation on a few 2-dimensional functions and compare it to a 2-layer Multi Layer Perceptron :  \n",
-    "Note that for the MLP mean squared error graphs, each epoch is 100 training examples.\n",
+    "We demonstrate our CMAC implementation with parameters resolution = 50 and n_AU =50 on a few 2-dimensional functions and compare it to a 2-layer Multi Layer Perceptron :  \n",
+    "Note that for the both mean squared error graphs, each epoch is 100 training examples.\n",
+    "\n",
     "- $f(\\textbf{x}) = cos(x_1) + sin(x_2)$ : \n",
     "#### CMAC\n",
     "![cmac](images/cmac_function_in_the_making.gif)\n",
@@ -155,7 +155,18 @@
     "And the associated mean squared error: \n",
     "![cmac](images/mse_mlp_sin_cos.png)\n",
     "\n",
-    "As we can observe, the MLP takes about 1100*100 = 110000 training examples to reach an MSE below 0.1 while CMAC only needs about 20."
+    "As we can observe, the MLP takes about 1100*100 = 110000 training examples to reach an MSE below 0.1 while CMAC only needs about 20*100 = 2000.\n",
+    "\n",
+    "\n",
+    "- $ \\sin (\\sqrt{( (2x_0 -4)^2)} + \\sqrt{((2x_1 - 4)^2)})$:\n",
+    "#### CMAC\n",
+    "![cmac](images/cmac_cone_in_the_making.gif)\n",
+    "And the associated mean squared error: \n",
+    "![cmac](images/mse_cmac_cone.png)\n",
+    "#### MLP\n",
+    "![cmac](images/mlp_cone_in_the_making.gif)\n",
+    "And the associated mean squared error: \n",
+    "![cmac](images/mse_mlp_cone.png)"
    ]
   },
   {
@@ -174,13 +185,15 @@
     "\n",
     "[5] J.S. Albus (1975). \"A New Approach to Manipulator Control: the Cerebellar Model Articulation Controller (CMAC)\". In: Trans. ASME, Series G. Journal of Dynamic Systems, Measurement and Control, Vol. 97, pp. 220–233, 1975.\n",
     "\n",
-    "[6] Henderson, Peter, and Dhirendra Singh (2013). \"Biologically Motivated Object Recognition.”\n",
+    "[6] J.S. Albus (1978). \"A theory of cerebellar function\"\n",
+    "\n",
+    "[7] Henderson, Peter, and Dhirendra Singh (2013). \"Biologically Motivated Object Recognition.”\n",
     "\n",
-    "[7] Rashad, https://www.quora.com/What-is-a-receptive-field-in-a-convolutional-neural-network\n",
+    "[8] Rashad, https://www.quora.com/What-is-a-receptive-field-in-a-convolutional-neural-network\n",
     "\n",
-    "[8] Stober, 2012. https://gist.github.com/stober/1792732\n",
+    "[9] Stober, 2012. https://gist.github.com/stober/1792732\n",
     "\n",
-    "[9] Pezeshki, 2017. https://github.com/mohammadpz/Theano_Tile_Coding"
+    "[10] Pezeshki, 2017. https://github.com/mohammadpz/Theano_Tile_Coding"
    ]
   }
  ],