Lecture 02

lectures/02_entropy.ipynb

@@ -0,0 +1 @@
+{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"02_entropy.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"j2l0x3S2XOXs","colab_type":"text"},"source":["# An intuitive introduction to the Entropy\n","\n","Let $X$ be a discrete random variable (RV) taking values from set $\\mathcal{X}$ with probability mass function $P(X)$.\n","\n","*Definition* the entropy $H(X)$ of the discrete random variable $X$ is\n","\\begin{equation}\n","H(X) = \\sum_{x\\in\\mathcal{X}}P(X)\\log \\frac{1}{P(X)} = -\\sum_{x\\in\\mathcal{X}}P(X)\\log P(X).\n","\\end{equation}\n","\n","How to make sense out of this definition? We'll, rather informally, argue below that the entrpy of an RV provides a lower bound on the amount of information provided by the RV, which we'll dfine as the average number of bits required to transmit the value the RV has taken.\n","\n","As a motivating example consider asking your friend for advice. The probabilities of his answers are given in the table below:\n","\n","| $x$      | $P(x)$ |\n","|----------|--------|\n","| OK       | $1/2$  |\n","| Average  | $1/4$  |\n","| Bad      | $1/8$  |\n","| Terrible | $1/8$  |\n","\n","To transmit the answer of your friend you must introduce an *encoding*, e.g.:\n","\n","| $x$      | $P(x)$ | Code 1 |\n","|----------|--------|--------|\n","| OK       | $1/2$  | 00     |\n","| Average  | $1/4$  | 01     |\n","| Bad      | $1/8$  | 10     |\n","| Terrible | $1/8$  | 11     |\n","\n","Under this encoding, we spend 2 bits per answer.\n","\n","However, we could also consider a variable length code, that uses shorter codewords for more frequent symbols:\n","\n","| $x$      | $P(x)$ | Code 2 |\n","|----------|--------|--------|\n","| OK       | $1/2$  | 0      |\n","| Average  | $1/4$  | 10     |\n","| Bad      | $1/8$  | 110    |\n","| Terrible | $1/8$  | 111    |\n","\n","Under this encoding the average number of bits to encode an answer is:\n","\\begin{equation}\n","\\mathbb{E}[L] = \\frac{1}{2} \\cdot 1 + \\frac{1}{4} \\cdot 2 + \\frac{1}{8} \\cdot 3 + \\frac{1}{8} \\cdot 3 = \\frac{7}{8}\n","\\end{equation}\n","\n","Thus, the new code is more efficient. Is it the best we can do?"]},{"cell_type":"markdown","metadata":{"id":"uOpiKOWvn92s","colab_type":"text"},"source":["### The code space\n","\n","We'll now try formalize the coding task, i.e. the assignment of code lengths to possible values of the RV. \n","\n","Let's first observe an important property of our code: in a variable length coding, no codeword can be the prefix of another one. Otherwise, decoding is not deterministic. Therefore, whenever a value is assigned a symbol of length $L$, $1/2^L$ of the code space is reserved and not available to other codes.\n","\n","This can be visualised as a code space. Below, we indicate the codes assigned to symbols in the example and  grey-out codes that are not available because the shorter codes are used:\n","\n","<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAMsAAACjCAYAAAAkXOoEAAAP30lEQVR4Xu2dz2sd1xXHv0q8URGtINmpJMUEVOjCISkmkGBENv0Tuoi6KsiQGFGtWihKI5uuslARNsE2WtStTCNMF9lVi2Ar1sLGXjRkk0WEswgkJTH2ImgRJypHmvtyNe89vZk536s3T/f7wEjWu/fMnXPPZ84599eMQR9pQBqopIExALuVSg6p0Nzc3JCurMsehQZu3LiBhw8ffg7gBwDfO38yZJTbYDLfMV3swbK7215ezp49i5dffvko+q3RNe7fv59F+z799FPcunWro6MXXngBZ86cwYkTJxrpLVQymdevX/8FgKcBPHXIz8O+K9etU7bKdZcEi6ub9yvnAMujR49w8+ZNzMzMYHJyEk+ePMHm5iYmJiZw+vRplxZNf1euXLGHdus/8izOLsoFlk8++QSvvPJKx5OYp3n8+LFgcdoPtbrCMJ86GTCbZxEsyll8lphRGCZYBEtrYEmVQMuzuLu4I0A5i1OXLGNMmUB7RxMVhu0biWBpCSwpwxwGLClh1miY0whD9RwS/NRPbi8s1hcpw0TBIlgqayA1LJUbMqSCgoWkeHkW/6Qfw7OE7rx79+7er97JyCBPk5IkUEyMYGkHLHEY9uKLLwoWoo3TROUCS8oEmulZWDP38iw0RH4UlAMsqRNowcIxTA0dO/XImGdxNuHQ6uz2ybNksEQ/5dAn88nNBkew8DSahWdJvcRcsDQ3SI2GNdddV01GzpJ6HkOwNO9wwdJcd4KFqDsTpTCMp9BswrCUa6943ZGnJM3gk/pdYZhPkSzPknKARLD4+rhTW7D4FMmAJfUAiWDx9TEdllGZISeprSOGBUvKMFawkHqd4VlGaYacpDbBwlakNn/5Ncp4cvtb0V8Co32ph97lWUgWwPIspOZ0iWEYYyw0xRJ47zyQYNnvoSyGjlOBwpzHSLkEXrBwLECwOPXI9iwpFioyYEk5QKIwzGmEoXpuYVgbYUk9QCJYBEsjDaSApVFDjrCSYCEpW57Fp0h2mJhiAEKw+PqYOilJakpPMWxjTOFZvDlLOQzTHvyUFuWQLc/iUJ5WHfuUV6qt0TCnOnPxLEFNKTyfwjCnEeY6GkZSW0fMKMCcHSxra2uYnZ09kGssLy9jfHzc1f+5hWEuZfWofAxhMYNatiPlotu9DGABwA6A6eIdkOcAfBOVeRXAbQBvA7jQRM+UMMxc8/z8PFZWVjA9PY2dnR0sLCxgamoKi4uLTdqVbYLvUtbxhyWAYnca4Cj/7bkesBhA7wN4C8BWUx3TYFldXcXS0lLHk5in2d7eFixNe4ZU75h5FnvyTkWgBC0FYL4AsF6ChQKKXUiwOI2SbYzO5nRVZ7dvyAm+wbINYK2Hnt4AcLIEy7MMjxKuNRKwsA1I8tqlgRoJfh1YLgK4BOCUJ0+JNTUSsDAm1VKZB/vJzW4nq30t2YNfFRbLTwyS1wB8zfIugsVpnSxjdDajb3VG+1q0B79qzmKwvBuFazYSZnUtVItHyGqpnQaLRsNq6f3ICrNgacke/KajYabvfqBV7gsKLHY1zbNU1vmRFjxmsJjums6zPFN4Ghs6Ht48S8re16SkT7vHEBafQhy1aZ7F0YZDq+YCS8oE2jtAoj34+yYqWJyUs57cKbftChZnJxfVBYtTjyxYUibQDFhSwlxjnsXZW77qgsWnP8op9anDHC8spqKUYaJgcRphqJ5DzpIaFlJXJBMjWEiqFSzteLV36E7twc/gnZIkdrvE5JCzlMMw7cFPZU1Oubl4lpQJNCNnCd045FXHTmvyVVeC79MfJcFPnUALFmcna+iYo0BGGMZpSW8p7PbJs2SUs6RIUJlPbjY4goWn0WzCsDafUs/rzm5JgoWn3WxgSZmgyrM0N0iDWfMszfV3oCZ7NCxFzC1Ymne2YGmuu66aowAL8XazFCXPQur2UYAlB8+itWEZLtFXGFb/KdaiPfj1G0+soQTfqUz2aJOzOV3VGe1LvdBTYRip1xWG+RQpWHz6i2tn51l4qtuXxDBGdptieYz2ybPsa1SwOC2VYYxxE9q4wkCwCBYnJlzP0uYVBoJFsLQKltCYNo7WaTRMsAiWGhrQPItylhrm0rsoO2dJ4VncN5lYgIaOSQpmDx2TmtURMwqwMFcYpBiAECwkqxQsPkWyYE45ACFYfH3cqS1YfIpkwZJyAEKw+PpYsJz2HYEUFChYSIaoSUm/ItnG6G/RQQns9qUYgJBnIfV6bmEYSW0jNQAhWEi9Llh8ipRn8ekvrq21YU5dso3R2Zyu6uz2KQzL6Cikthtj29uXPSzsDspJ3tzcXE63S7/XBw8eYGNj43MAPwD4vvgZ/x7+NuhnkzpVr7cEYLf1YdjY2Bh25fkaGykrDEu1Nmxzc9Ne3vs8gKcAPD3gJ6tMuE4VeVb2vHWAYGlshvsVcxiA0Krj/b4WLIJloAa0n0WwDDSSKgVy8Swp33mpeZYqllahjHKWCko6pAgjZ5FnkWfxWWFRm+VZUiXQgoXSzcpZGGpkwJI6gfbuZ5FnkWdhsEIZDUttjAxYUr7GTzkLxRSBHHKWtsNiXZkyTBQsgqWyBlLDUrkhQyooWEiKl2fxbQJjJPhxV2oPfouXkwiWdsCiPfiawXf7vxxGw2IlZb/quM0LFXPwLKkTaO9omGDR0LHbq5gAhmehNKSPEHbOIs+SQc6ytraG2dnZjkmZkS8vL2N8fNxlq4LFpb69V3ZcuXLlJwCWC0kLAHZ8UtPUzmLVsT0N5+fnsbKygunpaezs7GBhYQFTU1NYXFx0aVawuNQXYHkNwO8A/AzAOzat45OapnY2sKyurmJpaanjSczTbG9vC5aadpUoDHsbwDaAk8XPNQDPALhYgueNosyHAG5HTb8MIHgke/p9BeBNAG8B+BrA+wBOFeUNzK3i9+noO2uDfS4AeLWXfMEiz1ILFzYsW1tbuHbt2r8KKJ4tPExs+AaRwWPxsoVqtwD8sQDBjD78/aOinMEyVcBj92ZbglcLb2UQ2PcGnX1M7j8A/LuQ/QWA9QIgA+2AfMFCgKWWtanwAQ18+eWX+OCDD4JnsNwl9iZm3BaeGTzPFUD9DcAfAJwD8E0hzAAIUMW/l7Vt3uqvAP5cfBFfK3gtg8VCwS75goUAC3Nols0SazQs1dqwq1ev4t69e+XbtpGYcij26yIEC0/+EFaFuqGOwWJhWgi17Pvgfc4C+E/kWcpgvh55li75gkWwDOQz1RaCb7/9FpcuXcJnn332yyipj72JjYoFT2H5jEFgOUj5yR/fQwxLyEns+98Wdft5FrtugKWn/Gxg0WjYQCb6Fki10PPOnTv4+OOPzbOYHYZPObE3I7bk+2ERGlleYwl7yCnK5cuwrACYL2C0UMvCunLOYl4sDIuWc5aO/Cxg2cvkNM/SmJYUsHz33XdYX1/H5OSk5SwxLNbOOO8wYzVjtrDKRqrsUx6tCiFYqBuHYQbGP4t6Foa9BMASeZNVdTRsT342sDS2lAEVc5hnSQFLUGsxKVmGJVV3leXGXuiwgYG9eoLF2S1sWFIsgfcOQBxjWGLPEs/V9LQKwdISWFIugRcszk4uqgsWpx7ZnoU96ccYOk41GmaqH3IYVqv3BUstdXUXzgEWu+tU8yyCxWmAcfVc9rOEe07hWYjdkUSU9uCT1CpYfIpkhGFxC1IMQAgWXx93agsWnyJZsKQcgBAsvj4WLHq1N8mCeGKU4Dt1mUuCnzKnkmdxGmGonlsYRlJbRwwrDBMsCWbwL1zYX77j3a4rWDjYCBaOHk0KLQyLFyqeP39esPD6yCVJsLjUd6AyDZYglbW3XZ6F08mChaNHqmcRLLxOYUoSLDxtjoRn4d0uX9Lc3BxfaEYSHzx4gI2Njc8BxO+xb/JO+yZ1vi+uO6iuHXqxOxKw6HjZ5vSwhrZTrQ3b3Ny0jXnPA6jyTnpWGXu3vcmqIs/KnlcY1twGOzVzGNrWquP97pZncQKTCyx6tbdgcaKi1/iddi7L0RJ9twn+KKDtT+62t4+RsxzjbcW1LJUehtW6eoXCbTdGVvvafPpMC2CxwyT2kuzo89/iLLAmh4jHRyFVsMJEOUvlK1csyDLGiperXYzRvraf8t8SWMLxrKGP4jPAwjGuVftPsFTVFLMcC5Y2n/LfgtGwXscU2cks8cmR8Ukt1sX9Tsu341vt3/+K88gqm4PCsMqq6l0wB1jszlPNs1RM8HvBEk6QtJW7dpZxldPy7bC+ANW7gsVp/HWrjwIsde/pqMtX2M9SN2c57LR8u72BB+r10oE8i9MyRgEW77lhsYqGtAe/XxgWn2NszaxyWr6VU87itPtG1XOBZch78HvBEsCwlxGFt3tZHw46LV+wNLJ0QiUWLKNyyn+Ko5oqhmHl0bCQe4RX4VU9LT8cNG6QWQ5T+aMwrLKq0iX4JrnN8yzxnQ8RlvI8izUrPj2/6mn5Nj+zAcB+Chan/deqzvAstS5YszBjBr8FsNS86zTFs/MsOiPAZ0hD8iy+RpNqZwOLzgjgWIxg2d3laDKBFHaYozMCfJ0kWARLYwtiw9y4IX0qKmfhaTSbMCyoLIVn4XUHX1LbzwjIeg8+u7vZT+4UsLT5jACWZ0m1NizrPfiCha0BnzwGLC1YdexTAqm2wjCnItmez9mcruosWLQHP8EefHZns41RYVj9HmrB5q/6jU5QIzvPwtYhG2Z2++RZeBoVLE5dsmFhrzBoOyxbW1u4du2a7Vz8TY+usL/bmq+624aDqFcBvA7ANojZyuUPAWyVrmMLMn8P4C8Adg4zB8HSElhSrTBoOyylnZJmuOU9Ks4e6lQXLCxNNpXD9izsnIoFy82bNzEzM4PJyUk8efIENuQ7MTEB8rlhvWAp762PvU3wCtZ9PwXwXrSxy/6/DsC8S/AsXwF4s+hr2/diJ8PEnuUlALcjW7gMYCF4HHmWppQU9XKAxW411TzLAM/Sa+9JvPc+3tNi4VX5/+UwbKowfoPC5FiI92wRhtly/b8DsP0xJitsLvsoLOUXLBnA4rzF5NWjzV9lz2KwXCxOcQnng5mBnyy8Ra8TXuITX8qwhA1kcb0Ylj8BOBflSAd2aAoWpynk4lmCmhLvwe8Fiz3xy8n/2xEscXJeTtb7Jfj9YDHPcqpkEp0NZoJFsFTSwBHtwa/iWeL2luEYBEtdz3JAN4Klkqn0L5SbZ0m8RL+fZwn75eNDKkKOUsez1MlZukJAwSJYamngiGGxtpVHw0IIFr6rA8uvitNf4nOSDxsNi/f489/PUkvzFQqzn9wVLlmrSNvbxxg6jhWSGJZauj/qwvIsTo0LFp8CKx7f6rsIqbZgcSpSsPgUKFh8+js4AjE2hjZvrhIsvs4WLD79CRai/pSz8JS5F4bxxElS2zSgPfgY9I57+35QGXudxa7Boo80MEwN/LzGO+mrvLe+Shl7t72Vq1p27+jY/wNE+8URRoxnFgAAAABJRU5ErkJggg==\" />\n","\n","We can observe, that the length 1 code for \"OK\" uses $1/2$ of the available codes, the langth 2 code for \"Average\" uses $1/4$ and the two length 3 codes for \"Bad\" and \"Terrible\" each use $1/8$ of the code space.\n","\n","In general, a code of length $L$ uses $1/2^L$ of the code space. Equivalently, assignign a fraction $f$ of the code space to a symbol makes it use a symbol of length $L=\\log_2(1/f)$."]},{"cell_type":"markdown","metadata":{"id":"T_jOnTuVo0jj","colab_type":"text"},"source":["Assuming that we assign a fraction of a bit to a symbol, our optmal coding problem can be formulated as partitioning the code space into four regions (one for each value of the RV) such that the average length of the code is minised. \n","\n","Formally, let $p_1, p_2, p_3, p_4$ be the proibabilities asigned to the 4 symbols and let $f_1, f_2, f_3, f_4$ be the coding space fractoins assigned to them.\n","\n","We want to:\n","\\begin{align}\n","\\text{minimize } &p_1 \\log_2 \\frac{1}{f_1} +  p_2 \\log_2 \\frac{1}{f_2} + p_3 \\log_2 \\frac{1}{f_3} + p_4 \\log_2 \\frac{1}{f_4} \\\\\n","\\text{subject to: } & f_1 + f_2 + f_3 + f_4 = 1\n","\\end{align}\n","\n","For simplicity, we will solve this problem for the case of only two symbols:\n","\\begin{align}\n","\\text{minimize } &p_1 \\log_2 \\frac{1}{f_1} +  p_2 \\log_2 \\frac{1}{f_2} \\\\\n","\\text{subject to: } & f_1 + f_2 = 1\n","\\end{align}\n","\n","Notice first that $p_2 = 1-p_1$ and likewise $f_2 = 1-f_1$. Then our minimization objective becomes\n","\\begin{equation}\n","\\text{minimize } C = p_1 \\log_2 \\frac{1}{f_1} +  (1-p_1) \\log_2 \\frac{1}{1-f_1} \n","\\end{equation}\n","\n","To get the minimum over $f_1$ we compute the derivative of the expression with respect to $f_1$ and set it to zero:\n","\\begin{equation}\n","\\frac{\\partial C}{\\partial f_1} = \\frac{p_1}{\\log 2}\\frac{-1}{f_1} + \\frac{1 - p_1}{\\log 2}\\frac{1}{1 - f_1}\n","\\end{equation}\n","\n","Multiplying both sides by $\\log 2 f_1 (1- f_1)$ we obtain:\n","\\begin{align}\n","p_1(1-f_1) &= (1-p_1)f_1 \\\\\n","p_1 - p_1f_1 & =f_1 - p_1f_1 \\\\\n","f_1 &= p_1\n","\\end{align}\n","\n","Thus the optimal fraction of code space allocated to symbol 1 is $p_1$, the probability assigned to this symbol and the optimal code length is $\\log_2(\\frac{1}{p_1})$!\n","\n","We now see, that the entropy \n","\\begin{equation}\n","H(X) = \\sum_{x\\in\\mathcal{X}}P(X)\\log \\frac{1}{P(X)}\n","\\end{equation}\n","is simply the average code length!\n","\n","### A note about logarithm basis\n","\n","It is custommary to copute the entropy using natural logarithms, which gives its value in \"nats\". IF $\\log_2$ were used, the entrpy has units of bits and corresponds and lowerbounds the average amount of bits needed to transmit a value of the RV."]},{"cell_type":"markdown","metadata":{"id":"KOBFV5X6aeHg","colab_type":"text"},"source":["# Further Reading\n","1. Chris Olah \"Visual Information theory\": https://colah.github.io/posts/2015-09-Visual-Information/\n","2. JA Thomas ad TM Cover, \"Elements of Information Theory\", chapter 2"]}]}