Lecture-4-Expressive_TTS-zero_resource_TTS.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Expressive speech synthesis\n",
    "\n",
    "Text to Speech systems built using Neural Networks are proved being efifcient and robust, morover they can be scaled to multiple voices, but we still want to bring next level of naturalness of synthesis. As well as add controls to change intonation, highlight important words, transfer style of speeking of well know public speakers.\n",
    "\n",
    "# Prosody\n",
    "In linguistics, prosody is concerned with those elements of speech that are not individual phonetic segments but are properties of syllables and larger units of speech, including linguistic functions such as __intonation, tone, stress, and rhythm__\n",
    "\n",
    "Prosody conveys additional layer of information in our speech, it reflects mood of a speaker, encodes sarcasm. it also carries cultural context, and particular style of speeking prominent in one culture but absent in another.\n",
    "\n",
    "Hard part of this task is to __get a suitable representation__ of the style from a reference utterance which is dissentangled from speaker identity and linguistic content.\n",
    "\n",
    "<img src=\"assets/expressive_tts.png\" alt=\"expressive_tts\" style=\"height: 500px\"/>\n",
    "\n",
    "Data: \n",
    "A single speaker high-quality English dataset of audiobook recordings by Catherine Byers (the speaker from the 2013 Blizzard Challenge).\n",
    "- 147 hours\n",
    "- 49 books, read in an animated and emotive storytelling\n",
    "style.\n",
    "\n",
    "Image from [Towards End-to-End Prosody Transfer](https://arxiv.org/abs/1803.09047)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "def display_yotube_video(url, **kwargs):\n",
    "    \"\"\"\n",
    "    Displays a Youtube video in a Jupyter notebook.\n",
    "    \n",
    "    Args:\n",
    "        url (string): a link to a Youtube video.\n",
    "        **kwargs: further arguments for IPython.display.YouTubeVideo\n",
    "    \n",
    "    Returns:\n",
    "        YouTubeVideo: a video that is displayed in your notebook.\n",
    "    \"\"\"\n",
    "    id_ = url.split(\"=\")[-1]\n",
    "    return YouTubeVideo(id_, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDBgYFhsaGRodHRodIzAlHx8fHzAgIiYuLicxMy0nLSs1PVBCOTlLOSswRWFFS1NWW11bMkFlbWRYbVBZW1cBERISGRYYMBsbMF04Nz1dV1dXV1dXV1dXV1dXV1lXV1dXV1dfV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV1dXV//AABEIAWgB4AMBIgACEQEDEQH/xAAbAAEAAgMBAQAAAAAAAAAAAAAAAQQCAwUGB//EAEUQAAIBAgQDBQQFCgQFBQAAAAABAgMRBBIhMRNBUQUiYXGhFDKBkQYVQrHRFiNSU2JykpPB8DNDY+E1grLC8SRzg6Kz/8QAGQEBAAMBAQAAAAAAAAAAAAAAAAECBAMF/8QALREBAAIBAwQBAQgDAQEAAAAAAAERAgMEEhMhMVFBFAUiMkJScYHBYZGhYhX/2gAMAwEAAhEDEQA/APn4AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJM8qL44Tl4RbWDZlQyov0ci2sGzKhlQ6ORbWDZlQyodHItrBsyoZUOjkW1g2ZUMqHRyLawbMqGVDo5FtYNmVDKh0ci2sGzKhlQ6ORbWDZlQyodHItrBsyoZUOjkW1g2ZUMqHRyLawbMqGVDo5FtYNmVDKh0ci2sGzKhlQ6ORbWDZlRgUywnHyWgAFEgAAAAAAAAAAAAAAAAAAAAAAAAAAAACSCSAAAAk2Gs2mnb/ACrkRi20lq3okXMR2RiqUXOpQqRit5OLsvPoacJ/i0/34/ejt9qY3D0cRjeHGs61SVWnLPKKprNJ5mkld+B2ymYmoQ4FajKnJxnFxkt09GjA9b2kqVati4VKdOMafBfEStNZp04ybl5SfhsI4KMqs41cNClCniacKTUMudOrlcW379463/Ejn7HkjOlSlN5YRcnZuy1dkm2/gk2emwlChiHHPRpxUMS6cVBZMy4c3GEnzvKEVffVk4KldQq1KMaNb/1EVGMOHmisNN3y+D0v4jmPKkpN6I9LUwcEqlPgR9njhuJDEZO85cNSUuJzvN5bfDkbsVgopYqPssI0aShwKuTWSc4K+f7V07jmPLVacoScZJxlF2ae6a3RgerxGDUJ2oYWFbPiasKicMyiozSjC/2O673/AAJw/ZsOBklShrhpVLxouWuRyjJ1299tEmuQ6kDy86E4zUJRak7Wi1r3knH5pr5lnEdkYqlFzqUKkYrduLsvPoXO0/8AiFP93D//AI0yz2ljcPQr43hRrOtUdWnLPKKprNJqTSSu/ATlPwPOg9Vi6VPi4qNLDUnPDxjwoKnmvmcc85L7VuXJZjOGEpRk5OgozlTpSnajxo0pSzXi6V7pSSi78tuY5jyRnRoynJQhFylJ2SWrbPTvs/hShGnhqVZzxE4VrRdSMEpLLBPeKs277+Ohv7NwypYnDLD0I1aUqtTNWy5pLLVlFd/7NoKL8cw5jxwO7Tw9F4RYvJG0KTouNt617Rlbn3JZvOJwy+M2IBILUhBlCDk0optvZJXbIPZfR7AQp0I1LXqVFdy6LkkVymoS4eH+jteSvPLTXjq/ki2vo9SXvVJvysvxPRKDk8q3/BXNdbs+peycfn4HCc5S4X1Jh+s/4l+BH5P0XtOa87P+h1fq+rv3fX8DfT7PqpXdtOS1I5yU83X+jdVK9OUZ+D7r/A5NahOnLLOLjJcmrH0GdCVNpStd9DR2hgYYilKMlqk3GXNM6Y5oeBNZtNTKbj4TCAAZVgAAAAAAAAAAAAAAAAAAAAAAAAAAAABJBJAAAASbTUbjVtvlXIQbbd3q3uwDXSqXNu929d9d/MylWm8t5SeT3btvL5dDADiDk7Wu7Xvbx6mc605SzSnJyta7k27dLmAFDJ1ZZcmaWRO6jd5b9bF6v2tKVKVONKnTz2zuF1ezTsle0dUm7LVnPBE4RIzjWms1pSWf3rN97z6jjTyqOeWVbLM7K+9l8TAE8YEuTbvd3689NiG7u71bAFDKNWSlmUpKS+0m0/mTCvOMnKM5KT3kpNN331MAOIyhVlFSUZSSkrSs2rro+ohVnFNRlJJtNpNpNrZmIHEWKuMcqMKKjGMIvM7XvOVrZnd8lyXVlYkCMaEAkChB3Oxe3ODHhVbun9lrVx+HNHEBE42Pe0sTCos1Oal+6zRVizxUZNO6bT6rRlmHaVeO1Wfxeb7zlOj6Tb0nDd+Zvg5O123ZWXl0PL/W+I/T/wDqvwMZdqYh/wCbJeXd+4joyW9k6kYLNOSiusnY4va/0gi4SpULu6tKe2nNR/E89UqSk7ybk+rdzEvjpRHlCDUbjSzjufhbFAAMiwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAkgkgAAAJNxpLBs2vypkxFjJb9fA2OpH9XFe9zlz258uXrc2Uq3U+zqk6HGjZxUpJq6TWWKk346Pl0ZlT7JrPiXio8NScszS1gk3Hzs0MN2lKlTyKEG05uM3fNFzgoytZ22XNFip25OWZOlS7+bPbN3nOKi372my2KTGYrUOzKtSEZ045k73u1FKzS3b6yRqp4KrLiJQd6fvptKzva2vO+ljdT7SnGjwbRy9db6zjLr1igu06inVmlFSqzVR6bNTzK3x6ishi+yq6llcLeLlFRVnZpyvZO7Stvdlil2HVllV0pNXaloo2dVNN9fzT+Ztl9IqrlmyU72fObs2020811stFyMfr+rnzOFPV3a71neVVtb/60vkitag59bCVIJucGkmk34uOZempoL/aGN4sKNOOsaULXccrbe73eyUY/8po4sL34UbXTtmla1tt9uZeLruK4JsTYtQxBlYWFDEGVhYUMQZWFhQxBlYWFDEGVhYUMQZWFhQxBlYWFDEGVhYUMTSzeaGZN18LYoABiXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEkEkAAABJZsVi0bdn8qZssPRdSpCnG2aclFX2u3ZXLVXsmsmlGPEvHNeCe12tVJJ7rpryK+HqunUhUVrwkpK+107/0OjHt+tGyjFKKtZZp3um2m5Zrvd6Xsa8uV9lFKHZ1eTtGlNvy8E/+5fND6tr2i+FO0ttN9L/cW5dryShFRUoxhFSU1bNJNNSeV8rRW+qijB9sVW55owkqiippprMoxkuT095+hH3xhHsqo51oNxi6LSnfM9W7aKMW/QouNn18S7LtKo3WatF1mnLLdWs72WuxrxVaEo01CNnGNpOyV3ZLlvqm7vqWi/lCtYWJBYRYWJAEWFiSAFhYkARYWJAEWFiQBFhYEgRYWJAEWFiQBFhYkARYWJAGNiuyyVmYt34h0wQADCuAAAAAAAAAAAAAAAAAAAAAAAAAAAAAJIJIAAACS0VS4btn+ZzzRYWBJvpzRYmCV1dtLm0rv5aEEihcwlOg4VHJptPu55cNqOWV5JK95XUdNd/imLy8CjGPDzQUuJlazNubS89EtuRTBXj3st2pUMK6WSMqSd7wk6lpPSndz6f5mmm22186eGwDquOaCpq3e4ju7zkm9ZJaRUeu+z5cIFen/lNu5kwk4w/wlK0e5xMsbOFLM273uu/pfk9HzqYyhh3C1CzlGzu5ayXDlKbaeiaaW3WxzgTGnXyi3RwlLCunFza4jzRcXNxV0pNSb5J9yPz2LNWjg8toum2uTqOKV/eaktXZ7b3XJnFAnTufJbvLD4OpVyqzb1TjJ958SSyWWiWXK7peJQx+DjTrU7xUYPJnip3s3GLmr6u3e31+JQTtsG23d6t82IwmJ8lu9VwODUJTWXJmaclUbcfzbaUVdqTvbrvy5a8RhsEnLhuLfJSq2jlzPvppvvWtp4+7yOLcEdOfZbu1aeClVVlSSacnabS+yoxXeSWjb35fA1TjhfacqlHgqm45k1ZvM0tetjjgRp18lu3F4TJKMY01nezm/s8bKszd0m407v8Aa5cuXjKUFVmqTWRJP3rq9lmUW97Sb+CNALRhU+S0zp25xerWkk9n4GNiSC1BYixIJoRYWJAoLEWJAoRYqstlRmHeeIdMEAA890AABv8AZ/H0Hs/j6G4EtnSx9NHs/j6E+z+PobgDpY+mj2fx9B7P4+hvAOlj6aOB4+g4Hj6G4A6WPpp4Hj6DgePob0jqYHstS1qXt0X4iahWcMIcTgePoOB4+h6LFdmUrdxOL87nGq03F2Y7IjDCfhW4Hj6DgePobiAt08WrgeI4Hj6G0kHTx9NPA8RwPH0LFKGaSV7Xe5voYdZpZtYx06XfIInTwhQ4Hj6DgeJcrwvOeRd2Ppy+80hPTxVCCWQQyAAAkuFMum/Zfmc9RuwNJTr0oS92VSMX5OSTO0+wYVMri3T7uqUJt3c2r5ZPMkkldnnw73vzNuWMzPaac3Xpdixkk1Uk1JKUVCGaTi3FZrX2UnJP9x9Q+xI6pVZSlCKclGne+aEpJQ72vuM5dSrKVrv3VlVkkkull5v5mBHDL2OsuyqcsTWp55QjCooR7uZ95tK92trGyl2DGcFNVXaVnH83snGMu9Z6e9ZeRyKdKcruMZNRV5WTdl1fQyqYacKcKklaNS+Xxyu17dCJxy/UL+L7JjChKtGpKVm0k4W2qZLNptJ6X8ixS+jylkfFlaUb/wCH3r93aN7tLPv4eOnKqYKtC6lTmrWb7ra7yWXXq7oxeGqq96c1lV33WrLq+i3FTX4h0n2H3G1OcpJJq1PuzbhKVoO937tnobvycS3rNd5p/m72tNxtvvpexyKcKrhaMZuDktotpyV7WfXvP5kPC1UnenNKL1vFpJ+PR6r5isv1Do0+xVPLlqTtNxtLhq0VKMJXm82nv28bGdfsJRhUkqknkjms6bjybd234f8AkoVOza8aed05ZbXejul3tWt0u69zTOlJTcGm5XtZatvkvEREz4yHV7P7HpyVOdSUnBrM0o2i7qVoqV91l1RrwGBoVIUJTbUpup3Um1LKk1rfSxz3hqqvenNWdn3Xo3y232M6eAryk4xpVLrdZWraX1vtoJx/9C32f2NxqKqOco3lGPuXVnUjBu99+9flt8TbT7HhkdTNUnF05ySyWs4wm05tS0V46dTmzwtWN+5KybV0m43S1122XoS8FWyp8ObUk3om3ZPdpbITE/qGyrgpcKjOFObzxeZpNq/ElFL5JaFWm45k5axur26cyz7JiIOKUKil7yUU29oyzWXhJFeVGcYqThJRl7rcWk/J8y8fuLWMwsKWJ4TfcWVOSfWMW5fN3HaWDVKpCmmr5I5m3pmektelymBGM9u6FntClClWcYXlBRg1f7V4Rb+bbGOoKOInTpxds1oxV5PVLTqysCYxGUqckruLSezastr/AHamBIJEAkAQCQBDKbLhTZg3viHXTQADz3QAAF4AEvSCCQEIBJAAAEoZUveXmemwclZXseboJZteh1MNQUZpqT1vu/Ark45+XRxUlyPP4+Pfb6l9YVJZs7+dyviKa4U5c7rfdkQrHmHOABd3ADZRlFSTnHNHpexAnDL85G3U6ioq9ozWa7drXVzRhqtBSlKN4u1lm1sJV6UE3HWT2/ErLllNy04ySguFDk+/L9KX4FMlsFnWIqFNkEsghgAABJdKRdPQ2P5v4ctT4AAeg5hlGVmmrXXVJr5PQxAFuhjnGM4uObNLMrN07StJfZ5Wk9NDCtiFKEYZLKF1B5r2Tm5a6a72K4I4xdjqw7bccr4azRVk87t7sYyuud1BeXjoKXbk4xy5E97NvVO83e9v9R9Nt9Wco9t9GPovhcVg4VqqnncpJ5Z2WkmlocdXp6eN5QmIt5vB9q8KlGnw1NRd7t+N9NNPu8Da+3p5oyjTisqslfT3YJaJJf5a+Z65fQ/AOrw8tW22biveydrW6M3/AJDYDpV/mM4TuNK7mJWjGZeHfbLtFRp2jHk5uTf+Ju//AJX8jTLHrjquoWlduavo73Wj3Wj36nvvyGwHSr/MY/IbAdKv8xiNzpR8ScJeIp9s5ctqSeVZVmqSfdzRbT6u8d/TRCp225ZFwkowurKVrp03Dkklo+SPb/kNgOlX+Yx+Q2A6Vf5jI+o0fUp4S8PLtnNLNKldtSi7VJJZZubattfvvX0NsPpA1JS4MXltlWa9rea/38T2f5DYDpV/mMfkNgOlX+Yx9Ro+pOEvDx7ZeVRdNNd1u02m3FU0nfkvza08TRje0pVotSSV3B6bdyMo6LxzHv8A8hsB0q/zGPyFwHSr/MZMbnRjvUnCXzhzhraDW9u9e17W5ctfO5qPpn5C4DpV/mM8D21hYUMXWpQvkhK0bu7tbqd9LXx1JrFWcZhRAB3VAAAAAAAACky6UmYN74h000AA851AABeABL0gAAAAEIALuC7Mq1tUssf0pbfDqSiZiPKpCLbsi5GWeK6fuuX3M6dPsqNGzu5S2btZfIoVqc6Um6PeVr2XJETDNlqRMtjqWjZ6LlZW9DmVqjbertfYsxz1Lyk7afA7eFpQlTjmjGzjonZkYwiM+MvLg9fLsyi1/hw+Cs/mijiOwINJ05OL6PvX6ItTpGtDzwLk+y8QpZXTd/Db5mtYKrdrI01vfRfMh05R7YUqLndJ2LbwTUHKo1pslsWMLSVOCU077u2rLcJU56PNbxVn/fxJplzyvK4edBtrwcZd7RvVrb0NRDXjNxamyCSCGAAAEl0pHS4R6Gx85fw5as1TUDbwRwT0LceUNQNvBHBFnKGoG3gjgizlDUfUPoL/AMNp/vT/AOtnzTgn0b6JT4fZKa3Tna2rvmdjJvO+nH7umnlFurH3YVOtW/wk3FejR0SpiKahh2ltTin/AA2f9Db7THicO/e9PK/Wyb+B5s9+8OkduzcAVa1WpxqcIJOOrqX3S1y2+KKxFrTNLQK2D7qlT2ySdl4N3X32+BZE9iJsABCQAAD5H9J/+IYn/wBz+iPrh8m+ktO/aGJf+p/RG3Zfjn9nLUmoccG3hDhHp248oagbeEOELOUNQNvCHCFnKGoG3hDhCzlDUUmdLhHNZg33iHXSm7QADznYAAF8gkgl6YCSAgAAHS7IwUajzT91PSP6TXU7faFaVLDzlH3krRtybdtEUezUo0oJ6d1yv43v9yOhhW5vPPde5G3uq2/m/wC+ZLDqZTOThdg4es6rqSzRi07uV+8+W+/mdmWCUtczi3vl2ZeSXmyEsrt11Qc1Ojgoq60a8tWcLt7BVYVeIlJwsssl9my28NdfiepS3MuQGnCTlKjTlLSUopvzsS5Ny7uy+80Opw3k+y33H0/Z/D5cjbGWWOpIT0V2/M0KMJpZvNfA5PaParlPLH3E9fEw+tlp3dvEJjGfS/Ogm8uzfu9H4FWvVqQTUoSjb7T/AKMiv2jTnHW/jBrXzUkc6ripyVnJuPJPVr4kWvjp5TLXUqOUnJ7swBJVtqlJkEsgPOAAAPRKkedPXqnobdpNWxbzLjX8qXBHBL3DI4Zt5MPVUuCOCXeGTwxyOqo8EcEvcMcMcjqqPBPd/RKk1haMo9Zxl5Zm0/n955HhnuPosrYOH70v+pmbc5fcadrlyzp15xTTT2asynWoRpxi4rVTi27atu0G31dmXTViabnTlFbtaefL1PPiXoTDaVsK1KdSotm1GL6qK3+bkV8TOdWnGVG8akJaxas1dWaafhK/iXqVNQioraKsiaqC7lqr0p5nOm45stmpLR2d1qtt38xHFLhZ2rW3S11Wjj89CwVng4urxHrs1F7KWzn52sv/ACImPkmJjwQm6ajxJNyqStbkm03ZeGljZXxEKaTm0ruyMMbh+JFWSzRkpQu7K6Ijh82Z1Um5Jxte6UXyXmO3mUd47Qs3K+NqSjTfDaVR2jC+qu3vbw3+BXqVb0ZxbvUpWclz7rupW8bXLPAhKoqt7tKy1056266v5iq8l34ZUqqlmXODyy87J/1PnH0gp3xtd/ts+gQprPiI/p2k/jHL/wBp4XtGOavOXWz9Eatt2ymf8Mu6zrGHI4I4Rd4Q4Ru5MPVUuEOEXuEOEOR1VHhDhF3hE8IcjqqPCHCL3CI4Q5HVUuCeeZ67hHkXuYt3N03bTLlaAAYm0AAF8AEvTCCQBABlDdeYRL0NOjem4rlBJfKzIxeMy0HNbzp28nb/AHNGB7RpupJLNtpZXvbexTxWLhPCyjZqUbaPxfL5/cS81xqbfWyN3lUfyZe+jOEp1sQ4VY5o5G7Xa1TXTzPTz7CwUb/m9tWs8r/eQPFqc1/mv1L+BxtGEJcaVSc76NSltZeK5npo9g4J7U0/+eX4nke3cPCliqtOCtGLVle+8U/6gd+hLPTpLW0mrX1dve38kWO06rSSjz08vEwhUpwdGN72um/0e7bUqdo1nOfdtKEVquuvLqWjuOFNWbRBYxuTOskZRVtpblcrLdhN4xIACFwkAJUmQSyA80AAA9yqeiPDnvXVhFayWxp2+UY3bzd/GU8Yxi/P9NfDHDK9XHX0ireL3ObisRWjJZJzSf7V1f4lst3jE1Hdyw+z9XLG5mnayDhlGni6ygk5Rcurj+BFfG1rXg436Nbj6vA/+frf4X+GOGUHipTV76eGhSxUJ3zwlJO2qTsUneRfaHTH7Nyrvl3dzIWqGOr045YVJRiuSPN4XtirBqM1nXyl/udd46FtFJ+Gx1jcaeUd2fLabjTyrGP5h0frXFfrpeg+tcV+ul6HFqYubejy+CRz8RjMRGTalJRvps0cvqdO/wALVGz3Ex+P/svU/WeJu3xZXe70vp8CfrXFfrpeh5zDdutaVY3/AGo6P5F+n2lTn7ju+j0fyL9bSq/6cJ2+7jLj3/26n1riv10vQfWmK/XS9DlxryaWaydtUtjn4OvWjXnHPeKbbvqvC3Q5/Uad/h7O/wBHr8b59/Xd6T61xX66XoPrXFfrpehzZ4txTeW9tbLc24fF0qiTjNXfJuz+R2w1NLPwyaunudLvldfuuLtPE3b4srvd6a+hjRx9enFRhVkorZGuVlu0jTUxCXu6vx2LZZ6eMd6c9KNfUn7trf1liM2biyva1/AqSi223u9WVaOKrZnnjBLk43Zq7Rr1MmaErOLvp0OX1WEeIa42Ovn+Of8Atr3DGQr9k9ocdOMrKoumzXVHS4TNGOpGUXDz9XHLSynHLyq5BkLfBfQcB9C3Jy6ipkGQt8CXQn2eXQcjqR7U8g4Zc9nl0Hs0uhHKDqR7U+GeGluz6L7NLofO5bvzM24m6er9m5coy/hiADK9UAAF8GHFj1HFj1Jejzx9swYcWPUcWPUHPH2yN2Gspxb2Wr09CpOsuT+JFOvlvzvvra/mHHU1Y8QtVaTp1uJTcbXzJJp2ve8XY2YOhTqNRqTcE29bq97Ky15aMpPEr9BfNmDrvloGR6HsKnSoY2SVVOKg7Sk0lrl0vtc9JOvQkrOpS1/bV/vPnUqt97fI13QtD6VHEUE9KlJeUonkO1IRq9pTSkmm07rvJ2gnbTyscTQyhUcXeLaa2admLS6+LpzdScsk4qUm9Ytbsz7RX5zVNLZXVtt16nJljqz3qzf/ADv8RLF1Je9OT1b35vdgWasdnyfO2nzNZreIbjlc243uk3p5kqrG24atLOKqWYMeLHqOLHqQ7c8fbIkw4seo4sepBzx9qrIJZBLzwAAD0CqHny97Wuvoc84mfC+MuvhmpVIRbspSSb82XPZqU4x78YyaWdOSeV93TW19Hf5rkec9rXX0Hta6+hTjPpa4ejwtKlKnBzspS/1EtMyTb6NK7S/Z8UYU4UZZUm22oaZlu5tS9LPwPP8AtcevoT7YuoqfRcOk5pN2d/Hr4kcQ5vta6+hPtkepHGfSbheqWla+62fQz4pzfbI9R7ZHqOM+i4dLiDiHN9sj1Htkeo4z6Lhsq0Wm7bGpOxPtceo9qj/aJqfRcLVHtGcdJd77zZQxsFdtNOTu3uUfao/2iPaY/wBoVPouHXjjYP7Xz0K2Iw0J6xaT9Chx4Djw5NoVKLh08F3E7q0r78/mWeMcRYu32mZLHePoOMlw7PGMatZZZX2s/uOT7cg8bF7/AHCpLhNHPFqcLpx1TR3sF2/GWlVZZdV7r/A4Pt0f7RhPEwe/3HTDPPCezPuNvpa8ff8A9vbRxUXtJfM2qqeA40erL2F7bnT0bzR6Pf5mnHXifMPJ1fsqovDK3tFVMlUPN0/pDRa1ck+lrm2P0hw/6T/hZ2vH2wZbLVj8svQqoZKZ59fSLDfpv+Fma+keG/Tf8LIvH25zs9b9M/6d/MfLZ7vzPar6SYX9N/ws8TJ6s46kx8PW+zNHPT5corx/aAAcnrAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA//Z\n",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"https://www.youtube.com/embed/uyW0pOAKRME\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x10765ce50>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_yotube_video(\"https://www.youtube.com/watch?v=uyW0pOAKRME\", width=800, height=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prosody Encoding. \n",
    "\n",
    "Given a spectorgam of a reference prosody, we want to extract prosodic information from a reference spectrogram but remove information about\n",
    "- lingustic content;\n",
    "- speaker;\n",
    "- room characteristics;\n",
    "- ignore noise or other background sounds;\n",
    "\n",
    "\n",
    "Gooo baseline of prosody encoder is a convolution encoder which acts on spectrograms and outputs prosody embeddings. $ Prosody=Enc({Spectro}) $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/enc_text_speaker_prosody.png\" style=\"height: 500px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StyleTokens\n",
    "\n",
    "- Compute linear combination of learnable style vectors  with outputs of reference encoder and treat this as a stile embedding.\n",
    "- Style vectors are learned in unsupervised way, via $ \\large {\\frac{\\partial{L}}{\\partial{t_n}}, n \\in \\{0 ... N\\}}$\n",
    "- This approch makes possible to specify desired weight for each of the learned style tokens.\n",
    "\n",
    "<img src=\"assets/style_tokens.png\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "class StyleTokenLayer(nn.Module):\n",
    "\n",
    "    def __init__(self, n_tokens=10, emb_dim=512):\n",
    "        super(StyleTokenLayer, self).__init__()\n",
    "\n",
    "        self.tokens = nn.Parameter(torch.rand(emb_dim,  n_tokens))\n",
    "        nn.init.xavier_uniform_(self.tokens.data, gain=nn.init.calculate_gain('linear'))\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "\n",
    "        Parameters\n",
    "        ----------\n",
    "        x: torch.Tensor of shape (n_batch, n_hidden)\n",
    "\n",
    "        Returns\n",
    "        -------\n",
    "\n",
    "        \"\"\"\n",
    "        # style dot-product attention.\n",
    "        input_norms = torch.norm (x, dim=1, keepdim=True)\n",
    "        token_norms = torch.norm (self.tokens, dim=0, keepdim=True)\n",
    "        token_weights = F.softmax((x @ self.tokens) / input_norms / token_norms / 0.2, dim=-1)\n",
    "        return token_weights @ self.tokens.t()\n",
    "\n",
    "    def inference_with_softmax_temperature(self, x, T=1.0):\n",
    "        # style dot-product attention.\n",
    "        input_norms = torch.norm (x, dim=1, keepdim=True)\n",
    "        token_norms = torch.norm (self.tokens, dim=0, keepdim=True)\n",
    "        token_weights = F.softmax((x @ self.tokens) / input_norms / token_norms / T, dim=-1)\n",
    "        return token_weights @ self.tokens.t()\n",
    "\n",
    "    def inference_conditioned_on_style(self, token_weights):\n",
    "        return token_weights @ self.tokens.t()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SpeechSplit\n",
    "\n",
    "Ellegant idea on how to dissentangle Content, Rhythm, Timber and Pitch. Using informatioinal bottlenecks and RandomResampling operations, which by itself is a data augmentation! [SpeechSplit Demo](https://auspicious3000.github.io/SpeechSplit-Demo/)\n",
    "\n",
    "\n",
    "<img src=\"assets/speechflow.png\" alt=\"speechflow\" style=\"height: 400px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Zero speech chanllange\n",
    "\n",
    "https://zerospeech.com/\n",
    "\n",
    "- also known as TTS without Text\n",
    "- goal is to learn model which is able to speak in unknown language with data available only for infant\n",
    "- pushes the boundaries of representation learning approaches\n",
    "- important for low resource languages\n",
    "\n",
    "## Applications: \n",
    "- keyword search;\n",
    "- document classification;\n",
    "- image retrieval from speech;\n",
    "- speech to speech translation;\n",
    "\n",
    "<img src=\"assets/zero-speech.png\" style=\"height: 500px\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Open questions in TTS modelling.\n",
    "1. How can we reduce the amount of training data necessary for training TTS models?\n",
    "2. Training on data from real life, with bacgkround noises, low quality audio.\n",
    "3. Effective dissentangling of audio data into set of independent features.\n",
    "4. learning better universal graphical representaiton then text or phonemes."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}