01_spacy_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## spaCy &mdash; Industrial-Strength NLP in Python\n",
    "Tutorial sources: [here](https://nlpforhackers.io/complete-guide-to-spacy/) and [here](https://github.com/explosion/spacy-notebooks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[**spaCy**](https://spacy.io) is an industrial-strength natural language processing (_NLP_) library for Python. spaCy's goal is to take recent advancements in natural language processing out of research papers and put them in the hands of users to build production software.\n",
    "\n",
    "![spaCy](images/spacy.png)\n",
    "\n",
    "### It's really FAST ###\n",
    "Written in Cython, it was specifically designed to be as fast as possible\n",
    "\n",
    "### It's really ACCURATE ###\n",
    "spaCy implementation of its dependency parser is one of the best-performing in the world:\n",
    "[It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool](https://aclweb.org/anthology/P/P15/P15-1038.pdf)\n",
    "\n",
    "### Batteries included ###\n",
    "- **Index preserving tokenization** (details about this later)\n",
    "- **Models** for Part Of Speech tagging, Named Entity Recognition and Dependency Parsing\n",
    "- Supports **8 languages** out of the box\n",
    "- Easy and **beautiful visualizations**\n",
    "- Pretrained **word vectors**\n",
    "\n",
    "### Extensible ###\n",
    "It plays nicely with all the other already existing tools that you know and love: **Scikit-Learn**, **TensorFlow**, **gensim**\n",
    "\n",
    "### DeepLearning Ready ###\n",
    "It also has its own deep learning framework that’s especially designed for NLP tasks:\n",
    "[Thinc](https://github.com/explosion/thinc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "```bash\n",
    "# Install spaCy\n",
    "pip install spacy\n",
    "\n",
    "# Install spaCy English model\n",
    "python -m spacy download en\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load spaCy resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import spacy and English models\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Language Processing Pipelines\n",
    "\n",
    "- When you call nlp on a text, spaCy first **tokenizes** the text to produce a **Doc object**.\n",
    "- The Doc is then processed in several different steps – this is also referred to as the **processing pipeline**. \n",
    "- The pipeline used by the default models consists of a **tagger**, a **parser** and an **entity recognizer**.\n",
    "\n",
    "![pipeline](images/pipeline.svg)\n",
    "\n",
    "More info [here](https://spacy.io/usage/processing-pipelines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization\n",
    "The **Token** class exposes a lot of word-level attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Next\t0\tnext\tFalse\tFalse\tXxxx\tADJ\tJJ\n",
      "week\t5\tweek\tFalse\tFalse\txxxx\tNOUN\tNN\n",
      "I\t10\t-PRON-\tFalse\tFalse\tX\tPRON\tPRP\n",
      "'ll\t11\twill\tFalse\tFalse\t'xx\tVERB\tMD\n",
      "  \t15\t  \tFalse\tTrue\t  \tSPACE\t_SP\n",
      "be\t17\tbe\tFalse\tFalse\txx\tVERB\tVB\n",
      "in\t20\tin\tFalse\tFalse\txx\tADP\tIN\n",
      "Madrid\t23\tmadrid\tFalse\tFalse\tXxxxx\tPROPN\tNNP\n",
      ".\t29\t.\tTrue\tFalse\t.\tPUNCT\t.\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Next week I'll   be in Madrid.\")\n",
    "for token in doc:\n",
    "    print(\"{0}\\t{1}\\t{2}\\t{3}\\t{4}\\t{5}\\t{6}\\t{7}\".format(\n",
    "        token.text,\n",
    "        token.idx,\n",
    "        token.lemma_,\n",
    "        token.is_punct,\n",
    "        token.is_space,\n",
    "        token.shape_,\n",
    "        token.pos_,\n",
    "        token.tag_\n",
    "    ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Garfield is a cat.\n",
      "Snoopy is a dog.\n"
     ]
    }
   ],
   "source": [
    "# Print sentences (one sentence per line)\n",
    "doc = nlp(\"Garfield is a cat. Snoopy is a dog.\")\n",
    "\n",
    "for sent in doc.sents:\n",
    "    print(sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part Of Speech Tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Garfield\tgarfield\tPROPN\tNNP\tnsubj\tXxxxx\tTrue\tFalse\n",
      "is\tbe\tVERB\tVBZ\tROOT\txx\tTrue\tTrue\n",
      "a\ta\tDET\tDT\tdet\tx\tTrue\tTrue\n",
      "cat\tcat\tNOUN\tNN\tattr\txxx\tTrue\tFalse\n",
      ".\t.\tPUNCT\t.\tpunct\t.\tFalse\tFalse\n",
      "Snoopy\tsnoopy\tNOUN\tNN\tnsubj\tXxxxx\tTrue\tFalse\n",
      "is\tbe\tVERB\tVBZ\tROOT\txx\tTrue\tTrue\n",
      "a\ta\tDET\tDT\tdet\tx\tTrue\tTrue\n",
      "dog\tdog\tNOUN\tNN\tattr\txxx\tTrue\tFalse\n",
      ".\t.\tPUNCT\t.\tpunct\t.\tFalse\tFalse\n"
     ]
    }
   ],
   "source": [
    "# For each token, print corresponding part of speech tag\n",
    "for token in doc:\n",
    "    print('%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s\\t%s' % (token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "          token.shape_, token.is_alpha, token.is_stop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Named Entity Recognition\n",
    "Doing NER with spaCy is super easy and the pretrained model performs pretty well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Next week\t-->\tDATE\n",
      "London\t-->\tGPE\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Next week I'll be in London.\")\n",
    "for ent in doc.ents:\n",
    "    print(\"%s\\t-->\\t%s\" % (ent.text, ent.label_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Common entity types include *ORGANIZATION*, *PERSON*, *LOCATION*, *DATE*, *TIME*, *MONEY*, and *GPE* (geo-political entity). See complete list [here](https://spacy.io/usage/linguistic-features#entity-types)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\t-->\tCARDINAL\n",
      "9 a.m.\t-->\tTIME\n",
      "30%\t-->\tPERCENT\n",
      "just 2 days\t-->\tDATE\n",
      "WSJ\t-->\tORG\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"I just bought 2 shares at 9 a.m. because the stock went up 30% in just 2 days according to the WSJ\")\n",
    "for ent in doc.ents:\n",
    "     print(\"%s\\t-->\\t%s\" % (ent.text, ent.label_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**displaCy** comes in handy for a better visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"entities\" style=\"line-height: 2.5\">But \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Google\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " is starting from behind. The company made a late push into hardware, and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Apple\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "’s Siri, available on \n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    iPhones\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       ", and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Amazon\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       "’s \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Alexa\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " software, which runs on its \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Echo\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em; box-decoration-break: clone; -webkit-box-decoration-break: clone\">\n",
       "    Dot\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " devices, have clear leads in consumer adoption.</div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "text = \"\"\"But Google is starting from behind. The company made a late push into hardware, and Apple’s Siri, available on iPhones, and Amazon’s Alexa software, which runs on its Echo and Dot devices, have clear leads in consumer adoption.\"\"\"\n",
    "\n",
    "doc = nlp(text)\n",
    "displacy.render(doc, style='ent', jupyter=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chunking\n",
    "spaCy automatically detects noun-phrases as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall Street Journal \t NP \t Journal\n",
      "an interesting piece \t NP \t piece\n",
      "crypto currencies \t NP \t currencies\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(\"Wall Street Journal just published an interesting piece on crypto currencies\")\n",
    "for chunk in doc.noun_chunks:\n",
    "    print(chunk.text, '\\t', chunk.label_, '\\t', chunk.root.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the chunker also computes the *root* of the phrase, the main word of the phrase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dependency Parsing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall/NNP<--compound-- Street/NNP\n",
      "Street/NNP<--compound-- Journal/NNP\n",
      "Journal/NNP<--nsubj-- published/VBD\n",
      "just/RB<--advmod-- published/VBD\n",
      "published/VBD<--ROOT-- published/VBD\n",
      "an/DT<--det-- piece/NN\n",
      "interesting/JJ<--amod-- piece/NN\n",
      "piece/NN<--dobj-- published/VBD\n",
      "on/IN<--prep-- piece/NN\n",
      "crypto/JJ<--compound-- currencies/NNS\n",
      "currencies/NNS<--pobj-- on/IN\n"
     ]
    }
   ],
   "source": [
    "doc = nlp('Wall Street Journal just published an interesting piece on crypto currencies')\n",
    " \n",
    "for token in doc:\n",
    "    print(\"{0}/{1}<--{2}-- {3}/{4}\".format(\n",
    "        token.text, token.tag_, token.dep_, token.head.text, token.head.tag_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this doesn’t help visualizing the dependency tree, **displaCy** comes in handy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<svg xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" id=\"662-0\" class=\"displacy\" width=\"1370\" height=\"317.0\" style=\"max-width: none; height: 317.0px; color: #000000; background: #ffffff; font-family: Arial\">\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"50\">Wall</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"50\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"170\">Street</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"170\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"290\">Journal</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"290\">PROPN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"410\">just</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"410\">ADV</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"530\">published</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"530\">VERB</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"650\">an</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"650\">DET</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"770\">interesting</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"770\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"890\">piece</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"890\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1010\">on</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1010\">ADP</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1130\">crypto</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1130\">ADJ</tspan>\n",
       "</text>\n",
       "\n",
       "<text class=\"displacy-token\" fill=\"currentColor\" text-anchor=\"middle\" y=\"227.0\">\n",
       "    <tspan class=\"displacy-word\" fill=\"currentColor\" x=\"1250\">currencies</tspan>\n",
       "    <tspan class=\"displacy-tag\" dy=\"2em\" fill=\"currentColor\" x=\"1250\">NOUN</tspan>\n",
       "</text>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-0\" stroke-width=\"2px\" d=\"M70,182.0 C70,122.0 160.0,122.0 160.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-0\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M70,184.0 L62,172.0 78,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-1\" stroke-width=\"2px\" d=\"M190,182.0 C190,122.0 280.0,122.0 280.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-1\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M190,184.0 L182,172.0 198,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-2\" stroke-width=\"2px\" d=\"M310,182.0 C310,62.0 525.0,62.0 525.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-2\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">nsubj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M310,184.0 L302,172.0 318,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-3\" stroke-width=\"2px\" d=\"M430,182.0 C430,122.0 520.0,122.0 520.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-3\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">advmod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M430,184.0 L422,172.0 438,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-4\" stroke-width=\"2px\" d=\"M670,182.0 C670,62.0 885.0,62.0 885.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-4\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">det</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M670,184.0 L662,172.0 678,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-5\" stroke-width=\"2px\" d=\"M790,182.0 C790,122.0 880.0,122.0 880.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-5\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">amod</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M790,184.0 L782,172.0 798,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-6\" stroke-width=\"2px\" d=\"M550,182.0 C550,2.0 890.0,2.0 890.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-6\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">dobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M890.0,184.0 L898.0,172.0 882.0,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-7\" stroke-width=\"2px\" d=\"M910,182.0 C910,122.0 1000.0,122.0 1000.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-7\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">prep</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1000.0,184.0 L1008.0,172.0 992.0,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-8\" stroke-width=\"2px\" d=\"M1150,182.0 C1150,122.0 1240.0,122.0 1240.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-8\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">compound</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1150,184.0 L1142,172.0 1158,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "\n",
       "<g class=\"displacy-arrow\">\n",
       "    <path class=\"displacy-arc\" id=\"arrow-662-0-9\" stroke-width=\"2px\" d=\"M1030,182.0 C1030,62.0 1245.0,62.0 1245.0,182.0\" fill=\"none\" stroke=\"currentColor\"/>\n",
       "    <text dy=\"1.25em\" style=\"font-size: 0.8em; letter-spacing: 1px\">\n",
       "        <textPath xlink:href=\"#arrow-662-0-9\" class=\"displacy-label\" startOffset=\"50%\" fill=\"currentColor\" text-anchor=\"middle\">pobj</textPath>\n",
       "    </text>\n",
       "    <path class=\"displacy-arrowhead\" d=\"M1245.0,184.0 L1253.0,172.0 1237.0,172.0\" fill=\"currentColor\"/>\n",
       "</g>\n",
       "</svg>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "doc = nlp('Wall Street Journal just published an interesting piece on crypto currencies')\n",
    "displacy.render(doc, style='dep', jupyter=True, options={'distance': 120})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Vectors\n",
    "spaCy comes shipped with a Word Vector model as well. We’ll need to download a larger model for that:\n",
    "```bash\n",
    "python -m spacy download en_core_web_lg\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 2.0228e-01 -7.6618e-02  3.7032e-01  3.2845e-02 -4.1957e-01  7.2069e-02\n",
      " -3.7476e-01  5.7460e-02 -1.2401e-02  5.2949e-01 -5.2380e-01 -1.9771e-01\n",
      " -3.4147e-01  5.3317e-01 -2.5331e-02  1.7380e-01  1.6772e-01  8.3984e-01\n",
      "  5.5107e-02  1.0547e-01  3.7872e-01  2.4275e-01  1.4745e-02  5.5951e-01\n",
      "  1.2521e-01 -6.7596e-01  3.5842e-01 -4.0028e-02  9.5949e-02 -5.0690e-01\n",
      " -8.5318e-02  1.7980e-01  3.3867e-01  1.3230e-01  3.1021e-01  2.1878e-01\n",
      "  1.6853e-01  1.9874e-01 -5.7385e-01 -1.0649e-01  2.6669e-01  1.2838e-01\n",
      " -1.2803e-01 -1.3284e-01  1.2657e-01  8.6723e-01  9.6721e-02  4.8306e-01\n",
      "  2.1271e-01 -5.4990e-02 -8.2425e-02  2.2408e-01  2.3975e-01 -6.2260e-02\n",
      "  6.2194e-01 -5.9900e-01  4.3201e-01  2.8143e-01  3.3842e-02 -4.8815e-01\n",
      " -2.1359e-01  2.7401e-01  2.4095e-01  4.5950e-01 -1.8605e-01 -1.0497e+00\n",
      " -9.7305e-02 -1.8908e-01 -7.0929e-01  4.0195e-01 -1.8768e-01  5.1687e-01\n",
      "  1.2520e-01  8.4150e-01  1.2097e-01  8.8239e-02 -2.9196e-02  1.2151e-03\n",
      "  5.6825e-02 -2.7421e-01  2.5564e-01  6.9793e-02 -2.2258e-01 -3.6006e-01\n",
      " -2.2402e-01 -5.3699e-02  1.2022e+00  5.4535e-01 -5.7998e-01  1.0905e-01\n",
      "  4.2167e-01  2.0662e-01  1.2936e-01 -4.1457e-02 -6.6777e-01  4.0467e-01\n",
      " -1.5218e-02 -2.7640e-01 -1.5611e-01 -7.9198e-02  4.0037e-02 -1.2944e-01\n",
      " -2.4090e-04 -2.6785e-01 -3.8115e-01 -9.7245e-01  3.1726e-01 -4.3951e-01\n",
      "  4.1934e-01  1.8353e-01 -1.5260e-01 -1.0808e-01 -1.0358e+00  7.6217e-02\n",
      "  1.6519e-01  2.6526e-04  1.6616e-01 -1.5281e-01  1.8123e-01  7.0274e-01\n",
      "  5.7956e-03  5.1664e-02 -5.9745e-02 -2.7551e-01 -3.9049e-01  6.1132e-02\n",
      "  5.5430e-01 -8.7997e-02 -4.1681e-01  3.2826e-01 -5.2549e-01 -4.4288e-01\n",
      "  8.2183e-03  2.4486e-01 -2.2982e-01 -3.4981e-01  2.6894e-01  3.9166e-01\n",
      " -4.1904e-01  1.6191e-01 -2.6263e+00  6.4134e-01  3.9743e-01 -1.2868e-01\n",
      " -3.1946e-01 -2.5633e-01 -1.2220e-01  3.2275e-01 -7.9933e-02 -1.5348e-01\n",
      "  3.1505e-01  3.0591e-01  2.6012e-01  1.8553e-01 -2.4043e-01  4.2886e-02\n",
      "  4.0622e-01 -2.4256e-01  6.3870e-01  6.9983e-01 -1.4043e-01  2.5209e-01\n",
      "  4.8984e-01 -6.1067e-02 -3.6766e-01 -5.5089e-01 -3.8265e-01 -2.0843e-01\n",
      "  2.2832e-01  5.1218e-01  2.7868e-01  4.7652e-01  4.7951e-02 -3.4008e-01\n",
      " -3.2873e-01 -4.1967e-01 -7.5499e-02 -3.8954e-01 -2.9622e-02 -3.4070e-01\n",
      "  2.2170e-01 -6.2856e-02 -5.1903e-01 -3.7774e-01 -4.3477e-03 -5.8301e-01\n",
      " -8.7546e-02 -2.3929e-01 -2.4711e-01 -2.5887e-01 -2.9894e-01  1.3715e-01\n",
      "  2.9892e-02  3.6544e-02 -4.9665e-01 -1.8160e-01  5.2939e-01  2.1992e-01\n",
      " -4.4514e-01  3.7798e-01 -5.7062e-01 -4.6946e-02  8.1806e-02  1.9279e-02\n",
      "  3.3246e-01 -1.4620e-01  1.7156e-01  3.9981e-01  3.6217e-01  1.2816e-01\n",
      "  3.1644e-01  3.7569e-01 -7.4690e-02 -4.8480e-02 -3.1401e-01 -1.9286e-01\n",
      " -3.1294e-01 -1.7553e-02 -1.7514e-01 -2.7587e-02 -1.0000e+00  1.8387e-01\n",
      "  8.1434e-01 -1.8913e-01  5.0999e-01 -9.1960e-03 -1.9295e-03  2.8189e-01\n",
      "  2.7247e-02  4.3409e-01 -5.4967e-01 -9.7426e-02 -2.4540e-01 -1.7203e-01\n",
      " -8.8650e-02 -3.0298e-01 -1.3591e-01 -2.7765e-01  3.1286e-03  2.0556e-01\n",
      " -1.5772e-01 -5.2308e-01 -6.4701e-01 -3.7014e-01  6.9393e-02  1.1401e-01\n",
      "  2.7594e-01 -1.3875e-01 -2.7268e-01  6.6891e-01 -5.6454e-02  2.4017e-01\n",
      " -2.6730e-01  2.9860e-01  1.0083e-01  5.5592e-01  3.2849e-01  7.6858e-02\n",
      "  1.5528e-01  2.5636e-01 -1.0772e-01 -1.2359e-01  1.1827e-01 -9.9029e-02\n",
      " -3.4328e-01  1.1502e-01 -3.7808e-01 -3.9012e-02 -3.4593e-01 -1.9404e-01\n",
      " -3.3580e-01 -6.2334e-02  2.8919e-01  2.8032e-01 -5.3741e-01  6.2794e-01\n",
      "  5.6955e-02  6.2147e-01 -2.5282e-01  4.1670e-01 -1.0108e-02 -2.5434e-01\n",
      "  4.0003e-01  4.2432e-01  2.2672e-01  1.7553e-01  2.3049e-01  2.8323e-01\n",
      "  1.3882e-01  3.1218e-03  1.7057e-01  3.6685e-01  2.5247e-03 -6.4009e-01\n",
      " -2.9765e-01  7.8943e-01  3.3168e-01 -1.1966e+00 -4.7156e-02  5.3175e-01]\n"
     ]
    }
   ],
   "source": [
    "nlp = spacy.load('en_core_web_lg')\n",
    "print(nlp.vocab['banana'].vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The closest word vector in the vocabulary for “man” – “king” + “woman” is still “King” but “Queen” comes right after :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing Similarity\n",
    "\n",
    "Based on the word embeddings, spaCy offers a similarity interface for all of it’s building blocks: Token, Span, Doc and Lexeme. Here’s how to use that similarity interface:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'dog' is 0.66185343 similar to 'animal' and 0.23552851 similar to 'fruit'\n",
      "'banana' is 0.24272855 similar to 'animal' and 0.67148364 similar to 'fruit'\n"
     ]
    }
   ],
   "source": [
    "banana = nlp.vocab['banana']\n",
    "dog = nlp.vocab['dog']\n",
    "fruit = nlp.vocab['fruit']\n",
    "animal = nlp.vocab['animal']\n",
    " \n",
    "print(\"'dog' is %s similar to 'animal' and %s similar to 'fruit'\" % (dog.similarity(animal), dog.similarity(fruit)))\n",
    "print(\"'banana' is %s similar to 'animal' and %s similar to 'fruit'\" % (banana.similarity(animal), banana.similarity(fruit)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s now use this technique on entire texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8901766262114666\n",
      "0.9115828449161616\n",
      "0.7822956256736615\n",
      "0.7133323899064792\n",
      "0.6526212010025575\n"
     ]
    }
   ],
   "source": [
    "target = nlp(\"Cats are beautiful animals.\")\n",
    " \n",
    "doc1 = nlp(\"Dogs are awesome.\")\n",
    "doc2 = nlp(\"Some gorgeous creatures are felines.\")\n",
    "doc3 = nlp(\"Dolphins are swimming mammals.\")\n",
    "doc4 = nlp(\"Snoopy is a very smart dog.\")\n",
    "doc5 = nlp(\"Tomorrow it will rain a lot in Berlin.\")\n",
    " \n",
    "print(target.similarity(doc1))\n",
    "print(target.similarity(doc2))\n",
    "print(target.similarity(doc3))\n",
    "print(target.similarity(doc4))\n",
    "print(target.similarity(doc5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"king\" - \"man\" + \"woman\" = \"queen\"?\n",
    "There’s a really famous example of word embedding math: \"king\" - \"man\" + \"woman\" = \"queen\". Let’s test that out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['King', 'KING', 'king', 'KIng', 'Queen', 'QUEEN', 'queen', 'Prince', 'PRINCE', 'prince']\n"
     ]
    }
   ],
   "source": [
    "from scipy import spatial\n",
    " \n",
    "cosine_similarity = lambda x, y: 1 - spatial.distance.cosine(x, y)\n",
    " \n",
    "man = nlp.vocab['man'].vector\n",
    "woman = nlp.vocab['woman'].vector\n",
    "queen = nlp.vocab['queen'].vector\n",
    "king = nlp.vocab['king'].vector\n",
    " \n",
    "# We now need to find the closest vector in the vocabulary to the result of \"man\" - \"woman\" + \"queen\"\n",
    "maybe_queen = king - man + woman\n",
    "\n",
    "computed_similarities = []\n",
    " \n",
    "for word in nlp.vocab:\n",
    "    if word.has_vector:  # Ignore words without vectors\n",
    "        similarity = cosine_similarity(maybe_queen, word.vector)\n",
    "        computed_similarities.append((word, similarity))\n",
    " \n",
    "computed_similarities = sorted(computed_similarities, key=lambda item: -item[1])\n",
    "print([w[0].text for w in computed_similarities[:10]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "# Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. How many sentences are there in the following text?\n",
    "\n",
    "*Hint: doc.sents is not a list, but a 'generator'. Convert it to a list first!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print sentences (one sentence per line)\n",
    "text = \"The path of the righteous man is beset on all sides by the iniquities of the selfish and the tyranny of evil men. Blessed is he who, in the name of charity and good will, shepherds the weak through the valley of darkness, for he is truly his brother's keeper and the finder of lost children. And I will strike down upon thee with great vengeance and furious anger those who would attempt to poison and destroy My brothers. And you will know My name is the Lord when I lay My vengeance upon thee.\"\n",
    "\n",
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Print the sentences (one sentence per line, preceeded by the sentence number) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Do you want to learn more?\n",
    "\n",
    "Visit [spaCy website](https://spacy.io/)!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}