W7D1-Exercise_sol.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise W7D1: Review and Putting it All Together\n",
    "\n",
    "This exercise aims to draw together the topics we have covered in the _Base Camp_ portion of the Digital Methods class. At the end of the exercise, you should have a `DataFrame` with each row containing information on a Twitter account including their tweets, friends, followers, hashtags and mentions as well as some descriptive statistics.\n",
    "\n",
    "You will be able to reuse and modify this code for the second half of digital methods to download and analyze tweets for your projects. So, this exercise should provide you with a solid review of different things we have learned and help you for the the rest of the course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1. Identify a topic, authenticate, and get data.** First, identify a topic of interest to you and think about a keyword or hashtag capturing the topic. Possible topics could be Corona or Climate, but you are welcome to choose something else. Then load the `tweepy` module and use the built-in functionality to `search` Twitter for your keyword or hashtag. Create a variable that contains the data returned by your search.\n",
    "\n",
    "See [here](http://docs.tweepy.org/en/latest/api.html#help-methods) for more information about the `search` method."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have an object containing a number of tweets pertaining to our topic of interest. As you might remember, by default the Twitter API returns the data to us in JSON format. Now that we know about the elegance and beauty of `DataFrames`, we would prefer to work with that format of data rather than a dictionary-style JSON. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2. Turn raw API data into a DataFrame.** Your search returned a set of tweets about your chosen topic. Construct a `DataFrame` from your Twitter search object of the people who are tweeting about that topic that, at minimum, contains the unique `screen_names`, `followers_count`, `friends_count`, and `statuses_count` returned from your search. \n",
    "\n",
    "There are a number of ways to do this so you might want to review how to construct `DataFrames` (W6D1-Demo). You may also want to review navigating JSON objects (W4D2-Exercise_solutions). Also, your returned data might include the same account multiple times, so you will want to make sure that you are listing the account only once in your `DataFrame`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our neat `DataFrame` we can now easily find out details about the data we collected from Twitter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3. Get information about our data.** Use the `print` function and string operations to make Python tell you in plain language: a) How many unique accounts there are in your data, b) what the name of the _last_ account in your data is, and c) what the sum of followers is for all accounts in your data. That is make Python print out full sentences with the relevant information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4. Adding data to our DataFrame.** Loop through the indices of your `DataFrame`, collect the timeline for each account using the `user_timeline` method from tweepy, and store them in a new list \"timelines\". Note that you will want to build in some `sleep` time to avoid running into rate limits. You can find the syntax for how to do this on page 155 and the logic and examples on pages 209-12 in Brooker (2020)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add your list \"timelines\" to your current `DataFame`. To do this, we first need to turn our list into a new `DateFrame` with one column labeled `timelines` and then join our two `DateFrames` horizontally, i.e. along `axis = 1`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a deep breath. This was a major piece of coding. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you have the timeline, that is the statuses, for each of your accounts in the `DataFrame`. But these are still in the raw format which the Twitter API returns, so we need to transform them into a format that allows us to to work with them more easily. In the end, we want to get at the the topics and persons our accounts tweet about."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5. Getting the tweet texts from the timeline.** Extract the text from the tweets in each account's timeline, combine them into a list, turn the list of lists into a `DataFrame`, and join the new and old `DataFrames`. One way to do this is to 1) create an empty list 'tweets', 2) loop through the indices in your `DataFrame`, 3) for each index/row loop through the timeline, 4) create a temporary list, append the text for each timeline element to that list, then append the temporary list to 'tweets' 5) turn the list into a `DataFrame` and 6) merge the two `DataFrames` horizontally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6. Turning our list of tweet texts into a long string.** To get a sense of what our accounts usually tweet about, it might be useful to have their tweets in one long string that allows us to easily count the words they use. Create a list that holds the long string of tweets for each user. We can concatenate our list of tweets/strings using the [join](https://docs.python.org/2/library/string.html#string.join) command for which you can find a usage example [here](https://stackoverflow.com/a/493842).\n",
    "\n",
    "Turn the list into a `DataFrame` and merge it with your main `DataFrame` horizontally. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 7. Finding hashtags and mentions.** Now that we have all the tweets for each account in one long string, we can start looking at the topics the accounts are tweeting about and who they are interacting with. To do so, you can use the [`findall`](https://docs.python.org/3/library/re.html#re.findall) function from the `re` package to extract all hashtags (starting with a \"#\") and mentions (starting with an \"@\"). Add one column for hashtags and mentions respectively to your `DataFrame`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 8. Writing your insights to a file.** You have just generated some really awesome insights about the accounts you identified earlier. To share your insights, that is the topics/hashtags your accounts tweet about, you should now write the hashtags to a text file–if you want to remind yourself, we covered this in week 3 day 1. Can you make it so the text file first lists the name and then the hashtags the account uses?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9. Descriptive statistics about your accounts.** We closed last week with talking about descriptive statistics. For the accounts you gathered, there are at least three variables that you might be interested to know more about. What are the minimum, maximum, and mean for the number of followers, friends, and posted statuses in you data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 10. Visualizing influence.** To round off this exercise, let's plot some data from the accounts you collected. Make a bar plot to show which of the accounts has the most influence on Twitter. _Hint:_ You might want to look at `followers_count`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 11. Understanding influence.** Now that you know who is most influential among your accounts, try to see if the data you get from Twitter allows you to explore what might explain that influence. Look into your data and plot the follower count against another variable. Is there a pattern?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**THERE IS ALWAYS MORE.** If you got all the way through this exercise and are still hungry for more, here are some suggestions for other things you could do:\n",
    "\n",
    "1. To get an even better sense of what your accounts tweet about than just using hashtags, you could count the most used words. Create a list that, for each account has a dictionary of the frequency of each word with stop words removed. Remember, you can reuse your code from W3D1. You can get a list of stop words from [here](http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words). These are also stored in `stop_words.txt`. Add a column to your dataframe for most used words. \n",
    "2. Extract the number of favorites and retweets from the timelines you gathered. Is there any relationship between the number of followers and these figures? How about between these figures and the number of friends?\n",
    "3. Researchers often use Twitter because we can do respondent-driven sampling, i.e. we start with a few accounts and then collect the accounts that follow these accounts to get a broader picture of the network. Start exploring the networks of the accounts you collected using the [`followers`](https://tweepy.readthedocs.io/en/latest/api.html#API.followers) command.\n",
    "4. Given that the accounts you collected are similar in that they tweet about your topic of choice, it might be interesting know if there are issues that distinguish the accounts. Researchers often use term frequency-inverse document frequency to study such differences. [Here](https://www.freecodecamp.org/news/how-to-process-textual-data-using-tf-idf-in-python-cd2bbc0a94a3/) is a primer on the concept and a tutorial on how to implement it in Python. Can you find distinguishes your accounts from one another?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EXAMPLE SOLUTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 1**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we start by importing the tweepy module and loading our twitter credentials\n",
    "# remember that the AppCred file needs to be in the same folder as your notebook\n",
    "import tweepy\n",
    "from AppCred import CONSUMER_KEY, CONSUMER_SECRET\n",
    "from AppCred import ACCESS_TOKEN, ACCESS_TOKEN_SECRET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# then we set up our authentication to the twitter api\n",
    "# and create an object that allows us to interface with the api\n",
    "auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)\n",
    "auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)\n",
    "api = tweepy.API(auth)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now, we can use the `search` method to look around twitter for our keyword\n",
    "# and store the returned data in an object we call `euro2020_tweets`\n",
    "e2020_tweets = api.search(\"#EURO2020\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = []\n",
    "screen_names = []\n",
    "id_str = []\n",
    "location = []\n",
    "description = []\n",
    "followers_count = []\n",
    "friends_count = []\n",
    "statuses_count = []\n",
    "created_at = []\n",
    "\n",
    "\n",
    "for tweet in e2020_tweets:\n",
    "    if (tweet._json['user']['screen_name'] not in screen_names):\n",
    "        names.append(tweet._json['user']['name'])\n",
    "        screen_names.append(tweet._json['user']['screen_name'])\n",
    "        id_str.append(tweet._json['user']['id'])\n",
    "        location.append(tweet._json['user']['location'])\n",
    "        description.append(tweet._json['user']['description'])\n",
    "        followers_count.append(tweet._json['user']['followers_count'])\n",
    "        friends_count.append(tweet._json['user']['friends_count'])\n",
    "        statuses_count.append(tweet._json['user']['statuses_count'])\n",
    "        created_at.append(tweet._json['user']['created_at'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e2020_dictionary = {\n",
    "    \"name\": names,\n",
    "    \"screen_name\": screen_names,\n",
    "    \"id_str\": id_str,\n",
    "    \"location\": location,\n",
    "    \"description\": description,\n",
    "    \"followers_count\": followers_count,\n",
    "    \"friends_count\": friends_count,\n",
    "    \"statuses_count\": statuses_count,\n",
    "    \"created_at\": created_at\n",
    "}\n",
    "\n",
    "e2020_df = pd.DataFrame(e2020_dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"a) There are \" + str(e2020_df.name.count()) + \" unique accounts in the data.\")\n",
    "print(\"b) The last account in my data is called \" + str(e2020_df.name[12]))\n",
    "print(\"c) The accounts in my data have a total of \" + str(sum(e2020_df.followers_count)) + \" followers.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 4**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "timeline = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    statuses = api.user_timeline(e2020_df.screen_name[i])\n",
    "    timeline.append(statuses)\n",
    "    time.sleep(5)\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "timelines = pd.DataFrame({\"timelines\": timeline})\n",
    "e2020_df = pd.concat([e2020_df, timelines], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 5**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    tweets_no = len(e2020_df.timelines[i])\n",
    "    temp_text = []\n",
    "    for j in range(0,tweets_no):\n",
    "        temp_text.append(e2020_df.timelines[i][j]._json[\"text\"])\n",
    "#         print(j)\n",
    "    tweets.append(temp_text)\n",
    "    print(i)\n",
    "\n",
    "tweets = pd.DataFrame({\"tweets\": tweets})\n",
    "\n",
    "e2020_df = pd.concat([e2020_df, tweets], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 6**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    text.append(\" \".join(e2020_df['tweets'][i]))\n",
    "\n",
    "text = pd.DataFrame({\"text\": text})\n",
    "\n",
    "e2020_df = pd.concat([e2020_df, text], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 7**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mentions = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    mentions.append(re.findall('@\\S+', e2020_df['text'][i]))\n",
    "\n",
    "mentions = pd.DataFrame({'mentions': mentions})\n",
    "\n",
    "e2020_df = pd.concat([e2020_df, mentions], axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hashtags = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    hashtags.append(re.findall('#\\S+', e2020_df['text'][i]))\n",
    "\n",
    "hashtags = pd.DataFrame({'hashtags': hashtags})\n",
    "\n",
    "e2020_df = pd.concat([e2020_df, hashtags], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 8**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Open the file in write mode\n",
    "f = open('hashtags.txt','w', encoding = \"utf-8\")\n",
    "\n",
    "#Loop through each tuple and write it as a string, with a new line\n",
    "for i in e2020_df.index:\n",
    "    f.write(str(e2020_df[\"name\"][i]) + \": \" + str(e2020_df[\"hashtags\"][i]) + \"\\n\")\n",
    "\n",
    "#close the file.\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 9**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "e2020_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 10**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7), dpi= 120)\n",
    "\n",
    "plt.bar(e2020_df[\"name\"], e2020_df[\"followers_count\"])\n",
    "\n",
    "plt.title(\"Bar chart to showing follower count by account\")\n",
    "plt.xlabel(\"Twitter account\")\n",
    "plt.ylabel(\"Number of followers\")\n",
    "plt.xticks(rotation=90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 11**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(10, 7), dpi= 120)\n",
    "\n",
    "plt.plot(e2020_df[\"statuses_count\"], e2020_df[\"followers_count\"], \"o\")\n",
    "\n",
    "plt.title(\"Scatter plot to showing follower count by status count\")\n",
    "plt.xlabel(\"Number of statuses posted\")\n",
    "plt.ylabel(\"Number of followers\")\n",
    "plt.xticks(rotation=90)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Additional exercise. Counting word usage.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "user_words = []\n",
    "\n",
    "for i in e2020_df.index:\n",
    "    word_dict = {}\n",
    "    text = e2020_df.text[i].lower().split(' ')\n",
    "    for word in text:\n",
    "        if word not in word_dict:\n",
    "            word_dict[word] = e2020_df.text[i].count(word)\n",
    "    user_words.append(word_dict)\n",
    "\n",
    "f = open('stop_words.txt', 'r', errors='ignore')\n",
    "stop_words = []\n",
    "for line in f:\n",
    "    stop_words.append(line.strip(\"\\n\"))\n",
    "    \n",
    "for user in user_words:\n",
    "    for j in stop_words:\n",
    "        if j in user.keys():\n",
    "            user.pop(j)\n",
    "\n",
    "word_freq = pd.DataFrame({\"word_count\": user_words})\n",
    "e2020_df = pd.concat([e2020_df, word_freq], axis = 1)\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(e2020_df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}