Lecture on web scraping and (perhaps) text

[jlperla]

The lecture is planned for November 14th

My goal is primarily to help people realize that scraping the web and doing text analysis is Not scary! I don't want fear of it to be a reason they are not willing to get creative in the creation of new sources of data.

You guys can play around with the directory https://github.com/ubcecon/computing_and_datascience/tree/master/R_sandbox etc.

[JasmineHao]

Useful gadget for analyzing HTML
https://selectorgadget.com/

[jlperla]

Also see https://www.datacamp.com/community/tutorials/r-web-scraping-rvest and https://ropensci.org/tutorials/rselenium_tutorial/

[JasmineHao]

Something I think could be common for a class of websites.
I cannot webscrape https://www.ratebeer.com/ using Rvest tool, perhaps due to the cookies. Could leave for later research.

[jlperla]

Yeah, I think that there are a large number of sites where you really need to run selenium... it emulates both cookies and runs the javascript (at which point it can then be scraped by the other tools). It would be great if we should show a very minimal example of rselenium, if it is relatively easy to show.

Do you guys want to grab the R scraping textbooks from my office?

[chiyahn]

Relevant repos I have found in Github so far:

Useful R packages for data cleaning:

• PSID: https://github.com/floswald/psidR
• The World Wealth and Income Database: https://github.com/WIDworld/wid-r-tool
• The Survey of Professional Forecasters: https://github.com/joergrieger/Survey
• Democracy indices: https://github.com/xmarquez/democracyData (I highly recommend his book [Non-Democratic Politics] as well!)
• Uber trip data: https://github.com/fivethirtyeight/uber-tlc-foil-response
• Airbnb listings: https://github.com/tomslee/airbnb-data-collection

Fun examples (not necessarily economics):

• Spotify music data analysis: https://github.com/AsTimeGoesBy111/Spotify-Music-Data-Analysis
• Chinese poetry analysis: https://github.com/chinese-poetry/chinese-poetry
• South Park script anaylsis: https://github.com/pdrhlik/southparkr

[chiyahn]

Jasmine and I had some discussion about how the lecture can be delivered:

• In the first half of the lecture, I teach how data wrangling can be done seamlessly using tidyverse given that we have some clean dataset by:

  • Exploring some Github repos for data that can be used for awesome economic analysis.
  • Learning piping/ggplot/install_github/reshaping (for panel data) by following my tutorial that explores the relationship between democracy and income (which has been studied in econ literature for a long time) using democracyData package by Marquez at https://github.com/xmarquez/democracyData.
  • Some preliminary text analysis using tidytext?

• In the other half of the lecture, Jasmine and I teach how we can actually get the clean dataset with text mining and webscrapping by:

  • Replicating Hoberg and Phillips (2016)? We will explore more dataset & papers by the next week to settle this.
  • Webscrapping using plain text files
  • Brief introduction on Rvest?

[jlperla]

I think I really want to emphasize the webscraping more rather than talking about tidyverse transformations.

The goal should be about building people's confidence that they can (1) scrape numerical data from the web and (2) could work with text as data.

It is more important for me to show the tools than anything else.

[jlperla]

If all we did was give a 1.25 hour presentation on how to scrape a couple of websites, I would be happy.

To be clear, we do not need to have an economic application of getting the data, just that we should be scraping data that could be applied to economic problems. For example, you could even take a world-bank or whatever page that has a "download" button and say "lets pretend it didn't have that button", I will show you how you could have gotten the data anyways.

[arnavs]

Just so it's not forgotten, I wanted to link to the notes Jasmine Yang produced on this from a few months ago. If the issue has evolved since then, please feel free to disregard.

https://github.com/ubcecon/computing_and_datascience/blob/master/python_sandbox/Web-Scraping.md

[chiyahn]

Simple tutorial on using rvest I wrote yesterday: https://github.com/chiyahn/notes/blob/master/programming/data-mining/rvest/text-mining-with-rvest.md

[schrimpf]

Relatedly, I have attached code that scrapes the AER website to look at programming language usage. Patrick independently did the same thing, and his results are going to be part of the next AER annual report. His code is perhaps a bit nicer https://github.com/pbaylis/econ-program-usage

…

On Wed, Oct 31, 2018 at 2:25 PM Chiyoung Ahn ***@***.***> wrote: Simple tutorial on using rvest I wrote yesterday: https://github.com/chiyahn/notes/blob/master/programming/data-mining/rvest/text-mining-with-rvest.md — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJ-4vPG2C_--hskRDuhG1ka9EblmMVOaks5uqhVCgaJpZM4X8zdZ> .

[jlperla]

@pbaylis Can you give these guys your code to prepare as an Rmd file? I think it would be a nice example code to give people.

That said, I want to stress in class an example with data that is not "inside economics" so they don't think of this stuff as just a novelty.