Scrapes sites. Gets news. Eventually events.
###Installation
You should probably create a virtual environment, but
in any event doing pip install -r requirements.txt should do the trick. You
might (probably will) have to specify something along the lines of
--allow-all-external pattern --allow-unverified pattern for the pattern
library since it gets downloaded from its homepage.
The scraper requires a running MongoDB instance to dump the scraped stories into.
Make sure you have MongoDB installed
and type mongod at the terminal to begin the instance. MongoDB doesn't require you to prepare
the collection or database ahead of time, so when you run the program it should automatically
create a database called event_scrape with a collection called stories. Once you've run python scraper.py,
you can verify that the stories are in the Mongo database by opening a new terminal window and typing mongo. NOTE:
the scraper runs once every hour, meaning that stories may not immediately appear in your database. To force immediate scraping,
comment out the scheduling part at the end of scraper.py and uncomment call_scrape_func.
To interface with Mongo, enter mongo at the command line. From inside Mongo, type show dbs to verify that there's a database called event_scrape.
Enter the database with use event_scrape and type show collections to make sure there's a stories collection.
db.news_stories.find() will show you the first 20 entries.
###Running
After everything is installed, it's as simple as python scraper.py. That is
assuming, of course, that you wish to use the configuration seen in the
default_config.ini file. If not, just modify that. It's only set up to
specify the path to the file of URLs and the MongoDB collection that you want
to used.
The program will run once an hour and will pull from the RSS feeds specified in the URL file.
###Contributing
More RSS feeds are always useful. If there's something specific you want to see, just add it in and open a pull request. The only requirement is that each line contains one entry with the first column as a unique ID and the second the URL to the raw XML feed.