Pyspark Tutorial

Intended for a 3 parts Pyspark tutorial I'm writing for medium

Tested on Python 3.8. Check the requirements.txt file for package requirements.

Name	Name	Last commit message	Last commit date
Latest commit wtsimple Added link to first part of the tutorial in README Nov 9, 2020 9e2d0dc · Nov 9, 2020 History 46 Commits
bin	bin	Added code for tutorial part 1	Nov 8, 2020
data	data	Renamed data files to have the csv suffix	Nov 6, 2020
tests	tests	Fixed broken test	Nov 9, 2020
.gitignore	.gitignore	updated gitignore to ignore virtual env	Nov 9, 2020
README.md	README.md	Added link to first part of the tutorial in README	Nov 9, 2020
__init__.py	__init__.py	Added settings file to hold constants	Jul 14, 2020
data_loader.py	data_loader.py	data loader seems to be working OK	Jul 14, 2020
data_preprocessor.py	data_preprocessor.py	Model evaluator, produces right metrics for single classification model	Jul 24, 2020
model_evaluator.py	model_evaluator.py	Model evaluator, produces right metrics for single classification model	Jul 24, 2020
models_container.py	models_container.py	Renamed the ModelKinds enum to ModelTypes	Nov 6, 2020
requirements.txt	requirements.txt	Updated requirements	Nov 9, 2020
settings.py	settings.py	Renamed data files to have the csv suffix	Nov 6, 2020
spark_launcher.py	spark_launcher.py	Launcher, fixed "different versions" error	Jul 15, 2020
todo_list.md	todo_list.md	Upgraded pyspark to 3.0.1 also set up full requirements	Nov 4, 2020
tutorial_part_1_data_wrangling.py	tutorial_part_1_data_wrangling.py	Fixed broken test	Nov 9, 2020
tutorial_part_1_reading_from_database.py	tutorial_part_1_reading_from_database.py	Added code for tutorial part 1	Nov 8, 2020

Provide feedback