Skip to content

A Framework to run user-defined functions for Map&Reduce on a Distributed Cluster on GCP in one line. Uses a self-implemented version of Memcached to communicate and store intermediate data.

Notifications You must be signed in to change notification settings

rgrishigajra/Uni-CourseWork-Cloud-Project

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

99 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This is an overall project readme. Please go inside each folder for assignment relevant readmes and reports!

1.Assignment 1: A persistent Key value map that handles concurrent clients over TCP.

key_value_pair_cache

2.Assignment 2: A map reduce frame work with master co-ordinating workers running map and reduce workers over sockets with fault tolerance.

map_reduce_master

gcloud beta compute instances create key-value-server --zone us-central1-a --source-machine-image base-map-reduce --metadata startup-script='#! /bin/bash git clone https://github.com/rgrishigajra/Uni-CourseWork-Cloud-Project.git cd Uni-CourseWork-Cloud-Project export VAR=abc
python3 server_init.py'

gcloud compute instances delete key-value-server --zone us-central1-a --quiet

sudo kill -9 sudo lsof -t -i:9001

sudo google_metadata_script_runner --script-type startup --debug

gcloud compute instances set-service-account master-map-reduce --zone=us-central1-a --service-account cloud-computong-service-acc@rishabh-gajra.iam.gserviceaccount.com --scopes compute-rw,storage-ro

About

A Framework to run user-defined functions for Map&Reduce on a Distributed Cluster on GCP in one line. Uses a self-implemented version of Memcached to communicate and store intermediate data.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published