YATD-Big-Data-Project

Overview

The YATD-Big-Data-Project is a Task Distributer system to understand Distributed Systems for Big Data. It efficiently distributes tasks among multiple workers, utilizes a Redis database for task management, and implements a message broker for seamless communication between clients and workers.

Features

• Task Distribution: Distributes tasks among multiple worker nodes.
• Redis Integration: Stores and manages task information using a Redis database.
• Message Broker: Facilitates communication between clients and workers to ensure smooth task execution.
• Unique Task IDs: Generates unique identifiers for each task to track their status.
• Simulated Delays: Implements delays and checkpoints in the worker process to mimic real-world scenarios.

Technology Stack

• Python: The primary programming language used for development.
• Redis: Used as the database for storing task information.
• Message Broker: Manages the communication between clients and workers.

File Structure

YATD-Big-Data-Project/

│

├── backend.py # Main backend logic

├── broker.py # Message broker implementation

├── client.py # Client-side interface

└── worker.py # Worker process handling

Requirements

1. Redis client for Python
2. Kafka
3. Zookeeper

To Run

1. Start the zookeeper:

   sudo systemctl start zookeeper
   

2. Start the Kafka:

   sudo systemctl start kafka
   

3. Start the Redis server:

    sudo systemctl start redis