Writting...

YAStorm

A lightweight version of Apache Storm, purely written in Python; The primary reason of implementing this project is to make learning Storm much easier. The implementations refers to Apache Storm v1.0.0.

You can treat this project as a prototype system for Apache Storm.

Architecture

                                        +--------------------+
                                        | Supervisor: Worker |
                   +------------+     / +--------------------+
                   |  Zookeeper | ----
                   +------------+
+---------+        +------------+       +--------------------+
| Nimbus  | ---->  |  Zookeeper | ----  | Supervisor: Worker |
+---------+        +------------+       +--------------------+
                   +------------+
                   |  Zookeeper | ----  +--------------------+
                   +------------+     \ | Supervisor: Worker |
                                        +--------------------+

Nimbus and Supervisor are fail-fast and stateless, and their state is kept in Zookeeper or on local disk(s).

Nimbus

In Apache Storm, Nimbus plays a similar role as the Job Tracker in Hadoop. Nimbus is an Thrift service implemented by Thriftpy

Supervisor

The supervisor runs on each node. It receives assignments from Nimbus and then assign them to workers. It also monitors the health of the workers and respawns them if necessary.

Internals

Topology

Acker

https://github.com/apache/storm/blob/v1.0.0/storm-core/src/clj/org/apache/storm/daemon/acker.clj

References

• History of Apache Storm and lessons learned
• Storm Concepts
• Storm Codebase Structure
• Storm - Distributed and fault-tolerant realtime computation, Nathan Marz
• Storm @ Twitter
• Understanding Storm internal message buffers
• The Future of Apache Storm