feat: server setup + refactor

.gitignore

@@ -1,3 +1,8 @@
 node_modules
 db
-log
+log
+benchmarks
+data
+
+commit-errors.txt
+events.json

.gitpod.yml

@@ -0,0 +1,11 @@
+# This configuration file was automatically generated by Gitpod.
+# Please adjust to your needs (see https://www.gitpod.io/docs/introduction/learn-gitpod/gitpod-yaml)
+# and commit this file to your remote git repository to share the goodness with others.
+
+# Learn more from ready-to-use templates: https://www.gitpod.io/docs/introduction/getting-started/quickstart
+
+tasks:
+  - init: npm install
+    command: npm run prepare && npm run start:dev
+
+

.nvmrc

@@ -1 +1 @@
-v16.19.0
+v18.18.0

.prettierrc

@@ -0,0 +1 @@
+{}

CONTRIBUTING.md

@@ -0,0 +1,35 @@
+# StylusDB
+
+## Project Structure
+
+This project is a normal NodeJS project with a target of using as less NPM dependencies as much possible.
+
+### Directories
+
+- [`./raft`](./raft/) - This directory contains a very light weight implementation of the Raft Protocol. The code inside this folder has been taken from [unshitio/liferaft](https://github.com/unshiftio/liferaft)
+- [`./reference`](./reference/) - This folder is for experimentation purposes this is where the contributors store the code for WIP experiments which are to be referenced later.
+- [`./scripts`](./scripts/) - This folder contains a few scripts to avoid the hassle of manually running the cluster and clients
+
+### Files
+
+- [`./cli.js`](./cli.js) - This file contains the implementation for a minimalistic REPL or CLI for StylusDB cause what is a DB without a CLI 🤓.
+- [`./client.js`](./client.js) - This is a small implementation of a sample client to send messages to the nodes in the cluster.
+- [`./db.js`](./db.js) - This contains the implementation for connecting and storing data to our storage engine. Presently this acts as an interface to `LMDB` but in future this will be a generic interface to a host of storage engines.
+- [`./index.js`](./index.js) - Contains the implementation of the class defining the structure and callbacks for a node in the `StylusDB` Raft Node Cluster.
+- [`./raft-node.js`](./raft-node.js) - This file contains functions to instantiate a node using the class defined in index.js. This exists as a separate file for ease of understanding right now, might be merged with `index.js` later.
+- [`./server.js`](./server.js) - This file setsup a server to accept connections and respond to requests from client. This is the file where the Raft Node is instantiated and connected to the other nodes in the cluster.
+
+## How to Contribute?
+- Refer to issue tickets on [Github](https://github.com/techsavvyash/stylusdb-js)
+- A list of high level goals and roadmap can be found in [TODO.md](./TODO.md)
+- Rest of the steps remain same as how you'd contribute to any other open source project
+  - Fork the repository
+  - Make the required changes 
+  - Raise a PR and ask for reviews
+- In case of doubts or guidance reach out to [Yash Mittal](https://github.com/techsavvyash)
+
+## Pre-requisties 
+- Understanding of the [Raft Distributed Consensus Algorithm](ft.github.io/raft.pdf)
+- Working knowledge of [NodeJS](https://nodejs.com)
+- Knowledge of Distributed Systems and Distributed Databases
+- Knowledge of Database Internals

README.md

@@ -1,46 +1,52 @@
 # stylusdb-js
 
-A minimalistic implementation of a distributed KV Store
+A minimalistic implementation of a distributed KV Store based on the Raft consensus protocol. StylusDB aims to be a SOTA JS implementation of a RAFT based highthrouput KV.
 
 ### Configuration
-Modify config.json
+Modify `config.json`
 
 ### Installation
 ```sh
-nvm install 16
-nvm use 16
+nvm install 18
+nvm use 18
 npm install
 ```
 
+### Preparing Directories
+
+Create the following 6 directories before contiuing to start the server below:
+```bash
+mkdir db log
+```
+
+
 ### Starting Server
-Open four terminals and start a 4 node cluster.
 
 ```sh
-DEBUG=* node index.js --port 8081
-DEBUG=* node index.js --port 8082
-DEBUG=* node index.js --port 8083
-DEBUG=* node index.js --port 8084
+npm run start
 ```
 
+This will start a 4 node raft cluster, a proxy server and a user-shell in a [mprocs](https://github.com/pvolok/mprocs) tui for you to develop easily.
+
+
 ### Benchmarks
-Coming soon
+Coming soon 
 
 ### REPL
-Coming soon
+Coming soon (WIP with [ClI](./cli.js))
 
 ### TODOs
 - [ ] Dockerization
 - [ ] Memory profiling
 - [ ] Scripts to deploy to edge
 - [ ] Client implementation
+- [ ] Integrate YCSB
+  - [ ] Java based
+  - [ ] Go based (because TiKV uses this to benchmark themselves)
 
 ### Caveats
 1. WIP (Bleeding edge) - use it at your own risk.
-2. Built for learning puposed only
-
-### Contribution
-1. Project is actively looking for contributors. Look at open tickets
-
+2. Built for learning puposes only
 
 ### References
 1. [REPL](https://gist.github.com/goliatone/e8f38b75aa05b2d189f68a92c61af110)

config/mprocs.dev.yml

@@ -0,0 +1,9 @@
+procs:
+  raft-node-1:
+    shell: "node ./src/server.js --port 8081"
+  raft-node-2:
+    shell: "node ./src/server.js --port 8082"
+  raft-node-3:
+    shell: "node ./src/server.js --port 8083"
+  raft-node-4:
+    shell: "node ./src/server.js --port 8084"

config/mprocs.yml

@@ -0,0 +1,13 @@
+procs:
+  raft-node-1:
+    shell: "node ./src/server.js --port 8081"
+  raft-node-2:
+    shell: "node ./src/server.js --port 8082"
+  raft-node-3:
+    shell: "node ./src/server.js --port 8083"
+  raft-node-4:
+    shell: "node ./src/server.js --port 8084"
+  proxy:
+    shell: "node ./src/proxy.js"
+  user-shell:
+    cmd: ["zsh"]

docs/PROTOCOL.md

@@ -0,0 +1,26 @@
+# StylusDB Protocol
+
+This is a set of rules that we use to distinguish between different kinds of messages we are sending.
+
+## Proxy
+This defines the types of messages we need in the proxy.
+We use the '\n' as the delimiter for the messages
+
+### 1. Request from Client
+
+Legend: "?"
+
+### 2. Response from Cluster Node
+
+Legend: "+"
+
+
+### 3. Information response from Cluster Node
+
+
+
+
+
+## Raft Node Server
+- Normal raft node data/commands
+- Information Requests -- information about who is the current cluster leader and how to figure out which nodes are alive and which are not ; how many members are a member of the cluster

docs/TODO.md

@@ -0,0 +1,51 @@
+# TODO
+
+## Resources
+To browse the DB: https://www.mdbopener.com/
+-- Refer: Pelican -- This is rebuilding Pelican in JS
+
+## Roadmap / Things to try out
+
+- [ ] Turn message saving into a batch process
+- [ ] Fine tune `server.js` to expose set and get via CLI
+- [ ] Turn setInterval directly via sockets
+- [ ] Fine tune LMDB Manager
+- [ ] Compress commands together and in memory calculate the final process
+- [ ] Let candidates run the `GET` commands
+   - [ ] Make sure there are no race conditions here
+- [ ] Run RAFT on the browser - [Mozilla Docs on Socket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API)
+- [ ] Demo with message inter-play between nodes and other benchmarking 
+- [ ] Large Keys and Small Keys optimisation
+
+
+
+## Doubts :- 
+- [ ] In message of raft when we are sending "SET" as what it should be giving error as in raft.on("data", packet) the packet is bound to be an object which is not so in this case, (sending message in server is ambiguous, the what must be in the form of a raft packet)
+
+- [ ] When we send a message from follower to leader to set a value, there is no case to command all followers to save that message
+
+
+
+## List -- 27/03/24
+
+- [ ] Implement the orchestrator
+- [ ] Fix the key skipping error
+- [ ] Add the load balancer -- figure out if proxy can do it on its on or if we need a dedicated load balancer
+- [ ] Implement updating Get-Set based on keys in a batch
+- [ ] Figure out batch size -- because we get already connected error
+
+
+## Observations
+
+- we get pauses in between sets -- is there a settimeout in teh -- log entries/indexes are getting mised 
+- figure out why are log entries are not getting updated and throwing not found error.
+- make the proxy connect to all the nodes in the cluster
+   make this fault tolerant so that there is a exponential retry incase some node goes off
+- edit the "leader" fn to send a message to the proxy when the leader changes for it to know who the current leader is
+- implement a sort of load balancing to delegate requests between the raft nodes
+- figure out vs-code crashing with a lot of requests -- better memory management for the app maybe?
+
+## question
+
+- Load Balancing -- Should we have a separate load balancer? or is rotating queries at a proxy level enough?
+- Bug: Data is not being stored between restarts

docs/appendix/runtimes/README.md

@@ -0,0 +1,9 @@
+# Response Times
+
+This folder contains the response time of each request that is sent out during the testing for reference purposes.
+
+
+| File Name | Description | Link to file
+|-----------|-------------|-------------
+| `set_simd_1000.json`| Response times for each SET request in the batch of 1000 requests | [Link](./set_simd_1000.json) |
+| `get_simd_1000.json`| Response times for each GET request in the batch of 1000 requests | [Link](./get_simd_1000.json) |