Skip to content

This repo note the way I deploy data platform on the Kubernetes platform (on Premise).

Notifications You must be signed in to change notification settings

louisguitton/data-platform-notes

 
 

Repository files navigation

Data Architecture

Architecture

Technical Stack

Tools Distributions Descriptions Status
kubernetes Container orchestration Done
kubeadm Container orchestration Done
k0s Container orchestration Done
rke2 Container orchestration Done
k3s Container orchestration Done
Minio Secure object storage solution for reliable data storage in distributed environments. Done
Hive Metastore Schema management tool ensuring seamless evolution and organization of data. Done
Trino High-performance query engine for distributed data processing. Done
Spark Distributed processing engine Done
Doris Real-time data warehouse Done
StarRocks Real-time data warehouse Not started
Kafka Stream-processing platform Done
Flink Robust stream processing framework for real-time data analytics. In Process
RisingWave streaming database Not started
Camel K Open Source integration framework Not started
Airflow Workflow management platform Done
Superset Data exploration and data visualization Not started
Jupyter Notebook Web-based interactive development environment for notebooks, code, and data. Not started
MLflow A platform to streamline machine learning development Not started
Streamlit Python framework for data scientists and AI/ML engineers to deliver interactive data apps. Not started
Spring Boot Web framework for building RESTful APIs in Java. Not started
FastAPI Modern web framework for building RESTful APIs in Python. Not started
Jenkins Open-source CI/CD server Not started
Argo CD Argo CD is a declarative, GitOps continuous delivery tool for Kubernetes. Not started

About

This repo note the way I deploy data platform on the Kubernetes platform (on Premise).

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Smarty 49.0%
  • Mustache 37.6%
  • Python 9.1%
  • Shell 2.4%
  • Dockerfile 1.9%