Welcome to My Docker-Hosted Postgres to Snowflake Data Transfer Project!

> Hey There!, I am Shubham Dalvi

「 I am a data engineer with a passion for big data, distributed computing, and data visualization 」

About Me

✌️   Enjoy solving data problems

❤️   Passionate about big data technologies, distributed systems, and data visualizations

📧   Reach me : [email protected]

Skills and Technologies

Project Overview

This project demonstrates how to set up a PostgreSQL database hosted on Docker, create tables, and transfer data from PostgreSQL to Snowflake using Airbyte. The project includes setting up the environment, creating the database schema, and automating the data transfer process.

Table of Contents

• Technologies Used
• Skills Demonstrated
• Setup PostgreSQL on Docker
• Creating the Database and Tables
• Configuring Airbyte
• Data Transfer to Snowflake
• Usage Instructions

Technologies Used

• PostgreSQL: For database management.
• Docker: To containerize and host the PostgreSQL instance.
• Airbyte: To extract and load data from PostgreSQL to Snowflake.
• Snowflake: For data warehousing.
• Python: For scripting and automation.

Skills Demonstrated

• Data Engineering: Setting up and managing a PostgreSQL database in Docker.
• Docker: Containerizing services for ease of use and portability.
• ETL Processes: Automating data transfer from PostgreSQL to Snowflake.
• Airbyte Configuration: Setting up and managing data connectors.

Setup PostgreSQL on Docker

Initial Setup

Start by pulling the PostgreSQL Docker image and running a container.

docker pull postgres
docker run --name my_postgres -e POSTGRES_PASSWORD=mysecretpassword -d postgres

Accessing the PostgreSQL Database

Connect to the PostgreSQL instance running inside the Docker container.

docker exec -it my_postgres psql -U postgres

Creating the Database and Tables

Creating a Database

Create a new database for storing your data.

CREATE DATABASE my_database;

Creating Tables

Define the schema and create tables within the database.

CREATE TABLE transactions (
    transaction_id SERIAL PRIMARY KEY,
    amount DECIMAL(10, 2),
    transaction_date DATE
);

Configuring Airbyte

Setting Up Airbyte

Install and run Airbyte to handle the data transfer.

docker-compose up -d

Connecting PostgreSQL and Snowflake

Create a new connection in Airbyte, specifying PostgreSQL as the source and Snowflake as the destination. Map the tables and fields as needed.

Data Transfer to Snowflake

Running the Sync

Trigger the data sync from PostgreSQL to Snowflake using Airbyte's interface or API.

Usage Instructions

1. Ensure Docker and Airbyte are installed and running.
2. Set up PostgreSQL on Docker using the provided commands.
3. Create the necessary tables in PostgreSQL.
4. Configure Airbyte with the appropriate source and destination settings.
5. Trigger the data sync and verify the results in Snowflake.