Initial commit for the Saga Distributed Transactions Pattern on Azure

Author: error505

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Igor Iric
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,129 @@
+# Saga Distributed Transactions Pattern on Azure
+
+This repository provides an implementation of the **Saga Distributed Transactions Pattern** using Azure services, including Azure Functions, Azure Service Bus, and Azure Cosmos DB. The Saga Pattern is used to manage distributed transactions across multiple microservices, ensuring data consistency and reliability even when some services fail.
+
+## 🏗️ Architectural Overview
+
+The **Saga Pattern** breaks a distributed transaction into a series of smaller transactions that are executed in a sequence. If any transaction fails, the Saga Pattern triggers compensating transactions to undo the changes made by previous steps, ensuring **eventual consistency** across the entire system.
+
+### Architecture Diagram
+
+```mermaid
+graph TD
+    Client["Client"] -->|Initiates Transaction| OrchestratorFunction["Orchestrator Function (Azure Function)"]
+    OrchestratorFunction -->|Manages Steps| ActivityFunction["Activity Function (Azure Function)"]
+    OrchestratorFunction -->|Compensation Triggered| CompensatorFunction["Compensator Function (Azure Function)"]
+    ActivityFunction -->|Calls| ServiceBus["Azure Service Bus"]
+    ServiceBus -->|Triggers| CompensatorFunction
+    ActivityFunction -->|Manipulates| CosmosDB["Azure Cosmos DB"]
+    CompensatorFunction -->|Ensures| CosmosDB
+    OrchestratorFunction -->|Monitors| AppInsights["Application Insights"]
+    ActivityFunction -->|Monitors| AppInsights
+    CompensatorFunction -->|Monitors| AppInsights
+```
+
+### Components of the Architecture
+
+1. **Client**: Initiates a transaction request to the **Orchestrator Function**.
+2. **Orchestrator Function**: Manages the overall workflow of the Saga, coordinating the execution of the **Activity Function** and triggering the **Compensator Function** in case of a failure.
+3. **Activity Function**: Performs individual steps of the transaction, such as processing business logic and writing to **Azure Cosmos DB**.
+4. **Compensator Function**: Reverts or compensates for changes if a failure occurs in any step of the Saga.
+5. **Azure Service Bus**: Acts as a messaging backbone for communication between functions, triggering compensating transactions when necessary.
+6. **Azure Cosmos DB**: Serves as the data store where the transaction data is stored.
+7. **Application Insights**: Collects logs, metrics, and telemetry data for monitoring and troubleshooting purposes.
+
+## 📂 Repository Structure
+
+```
+/saga-distributed-transactions
+│
+├── README.md                                # Root README with architecture overview and getting started
+├── LICENSE                                  # MIT License
+│
+├── infrastructure
+│   ├── README.md                            # README for Infrastructure deployment
+│   ├── azure-resources.bicep                # Bicep template for all Azure resources
+│   └── .github/workflows/deploy-bicep.yml   # GitHub Action to deploy Azure resources
+│
+├── orchestrator-function
+│   ├── README.md                            # README for Orchestrator Function
+│   ├── orchestrator_function.py             # Python code for Orchestrator Function
+│   ├── requirements.txt                     # Dependencies for Orchestrator Function
+│   └── .github/workflows/deploy-orchestrator.yml # GitHub Action to deploy the Orchestrator Function
+│
+├── activity-function
+│   ├── README.md                            # README for Activity Function
+│   ├── activity_function.py                 # Python code for Activity Function
+│   ├── requirements.txt                     # Dependencies for Activity Function
+│   └── .github/workflows/deploy-activity.yml # GitHub Action to deploy the Activity Function
+│
+├── compensator-function
+│   ├── README.md                            # README for Compensator Function
+│   ├── compensator_function.py              # Python code for Compensator Function
+│   ├── requirements.txt                     # Dependencies for Compensator Function
+│   └── .github/workflows/deploy-compensator.yml # GitHub Action to deploy the Compensator Function
+```
+
+## 🚀 Getting Started
+
+### Step 1: Deploy the Infrastructure
+
+1. Navigate to the **`infrastructure`** folder.
+2. Follow the instructions in the [Infrastructure README](infrastructure/README.md) to deploy the required Azure resources using the Bicep template and GitHub Actions.
+
+### Step 2: Deploy the Azure Functions
+
+1. Deploy the **Orchestrator Function**:
+   - Navigate to the **`orchestrator-function`** folder.
+   - Follow the instructions in the [Orchestrator Function README](orchestrator-function/README.md) to deploy the function using GitHub Actions.
+
+2. Deploy the **Activity Function**:
+   - Navigate to the **`activity-function`** folder.
+   - Follow the instructions in the [Activity Function README](activity-function/README.md) to deploy the function using GitHub Actions.
+
+3. Deploy the **Compensator Function**:
+   - Navigate to the **`compensator-function`** folder.
+   - Follow the instructions in the [Compensator Function README](compensator-function/README.md) to deploy the function using GitHub Actions.
+
+### Step 3: Running the Solution
+
+1. **Trigger the Orchestrator Function**:
+   - Use an HTTP client (like Postman or `curl`) to send a POST request to the `/startSaga` endpoint of the Orchestrator Function.
+
+2. **Monitor the Workflow**:
+   - Check **Application Insights** in the Azure portal for logs and telemetry data to monitor the execution flow and identify any issues.
+
+## 💡 How It Works
+
+1. **Initiate a Transaction**:
+   - The **Client** initiates a transaction request to the **Orchestrator Function** by sending an HTTP request with order details.
+
+2. **Orchestrator Function Manages Workflow**:
+   - The **Orchestrator Function** processes the request, calls the **Activity Function** to handle the business logic, and waits for a success or failure response.
+
+3. **Activity Function Processes Data**:
+   - The **Activity Function** receives the order data and writes it to **Azure Cosmos DB**.
+   - If the transaction is successful, it completes the process; if there is an error, the **Compensator Function** is triggered.
+
+4. **Compensator Function Handles Failures**:
+   - The **Compensator Function** is triggered if a failure occurs during the transaction.
+   - It compensates for the failure by removing or rolling back any changes made to **Azure Cosmos DB**.
+
+5. **Application Insights Monitors the Flow**:
+   - All functions log their activities to **Application Insights** for monitoring and troubleshooting purposes.
+
+## 📝 About the Saga Pattern
+
+The **Saga Pattern** is a design pattern for managing distributed transactions in microservices architecture. It breaks down a large transaction into smaller sub-transactions that are executed in a sequence, ensuring eventual consistency across the system.
+
+- **Resilience**: By breaking down transactions into smaller steps and defining compensating actions, the Saga Pattern helps systems recover from failures gracefully.
+- **Scalability**: Decoupling transactions across services improves scalability, allowing independent scaling of microservices.
+- **Flexibility**: The Saga Pattern supports various consistency models, including eventual consistency, which is crucial for distributed systems.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+
+## 🙌 Contributing
+
+Contributions are welcome! Please open an issue or submit a pull request for any improvements or suggestions.

activity-function/.github/workflows/deploy-activity.yml

@@ -0,0 +1,43 @@
+name: Deploy Activity Function
+
+on:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+env:
+  AZURE_FUNCTIONAPP_NAME: 'activityFunctionApp'
+  AZURE_FUNCTIONAPP_PACKAGE_PATH: './activity-function'
+  AZURE_RESOURCE_GROUP: '<your-resource-group>'
+  FUNCTIONAPP_RUNTIME: 'python'
+  PYTHON_VERSION: '3.8'
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+
+    - name: Set up Python environment
+      uses: actions/setup-python@v2
+      with:
+        python-version: ${{ env.PYTHON_VERSION }}
+
+    - name: Login to Azure
+      uses: azure/login@v1
+      with:
+        creds: ${{ secrets.AZURE_CREDENTIALS }}
+
+    - name: Deploy Azure Function
+      uses: Azure/functions-action@v1
+      with:
+        app-name: ${{ env.AZURE_FUNCTIONAPP_NAME }}
+        package: '${{ env.AZURE_FUNCTIONAPP_PACKAGE_PATH }}'
+        publish-profile: ${{ secrets.AZURE_FUNCTIONAPP_PUBLISH_PROFILE }}
+
+    - name: Set Azure Function App Environment Variables
+      run: |
+        az functionapp config appsettings set --resource-group ${{ env.AZURE_RESOURCE_GROUP }} --name ${{ env.AZURE_FUNCTIONAPP_NAME }} --settings "ServiceBusConnection=${{ secrets.ServiceBusConnection }}" "CosmosDBConnectionString=${{ secrets.CosmosDBConnectionString }}" "FUNCTIONS_WORKER_RUNTIME=${{ env.FUNCTIONAPP_RUNTIME }}"

activity-function/.gitignore

@@ -0,0 +1,48 @@
+bin
+obj
+csx
+.vs
+edge
+Publish
+
+*.user
+*.suo
+*.cscfg
+*.Cache
+project.lock.json
+
+/packages
+/TestResults
+
+/tools/NuGet.exe
+/App_Data
+/secrets
+/data
+.secrets
+appsettings.json
+local.settings.json
+
+node_modules
+dist
+
+# Local python packages
+.python_packages/
+
+# Python Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Azurite artifacts
+__blobstorage__
+__queuestorage__
+__azurite_db*__.json

activity-function/README.md

@@ -0,0 +1,52 @@
+# Activity Function for Saga Distributed Transactions Pattern
+
+This folder contains the Python code for the **Activity Function**. The Activity Function performs individual steps of the transaction, such as processing a part of the business logic and writing to Cosmos DB.
+
+## 📑 Files
+
+- **`activity_function.py`**: Python code for the Activity Function.
+- **`requirements.txt`**: Lists the dependencies required by the Activity Function.
+- **`deploy-activity.yml`**: GitHub Action workflow to automate the deployment of the Activity Function.
+
+## 🚀 How to Deploy the Activity Function
+
+### Prerequisites
+
+1. **Azure Subscription**: You need an active Azure account.
+2. **Azure Function App**: Ensure the Azure Function App is created (using the Bicep template in the `/infrastructure` folder).
+3. **GitHub Secrets Configuration**:
+   - **`AZURE_CREDENTIALS`**: Azure service principal credentials in JSON format.
+   - **`AZURE_FUNCTIONAPP_PUBLISH_PROFILE`**: Publish profile for the Azure Function App.
+   - **`ServiceBusConnection`**: Connection string for the Azure Service Bus namespace.
+   - **`CosmosDBConnectionString`**: Connection string for the Azure Cosmos DB account.
+
+### Steps to Deploy
+
+1. **Add Required Secrets to GitHub**:
+   - Go to your repository's **Settings > Secrets and variables > Actions > New repository secret**.
+   - Add the following secrets:
+     - **`AZURE_CREDENTIALS`**: Your Azure service principal credentials.
+     - **`AZURE_FUNCTIONAPP_PUBLISH_PROFILE`**: The publish profile for your Azure Function App.
+     - **`ServiceBusConnection`**: The connection string for your Azure Service Bus namespace.
+     - **`CosmosDBConnectionString`**: The connection string for your Azure Cosmos DB account.
+
+2. **Run the GitHub Action**:
+   - Push your changes to the `main` branch or manually trigger the **Deploy Activity Function** workflow from the **Actions** tab.
+
+3. **Monitor the Deployment**:
+   - Go to the **Actions** tab in your GitHub repository.
+   - Select the **Deploy Activity Function** workflow to monitor the deployment progress.
+
+### 📝 How the Activity Function Works
+
+1. **Listen for Messages**:
+   - The Activity Function is triggered when it receives a message from the **Service Bus Topic** (`sagaServiceBusTopic`).
+   - This message contains order details, such as `orderId`, `customerName`, and `items`.
+
+2. **Process the Transaction**:
+   - The function writes the order details to **Azure Cosmos DB**.
+   - If successful, the data is stored in the Cosmos DB container; otherwise, it logs an error.
+
+## 📄 License
+
+This project is licensed under the MIT License - see the [LICENSE](../LICENSE) file for details.

activity-function/__init__.py

@@ -0,0 +1,38 @@
+import azure.functions as func
+import logging
+import os
+import json
+from azure.cosmos import CosmosClient, exceptions
+
+# Get environment variables
+cosmos_db_conn_str = os.getenv('CosmosDBConnectionString')
+cosmos_db_database_name = os.getenv('cosmosDbDatabaseName', 'myDatabase')
+cosmos_db_container_name = os.getenv('cosmosDbContainerName', 'myContainer')
+
+app = func.FunctionApp()
+
+@app.function_name(name="ActivityFunction")
+@app.service_bus_topic_trigger(
+    connection="ServiceBusConnection",
+    topic_name="sagaServiceBusTopic",
+    subscription_name="activitySubscription",
+)
+def run_activity(message: func.ServiceBusMessage):
+    logging.info('Activity Function: Processing message...')
+
+    try:
+        order_data = json.loads(message.get_body().decode('utf-8'))
+        logging.info(f"Processing order data: {order_data}")
+
+        cosmos_client = CosmosClient.from_connection_string(cosmos_db_conn_str)
+        database = cosmos_client.get_database_client(cosmos_db_database_name)
+        container = database.get_container_client(cosmos_db_container_name)
+
+        # Upsert order data to Cosmos DB
+        container.upsert_item(order_data)
+        logging.info(f"Order data stored in Cosmos DB: {order_data['orderId']}")
+
+    except exceptions.CosmosHttpResponseError as e:
+        logging.error(f"Cosmos DB error: {e}")
+    except Exception as e:
+        logging.error(f"Unexpected error in Activity Function: {e}")

activity-function/activity_function.py

@@ -0,0 +1,15 @@
+{
+    "version": "2.0",
+    "logging": {
+      "applicationInsights": {
+        "samplingSettings": {
+          "isEnabled": true,
+          "excludedTypes": "Request"
+        }
+      }
+    },
+    "extensionBundle": {
+      "id": "Microsoft.Azure.Functions.ExtensionBundle",
+      "version": "[4.*, 5.0.0)"
+    }
+  }

activity-function/host.json

@@ -0,0 +1,2 @@
+azure-functions
+azure-cosmos