feat: add intraledger zap support via BigQuery monitoring

.envrc

@@ -9,4 +9,10 @@ export REDIS_0_DNS=
 export REDIS_1_DNS=
 export REDIS_2_DNS=
 export REDIS_MASTER_NAME=
+
+# BigQuery configuration for intraledger payment monitoring
+export BIGQUERY_PROJECT_ID=galoy-reporting
+export BIGQUERY_DATASET_ID=dataform_galoy_staging
+export GOOGLE_APPLICATION_CREDENTIALS=
+export BIGQUERY_CREDENTIALS=
 use flake

.github/workflows/unit-tests.yml

@@ -0,0 +1,24 @@
+name: "Unit tests"
+
+on:
+  pull_request:
+    branches: [main]
+
+jobs:
+  unit-tests:
+    name: Unit tests
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Install Nix
+        uses: DeterminateSystems/nix-installer-action@v14
+
+      - name: Run the Magic Nix Cache
+        uses: DeterminateSystems/magic-nix-cache-action@v8
+
+      - name: Install dependencies inside nix develop
+        run: nix develop -c pnpm install
+
+      - name: Run check code
+        run: nix develop -c pnpm test

.gitignore

@@ -3,3 +3,4 @@ dist/
 .vscode/
 .DS_Store
 .direnv/
+coverage/

BIGQUERY_SETUP.md

@@ -0,0 +1,172 @@
+# BigQuery Setup for Intraledger Zap Monitoring
+
+## Overview
+
+This document outlines the setup of BigQuery for monitoring intraledger payments in the blink-nostr service. This implementation uses the existing Kafka streaming architecture and provides better separation of concerns by avoiding direct database access.
+
+## Why BigQuery?
+
+1. **Architectural Consistency**: Uses the existing Kafka → BigQuery pipeline
+2. **No Direct Database Access**: Removes the need for blink-nostr to connect to production MongoDB
+3. **Better Performance**: BigQuery is optimized for analytical queries
+4. **Scalability**: No additional load on production databases
+5. **Data Freshness**: Kafka streaming provides near real-time data
+
+## Implementation
+
+### 1. BigQuery Client Setup
+
+- **Added**: `@google-cloud/bigquery` dependency and `src/bigquery.js`
+- **Implemented**: BigQuery queries in `src/queries.js`
+- **Integrated**: BigQuery monitoring in `src/zapper.js` and `src/intraledger-monitor.js`
+
+### 2. Environment Variables
+
+```bash
+BIGQUERY_PROJECT_ID=galoy-reporting
+BIGQUERY_DATASET_ID=dataform_galoy_staging  # or dataform_galoy_bbw for production
+GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_key>
+# OR
+BIGQUERY_CREDENTIALS=<service_account_json_string>
+```
+
+### 3. Query Logic
+
+The BigQuery implementation queries the `mongodb_galoy_walletinvoices` table which is populated by the existing Kafka streaming pipeline. The query identifies intraledger payments by:
+
+- `paid = true`
+- `processingCompleted = true` 
+- `paymentRequest IS NOT NULL`
+- `selfGenerated = false` (heuristic for intraledger payments)
+
+## Deployment Steps
+
+### 1. Service Account Setup
+
+Create a BigQuery service account with the following permissions:
+- `BigQuery Data Viewer`
+- `BigQuery Job User`
+
+```bash
+# Create service account
+gcloud iam service-accounts create blink-nostr-bigquery \
+  --description="Service account for blink-nostr BigQuery access" \
+  --display-name="Blink Nostr BigQuery"
+
+# Grant BigQuery permissions
+gcloud projects add-iam-policy-binding galoy-reporting \
+  --member="serviceAccount:[email protected]" \
+  --role="roles/bigquery.dataViewer"
+
+gcloud projects add-iam-policy-binding galoy-reporting \
+  --member="serviceAccount:[email protected]" \
+  --role="roles/bigquery.jobUser"
+
+# Create and download key
+gcloud iam service-accounts keys create blink-nostr-bigquery-key.json \
+  --iam-account=blink-nostr-bigquery@galoy-reporting.iam.gserviceaccount.com
+```
+
+### 2. Environment Configuration
+
+**For Staging**:
+```bash
+export BIGQUERY_PROJECT_ID=galoy-reporting
+export BIGQUERY_DATASET_ID=dataform_galoy_staging
+export GOOGLE_APPLICATION_CREDENTIALS=/path/to/blink-nostr-bigquery-key.json
+```
+
+**For Production**:
+```bash
+export BIGQUERY_PROJECT_ID=galoy-reporting
+export BIGQUERY_DATASET_ID=dataform_galoy_bbw
+export GOOGLE_APPLICATION_CREDENTIALS=/path/to/blink-nostr-bigquery-key.json
+```
+
+### 3. Kubernetes Deployment
+
+Update the Kubernetes deployment to include the BigQuery credentials:
+
+```yaml
+apiVersion: v1
+kind: Secret
+metadata:
+  name: blink-nostr-bigquery-creds
+type: Opaque
+data:
+  keyfile: <base64-encoded-service-account-key>
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: blink-nostr
+spec:
+  template:
+    spec:
+      containers:
+      - name: blink-nostr
+        env:
+        - name: BIGQUERY_PROJECT_ID
+          value: "galoy-reporting"
+        - name: BIGQUERY_DATASET_ID
+          value: "dataform_galoy_staging"  # or dataform_galoy_bbw
+        - name: GOOGLE_APPLICATION_CREDENTIALS
+          value: "/etc/bigquery/keyfile"
+        volumeMounts:
+        - name: bigquery-creds
+          mountPath: /etc/bigquery
+          readOnly: true
+      volumes:
+      - name: bigquery-creds
+        secret:
+          secretName: blink-nostr-bigquery-creds
+```
+
+## Verification
+
+### 1. Check BigQuery Table
+
+Verify the `mongodb_galoy_walletinvoices` table exists and has recent data:
+
+```sql
+SELECT COUNT(*) as total_invoices,
+       COUNT(CASE WHEN paid = true THEN 1 END) as paid_invoices,
+       MAX(timestamp) as latest_timestamp
+FROM `galoy-reporting.dataform_galoy_staging.mongodb_galoy_walletinvoices`
+WHERE timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
+```
+
+### 2. Test Intraledger Payment Detection
+
+```sql
+SELECT _id, timestamp, paymentRequest, selfGenerated, paid, processingCompleted
+FROM `galoy-reporting.dataform_galoy_staging.mongodb_galoy_walletinvoices`
+WHERE paid = true 
+  AND processingCompleted = true
+  AND paymentRequest IS NOT NULL
+  AND selfGenerated = false
+  AND timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
+ORDER BY timestamp DESC
+LIMIT 10
+```
+
+### 3. Monitor Service Logs
+
+After deployment, monitor the blink-nostr service logs for:
+- Successful BigQuery connection
+- Query performance metrics
+- Processed intraledger payments
+
+## Performance Considerations
+
+1. **Query Frequency**: The adaptive polling (2-30 seconds) balances freshness with BigQuery costs
+2. **Data Latency**: Kafka streaming typically has <1 minute latency
+3. **Query Optimization**: Time-windowed queries limit data scanned
+4. **Cost Management**: BigQuery charges per query and data scanned
+
+## Future Improvements
+
+1. **Real-time Streaming**: Consider Pub/Sub for real-time notifications instead of polling
+2. **Query Optimization**: Refine the intraledger payment detection logic
+3. **Monitoring**: Add BigQuery-specific metrics and alerting
+4. **Caching**: Implement query result caching for frequently accessed data

README.md

@@ -7,10 +7,11 @@ This service connects Lightning Network payments to the Nostr protocol, enabling
 Blink Nostr is a bridge service that:
 
 1. Connects to a Lightning Network node (LND)
-2. Listens for paid invoices
-3. Retrieves zap request metadata from Redis
-4. Creates and signs Nostr zap receipt events (kind 9735)
-5. Broadcasts these events to specified Nostr relays
+2. Listens for paid Lightning invoices via LND subscriptions
+3. Monitors BigQuery for intraledger payments via optimized polling
+4. Retrieves zap request metadata from Redis
+5. Creates and signs Nostr zap receipt events (kind 9735)
+6. Broadcasts these events to specified Nostr relays
 
 ## Components
 
@@ -60,6 +61,13 @@ REDIS_2_DNS=<redis_sentinel_2_host>
 REDIS_0_SENTINEL_PORT=26379
 REDIS_1_SENTINEL_PORT=26379
 REDIS_2_SENTINEL_PORT=26379
+
+# BigQuery (for intraledger payment monitoring)
+BIGQUERY_PROJECT_ID=galoy-reporting
+BIGQUERY_DATASET_ID=dataform_galoy_staging
+GOOGLE_APPLICATION_CREDENTIALS=<path_to_service_account_key>
+# OR use credentials directly
+BIGQUERY_CREDENTIALS=<service_account_json_string>
 ```
 
 ## Installation
@@ -82,6 +90,22 @@ pnpm build
 node src/index.js
 ```
 
+## BigQuery Setup
+
+The service monitors BigQuery for intraledger payments using the existing Kafka streaming pipeline. The `walletinvoices` collection is automatically streamed from MongoDB to BigQuery via Kafka.
+
+For detailed setup instructions, see [BIGQUERY_SETUP.md](BIGQUERY_SETUP.md).
+
+The service uses:
+- BigQuery's optimized columnar storage for efficient querying
+- Time-windowed queries to limit dataset size and improve performance
+- Adaptive polling (2-30 seconds) with performance monitoring and circuit breakers
+- Service account authentication for secure access to BigQuery
+
+No additional database setup is required as the data is automatically available through the existing streaming pipeline.
+
+The service uses adaptive polling (2-30 seconds) and includes performance monitoring and circuit breakers.
+
 ## Development
 
 This project uses:
@@ -100,6 +124,16 @@ pnpm code:check
 pnpm prettier:fix
 ```
 
+## Testing
+
+Run the unit tests with:
+
+```bash
+pnpm test
+```
+
+This will execute the test suite using Vitest and display the results in the console.
+
 ## How It Works
 
 1. When a Lightning invoice is paid, the service receives a notification

package.json

@@ -9,6 +9,7 @@
     "orig:push": "docker push nicolasburtey/galoy-nostr:0.0.1",
     "orig:build-and-push": "yarn run build && yarn run push",
     "build": "pnpm install && tsup",
+    "test": "vitest",
     "code:check": "pnpm eslint:check && pnpm prettier:check",
     "eslint:check": "eslint . --ext .js",
     "eslint:fix": "eslint .  --ext .js --fix",
@@ -18,6 +19,7 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
+    "@google-cloud/bigquery": "^7.0.0",
     "ioredis": "5.3.1",
     "ln-service": "57.24.0",
     "nostr": "0.2.7",
@@ -32,12 +34,14 @@
     "eslint-config-prettier": "^9.0.0",
     "eslint-plugin-import": "^2.27.5",
     "eslint-plugin-prettier": "^5.0.0",
-    "prettier": "^3.3.3"
+    "prettier": "^3.3.3",
+    "vitest": "^1.2.2"
   },
   "pnpm": {
     "overrides": {
       "@scure/bip39": "1.1.1",
-      "@scure/bip32": "1.1.5"
+      "@scure/bip32": "1.1.5",
+      "form-data": "2.5.4"
     }
   }
 }