Add Docker containerization of unreal-index service

.dockerignore

@@ -0,0 +1,23 @@
+node_modules
+data
+*.log
+config.json
+.git
+.github
+.claude
+memory
+profiles
+NUL
+signature*
+perf-results-*.json
+test-*.mjs
+test-*.js
+test-*.ps1
+perf-stress-test.mjs
+*.bat
+src/perf-test.js
+src/stress-test.js
+src/grep-stress-test.js
+src/pool-stress-test.js
+src/profile.js
+src/*.test.js

DOCKER.md

@@ -0,0 +1,247 @@
+# Docker Setup for unreal-index
+
+## Prerequisites
+
+- **Docker Desktop for Windows** with WSL 2 backend enabled
+- At least **4 GB RAM** allocated to Docker (Settings > Resources > Memory)
+- The Windows watcher (`node src\watcher\watcher-client.js`) runs outside the container
+
+## Quick Start
+
+```bash
+# Build and start the service
+docker compose up -d
+
+# Verify it's running
+curl http://localhost:3847/health
+
+# Start the watcher on Windows (separate terminal)
+node src\watcher\watcher-client.js
+```
+
+Or use the convenience script:
+
+```bash
+./start-service.sh --docker
+```
+
+## Architecture
+
+```
+  Windows Host
+  ┌─────────────────────────────────────┐
+  │  Watcher (node watcher-client.js)   │
+  │      reads P4 workspace files       │
+  │      parses AS/C++/assets           │
+  │             │                       │
+  │             │ POST /internal/ingest  │
+  │             ▼                       │
+  │  ┌─────────────────────────────┐    │
+  │  │   Docker Container          │    │
+  │  │   ┌───────────────────┐     │    │
+  │  │   │  Node.js Service  │:3847│◄───┼── Claude Code / MCP
+  │  │   │  (Express API)    │     │    │
+  │  │   │  SQLite + Memory  │     │    │
+  │  │   └───────┬───────────┘     │    │
+  │  │           │                 │    │
+  │  │   ┌───────▼───────────┐     │    │
+  │  │   │  Zoekt            │     │    │
+  │  │   │  (index + web)    │     │    │
+  │  │   └───────────────────┘     │    │
+  │  │                             │    │
+  │  │  Volumes:                   │    │
+  │  │   /data/db      (SQLite)    │    │
+  │  │   /data/mirror  (Zoekt src) │    │
+  │  │   /data/zoekt-index (shards)│    │
+  │  └─────────────────────────────┘    │
+  └─────────────────────────────────────┘
+```
+
+## Configuration
+
+The container uses `config.docker.json` as the default config. Key settings:
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| `service.host` | `0.0.0.0` | Required for Docker port mapping |
+| `service.port` | `3847` | API port |
+| `data.dbPath` | `/data/db/index.db` | SQLite database path |
+| `data.mirrorDir` | `/data/mirror` | Zoekt mirror directory |
+| `data.indexDir` | `/data/zoekt-index` | Zoekt index shards |
+| `zoekt.parallelism` | `4` | Zoekt indexing threads |
+| `projects` | `[]` | Empty — data arrives via `/internal/ingest` |
+
+### Custom config
+
+Mount a custom config file:
+
+```yaml
+# docker-compose.override.yml
+services:
+  unreal-index:
+    volumes:
+      - ./my-config.json:/app/config.json:ro
+```
+
+### Environment variables
+
+- `UNREAL_INDEX_CONFIG` — path to config file (overrides default)
+- `NODE_OPTIONS` — Node.js options (default: `--max-old-space-size=3072`)
+
+## Data Persistence
+
+Data is stored in three Docker named volumes:
+
+| Volume | Contents | Purpose |
+|--------|----------|---------|
+| `unreal-index-db` | `index.db` | SQLite database |
+| `unreal-index-mirror` | Source files | Zoekt mirror for grep |
+| `unreal-index-zoekt` | Index shards | Zoekt search index |
+
+### Lifecycle
+
+```bash
+# Stop container (data preserved)
+docker compose down
+
+# Start again (data still there)
+docker compose up -d
+
+# DANGER: Remove data volumes
+docker compose down -v
+```
+
+### Backup
+
+```bash
+# Backup database
+docker compose exec unreal-index cp /data/db/index.db /data/db/index.db.bak
+
+# Copy database to host
+docker compose cp unreal-index:/data/db/index.db ./backup-index.db
+```
+
+### Restore
+
+```bash
+# Copy database into container
+docker compose cp ./backup-index.db unreal-index:/data/db/index.db
+
+# Restart to load
+docker compose restart
+```
+
+## Memory Configuration
+
+The container is limited to 4 GB RAM:
+- **3 GB** — Node.js heap (`--max-old-space-size=3072`)
+- **~1 GB** — Zoekt processes, OS overhead, SQLite mmap
+
+For larger codebases, increase both limits in `docker-compose.yml`:
+
+```yaml
+services:
+  unreal-index:
+    mem_limit: 6g
+    memswap_limit: 6g
+    environment:
+      - NODE_OPTIONS=--max-old-space-size=4096
+```
+
+Also ensure Docker Desktop has sufficient RAM allocated (Settings > Resources).
+
+## Troubleshooting
+
+### Port conflict
+
+```
+Error: listen EADDRINUSE: address already in use :::3847
+```
+
+Another process is using port 3847. Stop it or change the port mapping:
+
+```yaml
+ports:
+  - "3848:3847"  # Use 3848 on host
+```
+
+### Container OOM killed
+
+```bash
+# Check if container was OOM killed
+docker inspect unreal-index | grep -A 5 OOMKilled
+```
+
+Increase `mem_limit` in `docker-compose.yml` and Docker Desktop RAM allocation.
+
+### Slow queries after restart
+
+The in-memory index needs to reload from SQLite on startup (~10s for large indexes). Queries during this window may be slow or return empty results. The health check accounts for this with a 30s start period.
+
+### SQLite errors
+
+SQLite runs on a Docker named volume (ext4 filesystem). This avoids the performance issues of bind-mounting from Windows NTFS. If you see locking errors, ensure only one container is running:
+
+```bash
+docker compose ps
+```
+
+### Viewing logs
+
+```bash
+# Follow logs
+docker compose logs -f
+
+# Last 100 lines
+docker compose logs --tail=100
+```
+
+## Development
+
+Use the dev compose override for source mounting with auto-reload:
+
+```bash
+docker compose -f docker-compose.yml -f docker-compose.dev.yml up
+```
+
+This mounts `src/` and `public/` as read-only volumes and enables Node.js `--watch` mode. The container rebuilds only when dependencies change.
+
+To rebuild after dependency changes:
+
+```bash
+docker compose build
+```
+
+## Performance Testing
+
+Run the Docker performance test from the host:
+
+```bash
+node test-docker-perf.mjs
+```
+
+With a baseline comparison:
+
+```bash
+node test-docker-perf.mjs --baseline perf-baseline-wsl.json
+```
+
+Long-running stability test (30 minutes):
+
+```bash
+node test-docker-perf.mjs --long-run
+```
+
+## Docker vs WSL Comparison
+
+| Aspect | WSL (current) | Docker |
+|--------|---------------|--------|
+| **Setup** | Manual: Node 22, Go, Zoekt build, screen | `docker compose up -d` |
+| **Networking** | WSL mirrored mode, screen hacks | Standard port mapping |
+| **Persistence** | `~/.unreal-index/` on ext4 | Named volumes (ext4) |
+| **Updates** | `git pull && npm install` | `docker compose build && up -d` |
+| **Memory** | Shared with WSL | Isolated, configurable limit |
+| **Startup** | ~10s (warm) | ~15s (warm, includes container overhead) |
+| **SQLite perf** | Native ext4 | Named volume ext4 (equivalent) |
+| **Background** | Requires `screen` | Built-in container lifecycle |
+| **Portability** | Tied to this machine's WSL setup | Any Docker host |

Dockerfile

@@ -0,0 +1,35 @@
+# Stage 1: Build Zoekt binaries
+FROM golang:1.24-alpine AS zoekt-builder
+RUN apk add --no-cache git
+RUN go install github.com/sourcegraph/zoekt/cmd/zoekt-index@latest && \
+    go install github.com/sourcegraph/zoekt/cmd/zoekt-webserver@latest
+
+# Stage 2: Install Node.js dependencies (compile native modules for Linux)
+FROM node:22-slim AS node-builder
+RUN apt-get update && apt-get install -y build-essential python3 && rm -rf /var/lib/apt/lists/*
+WORKDIR /app
+COPY package.json package-lock.json ./
+RUN npm ci --omit=dev
+
+# Stage 3: Runtime
+FROM node:22-slim
+RUN apt-get update && apt-get install -y --no-install-recommends lsof procps && rm -rf /var/lib/apt/lists/*
+WORKDIR /app
+
+COPY --from=zoekt-builder /go/bin/zoekt-index /go/bin/zoekt-webserver /usr/local/bin/
+COPY --from=node-builder /app/node_modules ./node_modules
+COPY src/ ./src/
+COPY public/ ./public/
+COPY package.json config.docker.json docker-entrypoint.sh ./
+
+RUN chmod +x docker-entrypoint.sh && mkdir -p /data
+
+ENV NODE_ENV=production
+ENV NODE_OPTIONS="--max-old-space-size=3072"
+
+EXPOSE 3847
+
+HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 \
+  CMD node -e "fetch('http://127.0.0.1:3847/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))"
+
+ENTRYPOINT ["./docker-entrypoint.sh"]

config.docker.json

@@ -0,0 +1,13 @@
+{
+  "projects": [],
+  "exclude": ["**/Intermediate/**", "**/Binaries/**", "**/ThirdParty/**",
+              "**/__ExternalActors__/**", "**/__ExternalObjects__/**", "**/Developers/**"],
+  "service": { "port": 3847, "host": "0.0.0.0" },
+  "data": {
+    "dbPath": "/data/db/index.db",
+    "mirrorDir": "/data/mirror",
+    "indexDir": "/data/zoekt-index"
+  },
+  "zoekt": { "webPort": 6070, "parallelism": 4, "reindexDebounceMs": 5000 },
+  "watcher": { "debounceMs": 100 }
+}

docker-compose.dev.yml

@@ -0,0 +1,12 @@
+services:
+  unreal-index:
+    volumes:
+      - ./src:/app/src:ro
+      - ./public:/app/public:ro
+      - unreal-index-db:/data/db
+      - unreal-index-mirror:/data/mirror
+      - unreal-index-zoekt:/data/zoekt-index
+    ports:
+      - "3847:3847"
+      - "6070:6070"
+    command: ["node", "--watch", "src/service/index.js"]

docker-compose.yml

@@ -0,0 +1,27 @@
+services:
+  unreal-index:
+    build: .
+    container_name: unreal-index
+    ports:
+      - "3847:3847"
+    volumes:
+      - unreal-index-db:/data/db
+      - unreal-index-mirror:/data/mirror
+      - unreal-index-zoekt:/data/zoekt-index
+    mem_limit: 4g
+    memswap_limit: 4g
+    environment:
+      - NODE_OPTIONS=--max-old-space-size=3072
+    restart: unless-stopped
+    stop_grace_period: 30s
+    healthcheck:
+      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:3847/health').then(r=>r.ok?process.exit(0):process.exit(1)).catch(()=>process.exit(1))"]
+      interval: 15s
+      timeout: 5s
+      start_period: 30s
+      retries: 3
+
+volumes:
+  unreal-index-db:
+  unreal-index-mirror:
+  unreal-index-zoekt:

docker-entrypoint.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+set -e
+mkdir -p /data/db /data/mirror /data/zoekt-index
+if [ ! -f /app/config.json ]; then
+  cp /app/config.docker.json /app/config.json
+fi
+exec node src/service/index.js

src/service/index.js

@@ -42,7 +42,7 @@ class UnrealIndexService {
   }
 
   async loadConfig() {
-    const configPath = join(__dirname, '..', '..', 'config.json');
+    const configPath = process.env.UNREAL_INDEX_CONFIG || join(__dirname, '..', '..', 'config.json');
 
     if (!existsSync(configPath)) {
       throw new Error(
@@ -69,8 +69,11 @@ class UnrealIndexService {
     }
 
     // Validate projects
-    if (!this.config.projects || !Array.isArray(this.config.projects) || this.config.projects.length === 0) {
-      throw new Error(`config.json has no projects configured.`);
+    if (!this.config.projects || !Array.isArray(this.config.projects)) {
+      this.config.projects = [];
+    }
+    if (this.config.projects.length === 0) {
+      console.log('[Config] No projects configured — service will receive data via /internal/ingest');
     }
 
     for (const project of this.config.projects) {