[Pipelines] Port full Atlantis pipeline runtime (Phase 1)

[kdhillon]

Summary

Ports the full Atlantis pipeline runtime into the standalone pipelines repo (Phase 1). In this phase the service runs against the SHARED Atlantis Postgres + S3; no data migration and no Atlantis-side changes are included.

What was ported:

• PipelineService (DB-backed), OperationService, PipelineTimingService, serializers, and the Redis operation cache
• All 39 operation executors and the ~26 transitively-required services (storage, file, audio, AWS, auth0, elevenlabs, modal, kitsAI, splice/splicesearch, sagemaker, musicAi, mlgateway, voiceModel, llm, googleGenAI, midiProgressions, voyageai, treehouse, project, …)
• Internal workspace packages replicated under their original names: @atlantis/db (schema mirrors the real Atlantis tables + query/jsonb extensions + pg client), atlantis-types, @atlantis/splice-api; @atlantis/core aliased to the published @splice/atlantis-core
• BullMQ worker entrypoint, DB-backed /api/v1/pipeline-runs and /api/v1/pipeline-operations routes (contract preserved)
• Full test harness against an isolated local docker DB (Postgres :5434 / Redis :6381)

Guardrails (no destructive changes to the shared Atlantis DB)

• railway.toml start command no longer runs db:migrate — drizzle migrations exist only to stand up the local docker test DB
• cleanupDatabase refuses to TRUNCATE unless DATABASE_URL points at a localhost *_test database (NODE_ENV alone is insufficient since vitest sets it automatically)
• Root db:migrate script removed; db:generate retargeted to @atlantis/db
• No DDL / DROP / ALTER / sequence resets / row deletes against the shared DB

Auth

Replaces Auth0/JWT with a trusted-identity prehandler reading x-atlantis-user-id / x-atlantis-api-key-id forwarded by the Atlantis facade, plus an M2M shared secret (x-pipelines-internal-key). Auth fails closed in server/worker/production (PIPELINES_INTERNAL_API_KEY required at boot), and the secret comparison is timing-safe. Ownership/authorization enforced via hasAccessToRun / project.hasAccess.

Local Test Results

• pnpm lint ✅ · pnpm build ✅
• pnpm test:local ✅ — 141 tests / 15 files pass against the local docker Postgres+Redis (external integrations mocked, mirroring Atlantis)

AI Reviews

3-panel review (DB-safety/Guardrails, Security/Bootstrap, Test-harness/Config):

• Critical addressed:
  • cleanupDatabase could TRUNCATE the shared DB when DATABASE_URL was misconfigured — now hard-gated to a localhost *_test target
  • Trusted-identity auth failed open when PIPELINES_INTERNAL_API_KEY was unset — now fails closed at boot in server/worker/production
• Suggestions addressed: timing-safe internal-key comparison; start()/startWorkers() exit non-zero on startup failure; removed duplicate worker SIGINT/SIGTERM handlers; added API /health route for Railway; fixed root db:migrate/db:generate footgun; dropped unused ioredis-mock devDeps
• Remaining nitpicks (deferred): factory FK-zero defaults (matches Atlantis convention), no-op withApiKeyScopes scopes arg, native db.delete/db.update escape hatch (verbatim from Atlantis)

Out of scope (future phases)

Atlantis facade cutover, the phase-2 DB split (pg_dump/copy data migration, sequence resets, dropping cross-DB FKs) and the project_id ownership inversion.

Made with Cursor