Skip to content

About

Python deployment workflows for OCI and ONCE, with private SQLite state and VaultContext recovery

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

PocketDeploy

PocketDeploy provisions and operates OCI, DigitalOcean, Google Cloud and Hetzner servers and ONCE applications, with explicit resource ownership, resumable operations, and encrypted deployment recovery through VaultContext. It manages compute, firewall rules, DNS, outbound email configuration, and GitHub deployment access from versioned configuration.

Scope and recovery

PocketDeploy supports OCI, DigitalOcean Droplets, Google Compute Engine and Hetzner Cloud, existing provider networking, ONCE application delivery, Cloudflare DNS, Resend sending domains, and GitHub deployment environments. One active operator per deployment is supported; local locking does not coordinate operators on different machines. Existing production infrastructure requires a separate ownership-transfer procedure.

Vault checkpoints preserve deployment state, configuration bindings, and SSH authority. Application databases and uploaded files need their own backups and restore procedures. Restoring a controller checkpoint does not restore an application or establish that its data is recoverable.

The implementation uses Python and CLI tools, with a pinned Blue workflow DAG. colors.yml is desired configuration. A private SQLite file records identities, operations, resolved settings and recovery checkpoints. .envrc and colors.yml are tracked in Git; private bindings (.envrc.private), state (.colors.sqlite) and keys (.ssh/) remain ignored beside them.

The test deployment associated with this checkout was intentionally deleted on 10 October 2026. Its root configuration remains for explicitly authorized recreation. See verification status and history for the last recorded checks and recovery limitations.

Start with setup and the command sequence. Operator details cover configuration, state and ownership, application delivery, DNS, SMTP and GitHub, and Vault recovery.

Setup

Install Nix and devenv, then:

devenv shell
uv sync --locked --extra test
pocketdeploy --help

devenv.nix provides Python 3.12, uv, OCI CLI, Google Cloud CLI, OpenSSH, Git, GitHub CLI, curl, jq, SQLite and a pinned VaultContext CLI wrapper. devenv.lock and uv.lock pin the development environment and Python dependencies. Blue is pinned to a tested Git commit. direnv allow is optional; .envrc loads devenv and an optional ignored .envrc.private. Nothing automatically sources private files during deployment or restoration.

The development shell supplies Node.js 22 and installs the root package-lock.json with npm, adding node_modules/.bin to PATH for the pinned Cloudflare and Resend CLIs.

The portable launcher ./pocketdeploy is a symlink to the executable bundled in skills/pocketdeploy/. Copy that executable into a deployment repository or onto PATH; uv fetches its immutable PocketDeploy package pin and Python dependencies. It runs the published package, even inside this source checkout. For local development use uv run pocketdeploy or devenv’s pocketdeploy command. External tools still come from devenv or your PATH. Installed packages also provide the pocketdeploy command. Vault workflows require an authenticated account and an interactive user-unlocked session. Outside devenv, install vaultcontext separately or set its vault-command path.

For a new deployment, copy examples/colors.yml into its repository as colors.yml. The root configuration in this checkout belongs to the deleted test deployment; do not reuse its identity for a new deployment. Replace the example OCI identifiers with your existing compartment, public subnet and availability domain. Configure your OCI CLI authentication profile; the default auth mode is security_token. The existing subnet/VCN, internet gateway, routes and subnet security lists remain externally managed. Inherited subnet rules can grant access beyond the dedicated NSG; an NSG does not subtract those permissions.

Compute provider selection

Set provider-compute to oci (the existing default), digitalocean, gcp or hcloud. Use the provider configuration reference and the corresponding examples/colors-digitalocean.yml, examples/colors-gcp.yml or examples/colors-hcloud.yml. Hetzner requires the official hcloud CLI on PATH, separately from the current development shell packages. Google Cloud and DigitalOcean have passed disposable live deployment checks, including HTTPS, updates, reboot and SMTP TLS connectivity. Hetzner has passed initial live compute provisioning; its application validation is still pending. See verification. Application email delivery and populated application backup recovery remain separate checks. Portable launchers include the provider adapters available at their pinned package revision. Previously copied launchers retain that pin until deliberately updated.

Existing VPCs/subnets, routing, cloud accounts/projects and API enablement remain operator-managed. PocketDeploy creates a single public IPv4 Ubuntu 24.04 server, its owned firewall resources and boot storage. Hetzner also owns a separately recorded primary IPv4. Google Cloud supports explicitly configured disposable Local SSD storage. Resizing, replacement, extra persistent disks, adoption of existing reserved addresses and cross-cloud data migration are outside this version. Cloudflare, Resend, GitHub, ONCE and Vault workflows remain shared.

Provider scope is bound to local state and checked during Vault restoration. Hetzner binds location and network mode; its token API does not provide a stable project ID here, so retain the same project through token rotation and rely on recorded resource IDs and ownership labels. Other providers bind their configured account/project/network scope. Switching provider-compute cannot migrate an existing deployment: create a fresh deployment and separately restore application data and cut over DNS. Existing OCI state and checkpoints keep their original scope format.

Provider read recovery

Resend and Cloudflare reads recover from connection failures, timeouts, HTTP 429 throttling and HTTP 500/502/503/504 within the current operation. Only explicitly allowlisted reads use this recovery layer; PocketDeploy does not replay mutations. Authentication, permissions, certificate failures and exhausted daily/monthly quotas stop immediately. Provider response bodies and credentials stay private.

--provider-read-timeout SECONDS sets a per-read recovery budget (integer seconds from 0 to 3600, default 120) for plan, converge and delete, including dry runs. 0 disables PocketDeploy retries while retaining the normal request timeout and any retries performed by the provider CLI. Each read makes at most five subprocess attempts; subprocess execution, provider CLI retries and backoff all consume the same deadline. Backoff uses full jitter with exponential ceilings of 1, 2, 4 and 8 seconds. An available Retry-After takes precedence; when it cannot fit in the remaining budget, the read stops rather than retrying early. SMTP reads also respect the verification stage deadline. The pinned Cloudflare CLI omits response headers from errors, so its reads use backoff without Retry-After; Resend exposes that header in structured errors.

Transient read recovery continues the same DAG without repeating completed stages. Exhausted recovery preserves the normal durable operation state; uncertain mutation outcomes still require reconciliation before repeating them. This budget does not extend the entire command timeout or change OCI/GitHub retry behavior.

Portable skill and launcher

Install the skill for your agent from this repository:

npx skills add pocketcontext/pocketdeploy --skill pocketdeploy --agent codex --yes

Copy the installed skill's pocketdeploy executable into your deployment repository and make it executable. Track that launcher alongside colors.yml and .envrc. Its package commit is explicit in the script; updating the skill alone does not change a previously copied launcher.

Commands use colors.yml in the caller's current working directory only; parent directories are not searched. -f /path/to/colors.yml explicitly selects a deployment. State and key paths resolve relative to that configuration, never relative to the launcher. The launcher does not load .envrc.private; use direnv or your trusted shell. See the skill for operator instructions.

Commands

pocketdeploy init
pocketdeploy vault-save
pocketdeploy converge
pocketdeploy vault-save

# Routine update
pocketdeploy plan
pocketdeploy converge
pocketdeploy vault-save
pocketdeploy status
pocketdeploy ssh
pocketdeploy ssh --ssh-command 'uname -m'
pocketdeploy delete --dry-run

# Recovery after failed convergence
pocketdeploy steps
pocketdeploy resolve-step --step-id STEP_ID
pocketdeploy clear-pending-marker --app-host app.example.com

# After every application has left a repository
pocketdeploy retire-github-target --repository owner/old-repo --dry-run
pocketdeploy retire-github-target --repository owner/old-repo

init prepares the local deployment UUID, SQLite state, SSH client and host keys, known hosts and an empty .envrc.private if absent. It makes no cloud or Vault calls, does not resolve application bindings and preserves existing files. Repeating a healthy initialization is safe; missing authority for an existing instance still requires recovery. Use init → vault-save → converge → vault-save for a new deployment when encrypted recovery is configured. The first snapshot preserves authority before provisioning; the second records the resulting resource identities.

converge runs a DAG: preflight → keys → compute → host → service DNS → SMTP verification → SMTP credentials → applications → HTTPS verification → GitHub → GitHub keys. SMTP stages are skipped when Resend is disabled. Each external mutation records intent first and the verified result afterward. An unchanged converge makes no compute-provider mutations and no application replacement. It still rotates every GitHub deployment key. Host setup still verifies/enables services, and mutable image tags are pulled to check whether their immutable digest changed. Prefer digest-pinned app images for reproducibility. plan never creates cloud resources or keys; mutable image tags may need verification at converge time. Local lock files may be created, and taking the lock sets the state directory to mode 0700, including on a fresh plan. Use a dedicated deployment directory.

status returns safe inventory, operation and host application status. It never prints resolved environments, credentials, raw Docker metadata, ONCE labels or cloud response bodies. state.last_vault_backup reports the locally acknowledged document/version and UTC saved_at (or null when no receipt is known). Older receipts may lack a timestamp. This makes no Vault call and does not prove checkpoint freshness. ssh is explicitly interactive; commands you choose can print private information in your terminal.

compute-prevent-destroy: true is the default. To delete, deliberately set it to false, review delete --dry-run, then run delete. Its dedicated DAG is: preflight → retire GitHub environments → fence CI and stop applications → delete-services (sending key, SMTP domain and all owned DNS) → terminate compute, delete owned boot volumes and firewall → remove generated SSH keys and known hosts. Preflight checks all recorded targets and host readiness before any external mutation; dry-run uses these same checks without provisioning drift checks. Read permission checks cannot guarantee a later write will be authorized.

Only recorded resources with matching ownership may be removed. Host retirement uses the same lock as deployment, fences queued CI commands and disables restart before gracefully stopping all manifest-owned applications, including apps removed from desired configuration. Unclean stops or ownership changes block deletion. A stopped/terminating instance requires matching shutdown evidence; inspect failures and rerun delete to resume.

Deletion destroys the attached boot volume and previously retained boot volumes whose deployment ownership can be verified. There is no boot-volume retention option; remove the retired compute-retain-boot-volume field and COLORS_PAR_COMPUTE_RETAIN_BOOT_VOLUME override from existing deployments. It also revokes the owned Resend sending key, deletes its sending domain, and deletes owned email and website DNS records by recorded provider identity. Generated operator, server and GitHub SSH keys and known hosts are removed only after remote cleanup succeeds. A failed remote deletion preserves recovery keys. Configured SSH files without generation ownership records are retained and reported rather than blocking deletion of verified cloud resources. They are never adopted. Changed recorded SSH keys and unsafe cleanup paths still block deletion before cloud mutations.

Externally managed networking, management credentials, .envrc.private, Git configuration, local SQLite deletion receipts and Vault history remain. No backup is made automatically. Save any required recovery checkpoint before deletion. The result lists deleted and retained resources. Failures identify their stage and previously completed stages; completed provider deletions are reconciled on retry. GitHub jobs that already captured a retired environment may fail, but host retirement prevents them from restarting applications. After full deletion, converge generates new keys and provisions fresh infrastructure and application storage; it does not restore previous application data.

Command output

Commands print a readable summary to stdout by default. Use --json for scripts: stdout contains exactly one JSON envelope with schema_version: 1, command, ok, and result on success or error on failure. Error objects contain a safe code and message, plus the failing stage when known. Failures may also contain a validated diagnostic object: a caller-authored logical operation, fixed failure category, exit code or signal, elapsed seconds, and HTTP status when known. Application integrations may add validated task/user attribution. No command arguments, stdout, stderr, exception text or response bodies are included; there is no masking or output-excerpt mode. Unknown causes remain unknown rather than triggering a guess or a change in retry behavior. Validated diagnostics also remain attached to failed operation IDs in private state and appear in recent-operation status; retries retain this evidence. Command output is never journaled. Failures still exit nonzero. Output format does not change when redirected. Existing scripts that read JSON must add --json and read the result field.

pocketdeploy status --json | jq '.result'
pocketdeploy plan --json --quiet > plan.json
pocketdeploy converge --quiet
pocketdeploy plan --verbose

Progress and stage durations go to stderr. --quiet suppresses progress, while preserving the stdout result and text-mode error diagnostics. With --json, controller errors are part of the stdout envelope; launcher/runtime diagnostics may still reach stderr. Application failures identify a fixed operation code, such as application_image_pull or application_deploy, with troubleshooting guidance. ONCE deployment includes public hostname verification; a hostname containing an old VPS address must be updated or removed after recreation. Raw subprocess output remains suppressed. Exit codes are 0 for success, 1 for operation failure, 2 for invalid usage and 130 for interruption.

ssh passes through remote stdout/stderr and exit status without a result footer. It rejects --json; capture remote command output directly if needed.

rotate-host-key upgrades an existing instance from its metadata bootstrap key without reconciling applications, services or desired compute settings. It requires existing deployment state and owned compute. The command is idempotent: repeating it finishes an interrupted rotation or keeps the verified replacement. Use it before inspecting or retiring a deployment created by an older release, then run vault-save to checkpoint the new trust material. It makes no cloud mutations. Normal convergence includes this same rotation step.

Configuration

The flat Colors format and COLORS_PAR_* parameter namespace are preserved. For example COLORS_PAR_OCI_OCPUS overrides oci-ocpus. Application bindings:

once:
  applications:
    - host: wiki.example.com
      image: ghcr.io/example/wiki@sha256:REPLACE_WITH_DIGEST
      deploy-strategy: stop-first
      deploy-stop-timeout: 300
      deploy-ready-timeout: 60
      auto_update: false
      auto_backup: false
      smtp: false
      manage-dns: false
      env:
        LITESTREAM_ENDPOINT: app-wikicontext-production-litestream-endpoint

deploy-ready-timeout is the application HTTP readiness budget after replacement, from 1 to 3600 seconds (default 60). deploy-stop-timeout independently bounds container graceful shutdown, from 1 to 3600 seconds (default 300). The SSH operation budget includes all configured applications' shutdown and readiness budgets. Readiness failure retains the pending marker (see clear-pending-marker); it does not automatically roll back a container that may have written data. Host readiness requires a final HTTP 2xx response, following at most five redirects on the same origin within the probe budget. Redirect loops, missing locations and origin or scheme changes fail. This also applies to GitHub delivery. Configure health-path to succeed only when the application is usable. The bundled portable launcher includes readiness budgets and explicit application adoption. Previously copied launchers must be updated before using these features.

The mapping references COLORS_PAR_APP_WIKICONTEXT_PRODUCTION_LITESTREAM_ENDPOINT. Missing bindings fail before provisioning. Values stay strings. Alternatively, an env list contains literal KEY=value strings; there is no shell expansion. Never commit literal secrets. Quote version-like values such as "3.10"; YAML otherwise interprets them as numbers. Unknown configuration fields fail validation.

Core additions to the prior scaffold are schema-version: 1, state-file: .colors.sqlite, optional oci-image-id, once-version: v0.3.3, SSH file paths, and Vault settings. Old compute-api-version, provider-backend and infrastructure-state r2-* fields are not supported. Application storage parameter references can continue to point to existing R2 services.

OCI resource names default to <profile>-once-compute and <profile>-once-firewall. New DigitalOcean Droplets use <profile>-compute; Hetzner uses <profile>-server, <profile>-firewall and <profile>-ipv4. Existing identities and names are preserved. A profile is durable; an ONCE upgrade does not rename it. A separate UUID establishes controller identity and ownership tags. Exact resource IDs, provider scope and lifecycle are stored in SQLite. Names alone never authorize adoption or deletion. Existing application/shared resource names are preserved; application lifetime is independent of a VPS lifetime.

compute-require-existing-state: true rejects missing state. Use it for an established deployment. A recorded instance disappearing is an error, not an instruction to create a replacement. Instance resizing/replacement and boot storage drift fail explicitly in v1. Initial image discovery selects a compatible Canonical Ubuntu 24.04 image and pins its OCID in state; oci-image-id can pin it before creation. ONCE v0.3.3 is checksum-pinned for amd64 and arm64. Docker comes from the Ubuntu distribution package repository; its package version is not pinned.

compute-ssh-ports defaults to ["22"]. Add quoted TCP ports for SSH companion services, for example ["22", "2022"]. These ports share compute-ssh-sources and the existing owned firewall lifecycle on every provider. This setting only controls firewall ingress; it does not configure SSH's listening or connection port. The list must include "22", which the controller uses for SSH. An empty source list disables access to all configured ports.

Adopting an existing application

adopt-app transfers an existing ONCE application's ownership explicitly. It does not replace the container, initialize data, pull a newer image, change DNS, or install GitHub delivery. First disable and drain the previous deployment controller, including queued CI jobs. The required --previous-delivery-disabled flag records your assertion that this has happened; PocketDeploy cannot verify an external controller's lock or queue. Keep that authority disabled permanently after transfer.

Declare the desired application in a reviewed configuration, preserve the existing compute state and SSH authority, then obtain comparison evidence using status --json. The application entry includes full container_id, image_id, volumes (name/destination pairs), and settings_sha256. This fingerprint permits exact comparison without printing private ONCE settings. Never collect raw Docker inspection or ONCE labels in a transcript.

Activate the reviewed configuration before adopting, and use that same desired state for subsequent commands. After adoption, convergence with an older configuration that omits the application removes it. Suspend routine commands during transfer and never revert to an empty application list as a recovery step.

pocketdeploy -f candidate.yml status --json
pocketdeploy -f candidate.yml adopt-app \
  --app-host app.example.com \
  --container-id FULL_CONTAINER_ID \
  --image-id sha256:FULL_IMAGE_ID \
  --volume EXISTING_VOLUME:/storage \
  --settings-sha256 SETTINGS_FINGERPRINT \
  --previous-delivery-disabled

Repeat --volume for every named volume; both names and destinations must match. The operation requires a running container with automatic updates/backups disabled, no bind mounts, a matching repository digest on its locally inspected immutable image, and successful HTTP health. The configured health path must work on the existing image during adoption; change it afterward if the replacement introduces a new endpoint. A retired host, pending application operation, conflicting retained or managed ownership, changed settings/image/volumes, or failed health blocks transfer.

Adoption uses the same host lock as convergence and restricted GitHub delivery. It durably journals exact evidence before committing ownership, records the current settings as its baseline, and leaves application data in place. Repeating identical evidence after interruption is safe while the container and configuration remain unchanged. Conflicting evidence requires investigation; never manufacture ownership by editing the manifest. Next, converge with the active reviewed configuration to apply the new image and bindings, install generic GitHub delivery, and save an encrypted deployment checkpoint. Application data recovery remains separate.

State, interruption and ownership

The local database may contain secrets and is plaintext. The unused desired-configuration snapshot is no longer stored; a matching writable state open deletes legacy snapshots with SQLite secure deletion enabled. New operation fingerprints exclude resolved and literal application environment values. Actual application settings still drive reconciliation. Read-only opens leave old state untouched, and historical backups or filesystem snapshots can retain previous secret-bearing data; this cleanup does not erase those copies.

Database, journals, keys and snapshots stay ignored and private; SQLite uses full synchronous rollback journaling. Schema/profile/provider scope are checked on open. Resolved operation history is bounded; operations with pending steps remain until reconciled. CLI output is independently allowlisted.

Cloud calls are not database transactions. A lost create response leaves a pending operation. The next run discovers a uniquely tagged resource and records it. If no resource is visible, it refuses to issue another create blindly. Operators must establish the prior request's outcome before recovery. Deletion similarly records intent and can resume after the resource has disappeared.

adopt --instance-id PROVIDER_ID recovers an existing, matching, already UUID-tagged instance into existing state with the same deployment UUID. Restore the local state checkpoint first; adoption never creates a replacement deployment identity. It verifies and records the instance, firewall and boot volume, rejects a conflicting recorded instance, and resumes interrupted adoption. It deliberately does not take over an arbitrary production server, rewrite another manager's tags or migrate Terraform state. Full production ownership transfer remains a separate operation.

OCI and DigitalOcean firewall updates add and verify missing rules before removing stale rules. Google Cloud patches and verifies each owned SSH/HTTP rule after its operation completes; the two rules do not form one cloud transaction. An interrupted update retains its intent. If an attempted addition has an uncertain outcome and the desired rules are not visible, automatic retries stop until the operator establishes the outcome. Existing rules are preserved during addition failures; obsolete permissions can remain until recovery completes. Inspection alone does not clear a pending addition. After confirming the original request has settled, an authorized operator can restore the pending desired rules on that same owned NSG through OCI and rerun convergence. Do not clear the local intent to force another request while its outcome remains unknown.

A failed idempotent host request (host-bootstrap, applications-converge) marks its step failed with a fixed error code, never provider output; the ONCE pending marker on the host guards any uncertain delivery. Provider mutations, adopt-app, host-key rotation and quiescence keep their pending steps until a verified retry reconciles them. status reports pending_steps and failed_steps.

pocketdeploy steps
pocketdeploy resolve-step --step-id STEP_ID
pocketdeploy clear-pending-marker --app-host app.example.com

retire-github-target removes one former repository's delivery authority after colors.yml no longer references it. Under the host lock, and refusing while any deployment runs, it removes the host key and dispatcher files, then deletes the owned GitHub environment, then the local key pair and state records. A failed run can be repeated. Applications and other repositories are not touched.

steps lists pending step IDs, kinds and whether they are resolvable, without payloads or provider calls. resolve-step marks one stale pending step of an idempotent kind failed and records a resolve-step operation receipt; it refuses steps that guard uncertain provider mutations. clear-pending-marker removes one host's failed ONCE delivery marker only when the host lock is free (no running converge or CI deploy), the marker belongs to this deployment and is not a removal/retirement marker, the host is not retired, and no manifest entry, retained entry or ONCE-labelled container exists for the host. Otherwise it refuses and leaves the marker. It does not inspect or remove Docker volumes; confirm that no data needs recovery first. Repeating it after success reports that no marker remains.

One active operator per deployment is supported. A local advisory lock prevents overlap on one filesystem; Vault snapshots do not coordinate different machines. Stop the old operator before restoring on another machine and always reconcile cloud reality before mutations. Do not use a shared network filesystem for state.

Application delivery

SSH client and server Ed25519 keys are generated inside the deployment's .ssh/ directory. Cloud-init receives a bootstrap server key through provider metadata/user data; the OCI CLI request body travels over stdin, but instance metadata is not secret storage. SSH pins the bootstrap public key on first contact. Before sending application configuration or SMTP credentials, convergence installs a replacement server key and verifies it through a new SSH connection. The replacement key, its instance binding and rotation checkpoint live in the private SQLite state and are included in Vault snapshots. GitHub receives the replacement public key. Interrupted rotation reuses its recorded key; ordinary commands refuse to use unverified bootstrap trust and require rotate-host-key or convergence to finish first. The .ssh/ server key files retain bootstrap authority only. A checkpoint saved before rotation cannot recover the replacement key; save a new Vault checkpoint after rotation and preserve local state until it succeeds.

The bootstrap connection still relies on the initial key: an attacker who can read metadata and intercept that first connection could impersonate the host. Rotation removes enduring reliance on the metadata key; it does not independently authenticate first contact against that attacker. Container metadata filtering is defense in depth, not a substitute for protecting bootstrap access. Full host bootstrap installs a Docker pre-start hook blocking forwarded traffic to IMDS, including after Docker restarts. This does not restrict root or host-network processes. The standalone rotation command changes SSH trust only. Missing keys for an existing instance require restoration; they are not silently regenerated. When a private key path is customized, its public key defaults to that path plus .pub, including in Vault recovery sets. An explicit public key path overrides this default. The .ssh/github-<repository-hash> key paths are reserved for transient GitHub authority and cannot be used for other deployment files.

Application environment and configuration updates are reconciled, including changes to an existing hostname. Secrets travel to the host over SSH stdin; ONCE's current CLI requires host-local --env arguments, visible to sufficiently privileged processes on that trusted host. Raw commands/output are not logged.

rolling is suitable for disposable/stateless services. stop-first resolves an immutable image, disables the old restart policy, requires a clean graceful stop, then updates and verifies one replacement with the same named volumes. Automatic ONCE image updates are disabled. A durable host pending marker blocks blind retry after uncertainty; inspect the app/data before recovery. There is no automatic rollback after a possible database migration. Host locks do not fence another host. Removed owned apps retain their data volumes.

Stateful applications can configure an application-owned preparation command:

deploy-strategy: stop-first
deploy-prepare-command: [pocketcontext-container, prepare-upgrade]
deploy-prepare-timeout: 900

The command is an argv list supplied only by operator configuration, executed inside the exact existing managed container without a shell and under the host lock. It runs only when replacement is needed, after resolving images and before stopping the container. Exit zero must mean preparation (including any required backup verification) succeeded; command output is suppressed. The timeout is 1–3600 seconds, default 900. New instances do not run it. The hook is policy, so adding it alone does not recreate an unchanged healthy container.

A durable pending marker precedes preparation. Any failure, timeout or interruption retains it: the application may already be paused even when its container is running. Inspect application backup/paused state and follow its recovery procedure; never blindly repeat preparation or restart after an uncertain result. Later replacement failures likewise require operator recovery. No database/image rollback is automatic. Application preparation must fit its own timeout; ONCE v0.3.3 still has a fixed 120-second startup deadline for the replacement.

ONCE automatic backups and automatic updates remain unsupported. Managed DNS uses Cloudflare; managed sending domains use Resend. The local verification fixture uses HTTP with an explicit Host header; this is not public TLS validation. Clearing an application's final environment binding fails because the pinned ONCE CLI cannot express that update. Full adoption of once-pocketcontext-v2 is not implemented or performed.

DNS, SMTP and GitHub delivery

Use devenv shell for pinned cf, resend and gh tools. Cloudflare 1.0.0-beta.14 and Resend 2.23.0 dependencies are locked in the root package-lock.json and installed by devenv using npm; Cloudflare CLI is beta. s-nail runs only on the deployment host; host setup installs the Ubuntu distribution package when at least one managed application enables smtp: true. Its version is not pinned, and no local s-nail is required.

provider-dns: cloudflare
cloudflare-zone-id: YOUR_32_CHARACTER_ZONE_ID
provider-smtp: resend
smtp-domain: notifications.bigconfig.online
smtp-from: mail@notifications.bigconfig.online
smtp-port: 2587
smtp-security: starttls
resend-region: eu-west-1
once:
  applications:
    - host: www.bigconfig.online
      image: ghcr.io/pocketcontext/pocketcontext-website:latest
      github: pocketcontext/pocketcontext-website
      auto_update: false
      deploy-strategy: rolling
      smtp: true

For new configurations, use Resend port 2587 with smtp-security: starttls. Allowed pairs are 587/2587 with starttls and 465/2465 with implicit-tls. If omitted, the port remains 465 for backward compatibility; security is derived from the selected port. The explicit smtp-test client requires the configured TLS mode and verifies the server certificate.

ONCE passes server, port, username, password and sender into the application; it does not provide a universal SMTP TLS-mode switch. smtp-security controls the test client, not application mail libraries. Configure each application's supported TLS settings separately. For example, Basecamp Fizzy defaults to STARTTLS; implicit TLS additionally requires its SMTP_TLS=true setting. Do not infer application email success from the host-level probe alone. DigitalOcean documents blocking standard SMTP ports by default; verify 2587 connectivity on the intended Droplet before relying on delivery.

Supply CLOUDFLARE_API_TOKEN and RESEND_API_KEY through your trusted shell. Cloudflare needs zone read and DNS edit on the selected existing zone. Resend needs domain and API-key management. Use authenticated gh with repository administrator access for environment provisioning. These management credentials are not sent to GitHub Actions or the VPS. A domain-scoped sending key is stored privately in SQLite and transmitted to the host via SSH stdin.

On the VPS, ONCE deploy/update receives application environment values and SMTP credentials as command-line arguments. Processes permitted to inspect host process arguments can read them while ONCE runs; controller-side stdin transport does not prevent this host-side exposure.

The first implementation supports one Cloudflare zone and one sending domain per deployment. Existing unowned DNS records, Resend domains and GitHub environments require explicit recovery/ownership resolution; matching names never authorize adoption. With Cloudflare enabled, every application host defaults to a managed proxied A record (Auto TTL, 1); explicit manage-dns: false excludes a host. The A record targets the deployment's verified public IPv4. SMTP records remain DNS-only with TTL 300. With provider-dns: no-infra, application DNS remains externally managed; an explicit manage-dns: true requires Cloudflare.

After applications are ready, HTTPS verification connects to the recorded origin IP while retaining the application hostname for SNI and certificate validation. It then checks public HTTPS through Cloudflare with a unique query and Cache-Control: no-cache, no-store. Both checks require a successful response; origin redirects stay on the same HTTPS hostname. Configure health endpoints to bypass custom cache rules. PocketDeploy does not change zone-wide TLS settings, WAF rules or cache rules; preserve Full (strict) for the origin connection. GitHub applications with externally managed DNS also receive both checks.

Domain verification has its own DAG stage. For a domain that is not yet verified, it triggers verification once, then polls the recorded domain every ten seconds until verified. Already verified domains skip the trigger and polling. Convergence continues automatically without repeating compute, host setup or DNS reconciliation. SMTP credentials and applications wait for successful verification.

Use pocketdeploy converge --smtp-verification-timeout 900 to allow up to 15 minutes for verification. The timeout accepts integer seconds from 0 to 3600, defaulting to 600; 0 performs one immediate check without polling. The option requires converge without --dry-run and is not a colors.yml setting. Stage start and completion, including elapsed duration, appear on stderr unless --quiet is set. Add --verbose for periodic waiting updates. Authentication or domain identity errors stop immediately. Timeout returns smtp_verification_pending and preserves provisioned resources and recovery state; interruption also preserves them. After resolving the cause, a later convergence reconciles the existing resources. A lost Resend domain/key creation response requires operator reconciliation, not blind retries. Keys are reused; rotation is a separate operation.

GitHub environment names equal the durable deployment profile. Applications sharing a GitHub repository form one ordered deployment group with one environment and restricted SSH key. Each target retains its own volume and runtime bindings. The controller installs the key, main-only branch policy, SSH_PRIVATE_KEY environment secret and SERVER_IP, SERVER_USER, SSH_KNOWN_HOSTS, SITE_URL, SITE_URLS, POCKETDEPLOY_PROFILE, POCKETDEPLOY_DEPLOYMENT_ID variables. Every converge rotates every configured GitHub deployment key: it installs a new restricted public key alongside the old one, updates the SSH_PRIVATE_KEY secret, then removes the old host authority. The pending key is kept in ignored .ssh/github-<repository-hash> files only while rotation is pending; an interrupted rotation resumes with the same key. After the github step succeeds, the final github-keys step deletes both local files, including files left by older versions. If the github step fails, github-keys does not run and pending keys stay for the retry. init creates no GitHub keys, and Vault snapshots exclude them. plan, status and dry runs never generate, rotate or delete keys. Every converge therefore updates GitHub secrets and host authorized_keys, repairs a secret changed manually on GitHub, and invalidates a leaked deployment key. CI jobs that read the old secret before a converge and connect after it may fail and need a re-run. Provisioning an environment and its settings is not atomic; a release during setup may fail until convergence completes.

The website workflow discovers all existing environments after publishing its image. Every environment is an active target, without an opt-in flag. Missing settings fail the target; no environments skips deployment. The current workflow uses no-command SSH to update the configured image tag. The restricted handler also supports deploy ghcr.io/owner/repo@sha256:DIGEST; other commands are rejected. For a group this digest applies to every configured target, in configuration order. SITE_URLS is a JSON array of their public origins; SITE_URL remains the first origin for older workflows. The host lock spans the complete group. Images are resolved once per reference before any writer pauses; each target finishes health and volume verification before the next begins. A failure stops the group, preserves completed targets and fences the failed target with its pending marker. Narrowing the group's configured targets replaces its policy without touching other repositories' restricted keys. No automatic group rollback is attempted. Retiring an environment requires accounting for queued/running workflow snapshots.

pocketdeploy smtp-test --to YOUR_TEST_ADDRESS

This explicit command sends one message from smtp-from using s-nail on the VPS, the configured Resend port and certificate-verified TLS. Credentials travel over SSH stdin and use a temporary private configuration file, cleaned afterward. Success means SMTP acceptance, not inbox delivery. Convergence never sends a test message. A static website does not acquire email functionality from SMTP provisioning; its enquiry backend must consume email settings separately. Outbound SMTP does not create a receiving mailbox.

create has been removed without an alias; use converge for initial provisioning and subsequent updates.

Vault recovery

vault-id: YOUR_DEDICATED_VAULT_ID
vault-command: vaultcontext
# Optional known state document, also remembered locally:
# vault-state-document-id: DOCUMENT_ID

After vaultcontext login and user-operated vaultcontext unlock --timeout 3600:

pocketdeploy vault-save
pocketdeploy vault-restore \
  --document DOCUMENT_ID --version VERSION_ID --destination /absolute/path/to/recovery-check

Recovery files are .envrc.private, the SSH operator key pair, server key pair and known hosts, plus a consistent SQLite snapshot. They must exist before a complete recovery set can be saved. Create an empty private bindings file if the deployment needs none. File versions upload first; the SQLite snapshot uploads last and references their exact versions.

colors.yml, .envrc and package code come from Git. The snapshot records the configuration content SHA256 and Git commit when available; the content hash is authoritative, so unrelated Git commits do not prevent recovery. Preserve uncommitted configuration separately or commit it before saving: a commit ID alone cannot recover local edits. GitHub deployment keys are disposable and are excluded from Vault; convergence recreates missing keys using the recovered operator SSH authority and GitHub administrator access.

Before restoring, check out matching colors.yml in the destination directory. If -f selects a different configuration filename, use that same filename in the destination directory. Both the selected configuration and destination configuration must match the checkpoint hash. Restore never replaces configuration, even with --overwrite. Older snapshots containing configuration and GitHub keys remain readable: their configuration is staged only to verify its hash, and their GitHub keys are skipped. Existing Vault documents and historical versions are not deleted. The snapshot does not need to contain its own newly assigned Vault version ID. Vault's file limit is 8 MiB. Vault operations are explicit: converge, delete and failed workflows never save automatically or require Vault access. The retired vault-save-after-run field is rejected, including when false; remove it from existing configuration and remove COLORS_PAR_VAULT_SAVE_AFTER_RUN from the operator environment. Older Git configurations may contain this retired field; migrate it before creating a new recovery checkpoint.

Run vault-save after successful mutations and after failures that changed local state while a complete recovery set exists. Successful deletion removes generated keys, so a new complete checkpoint requires init to regenerate local authority or converge to recreate the deployment first. Preserve the local deletion receipt until then. Until that succeeds, Vault contains an older checkpoint and may lack new resource identities or operation outcomes. Preserve local state and reconcile cloud reality when recovering from an older snapshot. If saving fails, fix Vault access and retry vault-save; do not rerun deployment just to retry its backup.

Restore stages and validates all files before publishing them, with the database last. Existing destinations require --overwrite. A filesystem interruption between file replacements requires repeating restore; multi-file publication is not atomic. Restore never executes private bindings or starts deployments. Review the Git configuration and run plan before taking over. If the restored config contains machine-specific executable paths, update them locally.

Keep independent encrypted recovery material if VaultContext itself depends on the infrastructure being recovered. Vault encrypts remote copies, not the local working database. Passphrases are entered privately by the user, never retrieved from configuration or supplied in command arguments.

Development

devenv shell
uv sync --locked --extra test
uv run pytest -q
uv build
uv run python scripts/test-launcher.py

Tests use synthetic providers and state. They cover interrupted creates/deletes, ownership mismatches, immutable drift, private snapshot recovery, secret-safe errors, SSH host trust, stop-first delivery and idempotent convergence. Cloud mutation is never part of the default suite. Live verification results are in docs/verification.md when available; they are distinct from mocked coverage.

Publish and test package changes before updating the Git commit in the skill launcher. Pin a full published commit, never a moving branch. The root symlink keeps one launcher payload; commit the new pin after testing a copied launcher outside this checkout. The launcher integration check uses synthetic missing-state fixtures and never accesses a cloud account.

--help always prints ordinary help text, including with --json. A JSON usage error has command: null when argument parsing cannot identify a command.

Running ./pocketdeploy with no arguments prints the same help as --help to stdout and exits 0, without loading configuration or contacting providers. Unknown commands and a lone --json remain usage errors (exit 2).

Verbose diagnostics

Use pocketdeploy plan --verbose or pocketdeploy converge --verbose to see individual OCI and SSH requests start, finish, and report elapsed time on stderr. Long-running requests periodically report that they are still waiting. Labels are authored by PocketDeploy; arguments, credentials and raw provider responses are never printed. --json --verbose keeps one JSON result on stdout. --verbose and --quiet conflict and exit 2 before deployment work begins.

For oci-auth: security_token, cloud workflows check the selected profile's local token expiry before invoking OCI. An expired token fails with a safe renewal instruction instead of waiting for a provider request. This local check does not prove a token is otherwise valid; OCI may still reject revoked or invalid credentials. init and Vault workflows do not require OCI credentials.

Explicit DNS ownership transfer

adopt-dns transfers one existing application A record to this deployment. Configure Cloudflare and the managed application first. Disable and drain the previous DNS manager, checkpoint its state, and obtain the exact zone ID, record ID and current IPv4. The address must match this deployment's recorded owned compute. Multiple A records or AAAA/CNAME conflicts are refused.

Use pocketdeploy dns-evidence --dns-host app.example.com --json to obtain the allowlisted current record evidence without changing state or providers.

The --dns-settings-sha256 fingerprint is computed from the current record's allowlisted fields type, name, content, proxied, ttl, and comment (missing comment is null): SHA-256 of UTF-8 JSON with sorted keys and separators (',', ':'). Filter provider responses inside the inspecting process; never print raw provider metadata.

pocketdeploy adopt-dns --dns-host app.example.com \
  --dns-zone-id EXACT_ZONE_ID --dns-record-id EXACT_RECORD_ID \
  --dns-ipv4 CURRENT_OWNED_IPV4 --dns-settings-sha256 EXACT_FINGERPRINT \
  --previous-dns-manager-disabled

The command journals exact evidence before changing only the record's ownership comment. It preserves record identity, address, proxy setting and TTL. Retry an interrupted transfer with the original evidence; do not clear its journal or manually edit SQLite. Ordinary convergence and deletion refuse an incomplete transfer. After adoption, plan and converge apply the normal proxied-A policy. Keep the previous manager disabled and save a new Vault checkpoint before removing its obsolete local state.

Python integration

Applications can depend on a pinned PocketDeploy package and reuse its compute adapters, state journal, host keys and SSH transport. Host.run_python(connection, source, payload, timeout=1200) runs trusted Python as root over pinned SSH, passing the JSON payload through stdin and parsing JSON stdout. Put credentials in the payload, never in source text. Remote errors suppress stdout/stderr. Host.ssh(connection, command=None) provides an interactive or command session. These interfaces share the same transport as ONCE deployment operations.

Application controllers can use pocketdeploy.runtime.session(config, read_only=False, allow_create=False) to open matching state and its provider/host under a local deployment lock. Read-only plans with allow_create=True leave a missing state absent. recorded_operation(state, command, desired_hash) records success, failure and interruption. initialize(config, state, host, root) prepares local authority. delete_compute(config, state, cloud, host, operation) is for compute-only applications: it enforces destruction protection, refuses service resources and removes keys only after all recorded resources are gone. Callers must stop their application and finish data copying before deletion.

pocketdeploy.dns.CloudflareDNS(config, state, hosts) manages explicit A-record hostnames independently of ONCE. Use preflight(), plan(connection), converge(connection, operation_id), plan_delete() and delete(operation_id) under the deployment lock. It shares record ownership, pagination, pending-write reconciliation and deletion checks with ONCE DNS. Proxying defaults to true with Auto TTL; constructor arguments proxied and ttl can select another policy. Without cloudflare-zone-id, zone discovery requires one unambiguous common zone. COLORS_PAR_CLOUDFLARE_API_TOKEN maps to the CLI token only in its child environment; the operator environment remains unchanged.

For migration to new owned compute, evidence(host) returns allowlisted record identity and a routing-settings fingerprint. adopt(connection, operation_id, evidence) requires exact evidence and previous_dns_manager_disabled: true. It preserves the source address, record ID, proxy and TTL while transferring the ownership comment. A separate convergence updates routing to the target. Keep the previous manager disabled, preserve interrupted-operation state and coordinate application writers before switching production traffic.

Host.prepare_transport(connection) prepares pinned SSH arguments once on the controller thread. Its immutable transport supports concurrent run_python, tunnel(local_port, remote_port) as a loopback-only context manager, and rsync_shell() for the rsync remote-shell option. It never reopens controller state from worker threads. pocketdeploy.ssh_config.update(payload, owner=...) maintains application-owned SSH aliases under one shared local lock. Owners must keep their stable uppercase marker (Walter uses WALTER). Aliases live in ~/.ssh/config.d/00-<owner-lowercase>-<profile>.conf; the main SSH configuration is never rewritten. Configure an unconditional include before SSH options that could override the managed connection settings, for example:

Host *
    Include config.d/*

Host *
    # Your default SSH options follow here.

Installation requires that the include loads the managed file with its pinned host trust and explicit identity settings taking precedence. Conflicting aliases, unsafe files and ambiguous include layouts are refused. Existing managed blocks in the main configuration require deliberate removal before installing a drop-in; the updater does not silently migrate them. Removal deletes only the matching owned drop-in and leaves the main configuration and other deployments intact. check_only validates installation without writing files. Copied launchers and applications pinned to older PocketDeploy commits retain their previous behavior until their dependency is deliberately updated.

Structured diagnostics are available in pocketdeploy.diagnostics: build_diagnostic(operation, category, returncode=..., elapsed_seconds=...) and validate_diagnostic(value, allowed_operations=...). Use fixed operation names, never arguments or credentials. Treat remote JSON as untrusted: pass an explicit operation allowlist and explicit task/user allowlists if accepting attribution. DeployError(diagnostic=...) and Reporter.failure(diagnostic=...) preserve this allowlisted evidence. The diagnostics module uses only the standard library and can be injected into standalone remote programs before their source executes.

Host.run_python and prepared transport accept optional progress(stage, event) and allowed_stages arguments. Remote programs may emit newline-delimited JSON objects containing exactly stage and event (started, completed, failed) on stderr. Only allowlisted events are delivered, on the calling thread; raw stderr is discarded and buffers are bounded. output.remote_progress(stages) provides a callback that maps these events to verbose timed spans and heartbeats. Final JSON and process status remain authoritative; events never complete durable state. This allows one remote lock to cover all provisioning stages.

OCI KMS signing authority

Signing infrastructure has a separate lifetime from applications and compute. Configure one authority per OCI scaffold:

oci-region: eu-frankfurt-1
oci-kms:
  name: identitycontext
  tenancy-id: ocid1.tenancy.oc1.example
  binding-prefix: APP_IDENTITYCONTEXT

Provide the dedicated runtime identity's existing mailbox/alias privately as COLORS_PAR_APP_IDENTITYCONTEXT_OCI_USER_EMAIL (the prefix follows binding-prefix). OCI identity-domain tenancies require a unique email for user creation. Convergence validates this binding before making cloud changes; plans do not require it. The mailbox is not an application signing credential.

After init (or restoring the existing deployment checkpoint), use pocketdeploy kms-plan, then pocketdeploy kms-converge. These commands do not converge applications, rotate GitHub keys, or require application environment bindings. The controller creates a standard shared DEFAULT vault, an HSM-protected RSA3072 key, and a dedicated IAM user/group. Its policy grants only KEY_SIGN and KEY_READ on the exact key; the user has API-key capability only. The controller rejects unexpected group members, user group memberships, API keys and policy changes. Tenancy-wide policies remain the operator's authority; these checks cannot prove that no other policy grants additional access. The operator needs OCI permissions to create these KMS and IAM resources.

kms-converge writes an API authentication key to .ssh/kms-runtime.pem and shell exports to .ssh/kms-runtime.env, with 0600 files in a 0700 directory. Neither file contains the HSM signing private key, which stays in OCI. Do not print or commit these files. Source the generated exports in a trusted shell, then transfer the app-specific bindings into the scaffold's ignored .envrc.private using a private-file operation. The prefix above produces COLORS_PAR_APP_IDENTITYCONTEXT_ bindings with these suffixes:

  • OCI_TENANCY_ID, OCI_USER_ID, OCI_FINGERPRINT, OCI_REGION, OCI_PRIVATE_KEY
  • KMS_KEY_ID, KMS_KEY_VERSION_ID, KMS_CRYPTO_ENDPOINT, KMS_MANAGEMENT_ENDPOINT

The service uses OCI SHA_256_RSA_PKCS1_V1_5 for JWT RS256 and the configured key version as its JWKS kid. It must select an explicit version and retain the old public keys throughout token expiry and verification-cache overlap. Runtime metadata and public key retrieval use the management endpoint; signing uses the crypto endpoint. Credentials grant no key rotation, export, encryption or deletion authority. Save a vault-save checkpoint after provisioning; it includes both runtime credential files. Missing local API credentials require checkpoint recovery and are never silently replaced.

Provisioning records intent before each mutation and records verified ownership using deployment UUID/role tags. A lost response is recovered only by matching those tags and the pending operation. Names never establish ownership. A missing recorded resource or an uncertain create with no matching resource blocks retries. A still-provisioning vault/key is left pending; retry after OCI reports readiness. For an older pending user creation rejected for missing email, use kms-resolve-user-create --kms-event-id EVENT_ID. It re-reads OCI Audit, requires the exact pending operation tag and runtime user name with HTTP 400/email rejection, and verifies no matching user exists before atomically settling that intent. It never retries the create. Supply the dedicated email binding and run kms-converge afterward. Network failures and uncorrelated audit events remain blocked. Audit read permissions are required only for this recovery command. Existing untagged infrastructure is not adopted by name. An ownership transfer requires a separately reviewed procedure; restoring this deployment's checkpoint recovers its already-owned resources.

pocketdeploy kms-rotate creates one new key version and retains every old version. It does not update exported bindings or running applications. Publish both public keys, then run kms-converge, deliberately select the new version, and deploy the service. If the rotation response is lost, inspect the exact current version and use kms-resolve-rotation --kms-version-id OCID; this verifies its key relationship and completes the pending intent without creating another version. Never retry an uncertain rotation by editing controller state.

Ordinary delete retains all KMS and IAM signing resources, their private credentials and their state. No KMS deletion command is provided. Retirement requires a separate explicit OCI procedure after consumers and retained tokens are accounted for; preserve recovery metadata and use OCI's scheduled-deletion and cancellation process. Restoring a checkpoint never rotates or recreates signing material. OCI KMS outage fails signing closed; application fallback is an independently authorized application feature.

The implementation is validated with synthetic provider responses and local credential-generation tests. A live disposable provisioning/signing/rotation check remains required before claiming provider verification.

Provider references: vault types, key permissions, signing algorithms.

About

Python deployment workflows for OCI and ONCE, with private SQLite state and VaultContext recovery

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages