PocketDeploy provisions and operates OCI, DigitalOcean, Google Cloud and Hetzner servers and ONCE applications, with explicit resource ownership, resumable operations, and encrypted deployment recovery through VaultContext. It manages compute, firewall rules, DNS, outbound email configuration, and GitHub deployment access from versioned configuration.
PocketDeploy supports OCI, DigitalOcean Droplets, Google Compute Engine and Hetzner Cloud, existing provider networking, ONCE application delivery, Cloudflare DNS, Resend sending domains, and GitHub deployment environments. One active operator per deployment is supported; local locking does not coordinate operators on different machines. Existing production infrastructure requires a separate ownership-transfer procedure.
Vault checkpoints preserve deployment state, configuration bindings, and SSH authority. Application databases and uploaded files need their own backups and restore procedures. Restoring a controller checkpoint does not restore an application or establish that its data is recoverable.
The implementation uses Python and CLI tools, with a pinned Blue workflow DAG.
colors.yml is desired configuration. A private SQLite file records identities,
operations, resolved settings and recovery checkpoints. .envrc and colors.yml
are tracked in Git; private bindings (.envrc.private), state (.colors.sqlite)
and keys (.ssh/) remain ignored beside them.
The test deployment associated with this checkout was intentionally deleted on 10 October 2026. Its root configuration remains for explicitly authorized recreation. See verification status and history for the last recorded checks and recovery limitations.
Start with setup and the command sequence. Operator details cover configuration, state and ownership, application delivery, DNS, SMTP and GitHub, and Vault recovery.
Install Nix and devenv, then:
devenv shell
uv sync --locked --extra test
pocketdeploy --helpdevenv.nix provides Python 3.12, uv, OCI CLI, Google Cloud CLI, OpenSSH, Git, GitHub CLI, curl,
jq, SQLite and a pinned VaultContext CLI wrapper. devenv.lock and uv.lock pin the development environment and
Python dependencies. Blue is pinned to a tested Git commit. direnv allow is
optional; .envrc loads devenv and an optional ignored .envrc.private.
Nothing automatically sources private files during deployment or restoration.
The development shell supplies Node.js 22 and installs the root package-lock.json
with npm, adding node_modules/.bin to PATH for the pinned Cloudflare and Resend CLIs.
The portable launcher ./pocketdeploy is a symlink to the executable bundled in
skills/pocketdeploy/. Copy that executable into a deployment repository or onto
PATH; uv fetches its immutable PocketDeploy package pin and Python dependencies.
It runs the published package, even inside this source checkout. For local
development use uv run pocketdeploy or devenv’s pocketdeploy command.
External tools still come from devenv or your PATH. Installed packages also
provide the pocketdeploy command. Vault workflows require
an authenticated account and an interactive user-unlocked session. Outside
devenv, install vaultcontext separately or set its vault-command path.
For a new deployment, copy examples/colors.yml into its repository as
colors.yml. The root configuration in this checkout belongs to the deleted
test deployment; do not reuse its identity for a new deployment. Replace the example
OCI identifiers with your existing compartment, public subnet and
availability domain. Configure your OCI CLI authentication profile; the default
auth mode is security_token. The existing subnet/VCN, internet gateway, routes
and subnet security lists remain externally managed. Inherited subnet rules can
grant access beyond the dedicated NSG; an NSG does not subtract those permissions.
Set provider-compute to oci (the existing default), digitalocean, gcp or hcloud.
Use the provider configuration reference
and the corresponding examples/colors-digitalocean.yml, examples/colors-gcp.yml
or examples/colors-hcloud.yml. Hetzner requires the official hcloud CLI
on PATH, separately from the current development shell packages.
Google Cloud and DigitalOcean have passed disposable live deployment checks,
including HTTPS, updates, reboot and SMTP TLS connectivity. Hetzner has passed
initial live compute provisioning; its application validation is still pending. See
verification. Application email delivery and
populated application backup recovery remain separate checks.
Portable launchers include the provider adapters available at their pinned
package revision. Previously copied launchers retain that pin until deliberately updated.
Existing VPCs/subnets, routing, cloud accounts/projects and API enablement remain operator-managed. PocketDeploy creates a single public IPv4 Ubuntu 24.04 server, its owned firewall resources and boot storage. Hetzner also owns a separately recorded primary IPv4. Google Cloud supports explicitly configured disposable Local SSD storage. Resizing, replacement, extra persistent disks, adoption of existing reserved addresses and cross-cloud data migration are outside this version. Cloudflare, Resend, GitHub, ONCE and Vault workflows remain shared.
Provider scope is bound to local state and checked during Vault restoration.
Hetzner binds location and network mode; its token API does not provide a stable
project ID here, so retain the same project through token rotation and rely on
recorded resource IDs and ownership labels. Other providers bind their configured
account/project/network scope. Switching provider-compute cannot migrate an existing
deployment: create a fresh deployment and separately restore application data and
cut over DNS. Existing OCI state and checkpoints keep their original scope format.
Resend and Cloudflare reads recover from connection failures, timeouts, HTTP 429 throttling and HTTP 500/502/503/504 within the current operation. Only explicitly allowlisted reads use this recovery layer; PocketDeploy does not replay mutations. Authentication, permissions, certificate failures and exhausted daily/monthly quotas stop immediately. Provider response bodies and credentials stay private.
--provider-read-timeout SECONDS sets a per-read recovery budget (integer seconds
from 0 to 3600, default 120) for plan, converge and delete, including dry runs. 0 disables PocketDeploy retries while
retaining the normal request timeout and any retries performed by the provider
CLI. Each read makes at most five subprocess attempts; subprocess execution,
provider CLI retries and backoff all consume the same deadline. Backoff uses
full jitter with exponential ceilings of 1, 2, 4 and 8 seconds. An available Retry-After takes
precedence; when it cannot fit in the remaining budget, the read stops rather
than retrying early. SMTP reads also respect the verification stage deadline.
The pinned Cloudflare CLI omits response headers from errors, so its reads use
backoff without Retry-After; Resend exposes that header in structured errors.
Transient read recovery continues the same DAG without repeating completed stages. Exhausted recovery preserves the normal durable operation state; uncertain mutation outcomes still require reconciliation before repeating them. This budget does not extend the entire command timeout or change OCI/GitHub retry behavior.
Install the skill for your agent from this repository:
npx skills add pocketcontext/pocketdeploy --skill pocketdeploy --agent codex --yesCopy the installed skill's pocketdeploy executable into your deployment
repository and make it executable. Track that launcher alongside colors.yml
and .envrc. Its package commit is explicit in the script; updating the skill
alone does not change a previously copied launcher.
Commands use colors.yml in the caller's current working directory only;
parent directories are not searched. -f /path/to/colors.yml explicitly selects a deployment. State and key
paths resolve relative to that configuration, never relative to the launcher.
The launcher does not load .envrc.private; use direnv or your trusted shell.
See the skill for operator instructions.
pocketdeploy init
pocketdeploy vault-save
pocketdeploy converge
pocketdeploy vault-save
# Routine update
pocketdeploy plan
pocketdeploy converge
pocketdeploy vault-save
pocketdeploy status
pocketdeploy ssh
pocketdeploy ssh --ssh-command 'uname -m'
pocketdeploy delete --dry-run
# Recovery after failed convergence
pocketdeploy steps
pocketdeploy resolve-step --step-id STEP_ID
pocketdeploy clear-pending-marker --app-host app.example.com
# After every application has left a repository
pocketdeploy retire-github-target --repository owner/old-repo --dry-run
pocketdeploy retire-github-target --repository owner/old-repoinit prepares the local deployment UUID, SQLite state, SSH client and host
keys, known hosts and an empty .envrc.private if absent. It makes no cloud or
Vault calls, does not resolve application bindings and preserves existing files.
Repeating a healthy initialization is safe; missing authority for an existing
instance still requires recovery. Use init → vault-save → converge → vault-save for a new deployment
when encrypted recovery is configured. The first snapshot preserves authority
before provisioning; the second records the resulting resource identities.
converge runs a DAG: preflight → keys → compute → host → service DNS →
SMTP verification → SMTP credentials → applications → HTTPS verification → GitHub
→ GitHub keys. SMTP stages are skipped when Resend is disabled.
Each external mutation records intent first and the verified result afterward.
An unchanged converge makes no compute-provider mutations and no application replacement.
It still rotates every GitHub deployment key.
Host setup still verifies/enables services, and mutable image tags are pulled to
check whether their immutable digest changed. Prefer digest-pinned app images
for reproducibility. plan never creates cloud resources or keys; mutable image
tags may need verification at converge time. Local lock files may be created,
and taking the lock sets the state directory to mode 0700, including on a fresh
plan. Use a dedicated deployment directory.
status returns safe inventory, operation and host application status.
It never prints resolved environments, credentials,
raw Docker metadata, ONCE labels or cloud response bodies.
state.last_vault_backup reports the locally acknowledged document/version and
UTC saved_at (or null when no receipt is known). Older receipts may lack a
timestamp. This makes no Vault call and does not prove checkpoint freshness.
ssh is explicitly
interactive; commands you choose can print private information in your terminal.
compute-prevent-destroy: true is the default. To delete, deliberately set it to
false, review delete --dry-run, then run delete. Its dedicated DAG is:
preflight → retire GitHub environments → fence CI and stop applications →
delete-services (sending key, SMTP domain and all owned DNS) → terminate compute,
delete owned boot volumes and firewall → remove generated SSH keys and known hosts.
Preflight checks all recorded targets and host readiness before any external
mutation; dry-run uses these same checks without provisioning drift checks.
Read permission checks cannot guarantee a later write will be authorized.
Only recorded resources with matching ownership may be removed. Host retirement
uses the same lock as deployment, fences queued CI commands and disables restart
before gracefully stopping all manifest-owned applications, including apps removed
from desired configuration. Unclean stops or ownership changes block deletion.
A stopped/terminating instance requires matching shutdown evidence; inspect
failures and rerun delete to resume.
Deletion destroys the attached boot volume and previously retained boot volumes
whose deployment ownership can be verified. There is no boot-volume retention
option; remove the retired compute-retain-boot-volume field and
COLORS_PAR_COMPUTE_RETAIN_BOOT_VOLUME override from existing deployments.
It also revokes the owned Resend sending key, deletes its sending domain,
and deletes owned email and website DNS records by recorded provider identity.
Generated operator, server and GitHub SSH keys and known hosts are removed only
after remote cleanup succeeds. A failed remote deletion preserves recovery keys.
Configured SSH files without generation ownership records are retained and
reported rather than blocking deletion of verified cloud resources. They are
never adopted. Changed recorded SSH keys and unsafe cleanup paths still block
deletion before cloud mutations.
Externally managed networking, management credentials, .envrc.private, Git
configuration, local SQLite deletion receipts and Vault history remain. No backup
is made automatically. Save any required recovery checkpoint before deletion.
The result lists deleted and retained resources. Failures identify their stage
and previously completed stages; completed provider deletions are reconciled on
retry. GitHub jobs that already captured a retired environment may fail, but
host retirement prevents them from restarting applications. After full deletion,
converge generates new keys and provisions fresh infrastructure and application
storage; it does not restore previous application data.
Commands print a readable summary to stdout by default. Use --json for
scripts: stdout contains exactly one JSON envelope with schema_version: 1,
command, ok, and result on success or error on failure. Error objects
contain a safe code and message, plus the failing stage when known.
Failures may also contain a validated diagnostic object: a caller-authored logical
operation, fixed failure category, exit code or signal, elapsed seconds, and HTTP
status when known. Application integrations may add validated task/user attribution.
No command arguments, stdout, stderr, exception text or response bodies are included;
there is no masking or output-excerpt mode. Unknown causes remain unknown rather
than triggering a guess or a change in retry behavior. Validated diagnostics also remain attached to failed operation IDs in private state
and appear in recent-operation status; retries retain this evidence. Command output
is never journaled.
Failures still exit nonzero. Output format does not change when redirected.
Existing scripts that read JSON must add --json and read the result field.
pocketdeploy status --json | jq '.result'
pocketdeploy plan --json --quiet > plan.json
pocketdeploy converge --quiet
pocketdeploy plan --verboseProgress and stage durations go to stderr. --quiet suppresses progress, while
preserving the stdout result and text-mode error diagnostics. With --json,
controller errors are part of the stdout envelope; launcher/runtime diagnostics
may still reach stderr. Application failures identify a fixed operation code, such as
application_image_pull or application_deploy, with troubleshooting guidance.
ONCE deployment includes public hostname verification; a hostname containing an
old VPS address must be updated or removed after recreation. Raw subprocess
output remains suppressed. Exit codes are 0 for success, 1 for operation failure,
2 for invalid usage and 130 for interruption.
ssh passes through remote stdout/stderr and exit status without a result
footer. It rejects --json; capture remote command output directly if needed.
rotate-host-key upgrades an existing instance from its metadata bootstrap key
without reconciling applications, services or desired compute settings. It
requires existing deployment state and owned compute. The command is idempotent:
repeating it finishes an interrupted rotation or keeps the verified replacement.
Use it before inspecting or retiring a deployment created by an older release,
then run vault-save to checkpoint the new trust material. It makes no cloud
mutations. Normal convergence includes this same rotation step.
The flat Colors format and COLORS_PAR_* parameter namespace are preserved.
For example COLORS_PAR_OCI_OCPUS overrides oci-ocpus. Application bindings:
once:
applications:
- host: wiki.example.com
image: ghcr.io/example/wiki@sha256:REPLACE_WITH_DIGEST
deploy-strategy: stop-first
deploy-stop-timeout: 300
deploy-ready-timeout: 60
auto_update: false
auto_backup: false
smtp: false
manage-dns: false
env:
LITESTREAM_ENDPOINT: app-wikicontext-production-litestream-endpointdeploy-ready-timeout is the application HTTP readiness budget after replacement,
from 1 to 3600 seconds (default 60). deploy-stop-timeout independently bounds
container graceful shutdown, from 1 to 3600 seconds (default 300). The SSH
operation budget includes all configured applications' shutdown and readiness
budgets. Readiness failure retains the pending marker (see clear-pending-marker); it does not automatically
roll back a container that may have written data. Host readiness requires a final
HTTP 2xx response, following at most five
redirects on the same origin within the probe budget. Redirect loops, missing
locations and origin or scheme changes fail. This also applies to GitHub delivery.
Configure health-path to succeed only when the application is usable. The bundled portable launcher includes readiness budgets and explicit application
adoption. Previously copied launchers must be updated before using these features.
The mapping references
COLORS_PAR_APP_WIKICONTEXT_PRODUCTION_LITESTREAM_ENDPOINT. Missing bindings
fail before provisioning. Values stay strings. Alternatively, an env list
contains literal KEY=value strings; there is no shell expansion. Never commit
literal secrets. Quote version-like values such as "3.10"; YAML otherwise
interprets them as numbers. Unknown configuration fields fail validation.
Core additions to the prior scaffold are schema-version: 1,
state-file: .colors.sqlite, optional oci-image-id, once-version: v0.3.3,
SSH file paths, and Vault settings. Old compute-api-version, provider-backend
and infrastructure-state r2-* fields are not supported. Application storage
parameter references can continue to point to existing R2 services.
OCI resource names default to <profile>-once-compute and
<profile>-once-firewall. New DigitalOcean Droplets use <profile>-compute;
Hetzner uses <profile>-server, <profile>-firewall and <profile>-ipv4.
Existing identities and names are preserved. A profile is durable; an ONCE upgrade does not rename
it. A separate UUID establishes controller identity and ownership tags. Exact
resource IDs, provider scope and lifecycle are stored in SQLite. Names alone
never authorize adoption or deletion. Existing application/shared resource names
are preserved; application lifetime is independent of a VPS lifetime.
compute-require-existing-state: true rejects missing state. Use it for an
established deployment. A recorded instance disappearing is an error, not an
instruction to create a replacement. Instance resizing/replacement and boot
storage drift fail explicitly in v1. Initial image discovery selects a compatible
Canonical Ubuntu 24.04 image and pins its OCID in state; oci-image-id can pin it
before creation. ONCE v0.3.3 is checksum-pinned for amd64 and arm64. Docker comes
from the Ubuntu distribution package repository; its package version is not pinned.
compute-ssh-ports defaults to ["22"]. Add quoted TCP ports for SSH companion
services, for example ["22", "2022"]. These ports share compute-ssh-sources
and the existing owned firewall lifecycle on every provider. This setting only
controls firewall ingress; it does not configure SSH's listening or connection port.
The list must include "22", which the controller uses for SSH.
An empty source list disables access to all configured ports.
adopt-app transfers an existing ONCE application's ownership explicitly. It does
not replace the container, initialize data, pull a newer image, change DNS, or
install GitHub delivery. First disable and drain the previous deployment
controller, including queued CI jobs. The required --previous-delivery-disabled
flag records your assertion that this has happened; PocketDeploy cannot verify an
external controller's lock or queue. Keep that authority disabled permanently
after transfer.
Declare the desired application in a reviewed configuration, preserve the existing
compute state and SSH authority, then obtain comparison evidence using
status --json. The application entry includes full container_id, image_id,
volumes (name/destination pairs), and settings_sha256. This fingerprint permits
exact comparison without printing private ONCE settings. Never collect raw Docker
inspection or ONCE labels in a transcript.
Activate the reviewed configuration before adopting, and use that same desired state for subsequent commands. After adoption, convergence with an older configuration that omits the application removes it. Suspend routine commands during transfer and never revert to an empty application list as a recovery step.
pocketdeploy -f candidate.yml status --json
pocketdeploy -f candidate.yml adopt-app \
--app-host app.example.com \
--container-id FULL_CONTAINER_ID \
--image-id sha256:FULL_IMAGE_ID \
--volume EXISTING_VOLUME:/storage \
--settings-sha256 SETTINGS_FINGERPRINT \
--previous-delivery-disabledRepeat --volume for every named volume; both names and destinations must match.
The operation requires a running container with automatic updates/backups disabled,
no bind mounts, a matching repository digest on its locally inspected immutable
image, and successful HTTP health. The configured health path must work on the
existing image during adoption; change it afterward if the replacement introduces
a new endpoint. A retired host, pending application operation, conflicting retained
or managed ownership, changed settings/image/volumes, or failed health blocks
transfer.
Adoption uses the same host lock as convergence and restricted GitHub delivery. It durably journals exact evidence before committing ownership, records the current settings as its baseline, and leaves application data in place. Repeating identical evidence after interruption is safe while the container and configuration remain unchanged. Conflicting evidence requires investigation; never manufacture ownership by editing the manifest. Next, converge with the active reviewed configuration to apply the new image and bindings, install generic GitHub delivery, and save an encrypted deployment checkpoint. Application data recovery remains separate.
The local database may contain secrets and is plaintext. The unused desired-configuration snapshot is no longer stored; a matching writable state open deletes legacy snapshots with SQLite secure deletion enabled. New operation fingerprints exclude resolved and literal application environment values. Actual application settings still drive reconciliation. Read-only opens leave old state untouched, and historical backups or filesystem snapshots can retain previous secret-bearing data; this cleanup does not erase those copies.
Database, journals, keys and snapshots stay ignored and private; SQLite uses full synchronous rollback journaling. Schema/profile/provider scope are checked on open. Resolved operation history is bounded; operations with pending steps remain until reconciled. CLI output is independently allowlisted.
Cloud calls are not database transactions. A lost create response leaves a pending operation. The next run discovers a uniquely tagged resource and records it. If no resource is visible, it refuses to issue another create blindly. Operators must establish the prior request's outcome before recovery. Deletion similarly records intent and can resume after the resource has disappeared.
adopt --instance-id PROVIDER_ID recovers an existing, matching, already UUID-tagged
instance into existing state with the same deployment UUID. Restore the local
state checkpoint first; adoption never creates a replacement deployment identity.
It verifies and records the instance, firewall and boot volume, rejects a
conflicting recorded instance, and resumes interrupted adoption. It deliberately does not take over an arbitrary production
server, rewrite another manager's tags or migrate Terraform state. Full production
ownership transfer remains a separate operation.
OCI and DigitalOcean firewall updates add and verify missing rules before removing stale rules. Google Cloud patches and verifies each owned SSH/HTTP rule after its operation completes; the two rules do not form one cloud transaction. An interrupted update retains its intent. If an attempted addition has an uncertain outcome and the desired rules are not visible, automatic retries stop until the operator establishes the outcome. Existing rules are preserved during addition failures; obsolete permissions can remain until recovery completes. Inspection alone does not clear a pending addition. After confirming the original request has settled, an authorized operator can restore the pending desired rules on that same owned NSG through OCI and rerun convergence. Do not clear the local intent to force another request while its outcome remains unknown.
A failed idempotent host request (host-bootstrap, applications-converge)
marks its step failed with a fixed error code, never provider output; the ONCE
pending marker on the host guards any uncertain delivery. Provider mutations,
adopt-app, host-key rotation and quiescence keep their pending steps until a
verified retry reconciles them. status reports pending_steps and failed_steps.
pocketdeploy steps
pocketdeploy resolve-step --step-id STEP_ID
pocketdeploy clear-pending-marker --app-host app.example.comretire-github-target removes one former repository's delivery authority after
colors.yml no longer references it. Under the host lock, and refusing while any
deployment runs, it removes the host key and dispatcher files, then deletes the
owned GitHub environment, then the local key pair and state records. A failed run
can be repeated. Applications and other repositories are not touched.
steps lists pending step IDs, kinds and whether they are resolvable, without
payloads or provider calls. resolve-step marks one stale pending step of an
idempotent kind failed and records a resolve-step operation receipt; it refuses
steps that guard uncertain provider mutations. clear-pending-marker removes one
host's failed ONCE delivery marker only when the host lock is free (no running
converge or CI deploy), the marker belongs to this deployment and is not a
removal/retirement marker, the host is not retired, and no manifest entry,
retained entry or ONCE-labelled container exists for the host. Otherwise it
refuses and leaves the marker. It does not inspect or remove Docker volumes;
confirm that no data needs recovery first. Repeating it after success reports
that no marker remains.
One active operator per deployment is supported. A local advisory lock prevents overlap on one filesystem; Vault snapshots do not coordinate different machines. Stop the old operator before restoring on another machine and always reconcile cloud reality before mutations. Do not use a shared network filesystem for state.
SSH client and server Ed25519 keys are generated inside the deployment's .ssh/
directory. Cloud-init receives a bootstrap server key through provider metadata/user data;
the OCI CLI request body travels over stdin, but instance metadata is not secret
storage. SSH pins the bootstrap public key on first contact. Before sending
application configuration or SMTP credentials, convergence installs a replacement
server key and verifies it through a new SSH connection. The replacement key,
its instance binding and rotation checkpoint live in the private SQLite state
and are included in Vault snapshots. GitHub receives the replacement public key.
Interrupted rotation reuses its recorded key; ordinary commands refuse to use
unverified bootstrap trust and require rotate-host-key or convergence to finish
first. The .ssh/ server key files retain bootstrap authority only. A checkpoint
saved before rotation cannot recover the replacement key; save a new Vault
checkpoint after rotation and preserve local state until it succeeds.
The bootstrap connection still relies on the initial key: an attacker who can
read metadata and intercept that first connection could impersonate the host.
Rotation removes enduring reliance on the metadata key; it does not independently
authenticate first contact against that attacker. Container metadata filtering is
defense in depth, not a substitute for protecting bootstrap access. Full host
bootstrap installs a Docker pre-start hook blocking forwarded traffic to IMDS,
including after Docker restarts. This does not restrict root or host-network
processes. The standalone rotation command changes SSH trust only.
Missing keys for an
existing instance require restoration; they are not silently regenerated.
When a private key path is customized, its public key defaults to that path plus
.pub, including in Vault recovery sets. An explicit public key path overrides
this default. The .ssh/github-<repository-hash> key paths are reserved for
transient GitHub authority and cannot be used for other deployment files.
Application environment and configuration updates are reconciled, including
changes to an existing hostname. Secrets travel to the host over SSH stdin; ONCE's
current CLI requires host-local --env arguments, visible to sufficiently
privileged processes on that trusted host. Raw commands/output are not logged.
rolling is suitable for disposable/stateless services. stop-first resolves an
immutable image, disables the old restart policy, requires a clean graceful stop,
then updates and verifies one replacement with the same named volumes. Automatic
ONCE image updates are disabled. A durable host pending marker blocks blind
retry after uncertainty; inspect the app/data before recovery. There is no
automatic rollback after a possible database migration. Host locks do not fence
another host. Removed owned apps retain their data volumes.
Stateful applications can configure an application-owned preparation command:
deploy-strategy: stop-first
deploy-prepare-command: [pocketcontext-container, prepare-upgrade]
deploy-prepare-timeout: 900The command is an argv list supplied only by operator configuration, executed inside the exact existing managed container without a shell and under the host lock. It runs only when replacement is needed, after resolving images and before stopping the container. Exit zero must mean preparation (including any required backup verification) succeeded; command output is suppressed. The timeout is 1–3600 seconds, default 900. New instances do not run it. The hook is policy, so adding it alone does not recreate an unchanged healthy container.
A durable pending marker precedes preparation. Any failure, timeout or interruption retains it: the application may already be paused even when its container is running. Inspect application backup/paused state and follow its recovery procedure; never blindly repeat preparation or restart after an uncertain result. Later replacement failures likewise require operator recovery. No database/image rollback is automatic. Application preparation must fit its own timeout; ONCE v0.3.3 still has a fixed 120-second startup deadline for the replacement.
ONCE automatic backups and automatic updates remain unsupported. Managed DNS uses
Cloudflare; managed sending domains use Resend. The local verification fixture uses HTTP with an explicit Host
header; this is not public TLS validation. Clearing an application's final
environment binding fails because the pinned ONCE CLI cannot express that update.
Full adoption of once-pocketcontext-v2 is not implemented or performed.
Use devenv shell for pinned cf, resend and gh tools. Cloudflare
1.0.0-beta.14 and Resend 2.23.0 dependencies are locked in
the root package-lock.json and installed by devenv using npm;
Cloudflare CLI is beta. s-nail runs only on the deployment host; host setup installs
the Ubuntu distribution package when at least one managed application enables
smtp: true. Its version is not pinned, and no local s-nail is required.
provider-dns: cloudflare
cloudflare-zone-id: YOUR_32_CHARACTER_ZONE_ID
provider-smtp: resend
smtp-domain: notifications.bigconfig.online
smtp-from: mail@notifications.bigconfig.online
smtp-port: 2587
smtp-security: starttls
resend-region: eu-west-1
once:
applications:
- host: www.bigconfig.online
image: ghcr.io/pocketcontext/pocketcontext-website:latest
github: pocketcontext/pocketcontext-website
auto_update: false
deploy-strategy: rolling
smtp: trueFor new configurations, use Resend port 2587 with smtp-security: starttls.
Allowed pairs are 587/2587 with starttls and 465/2465 with
implicit-tls. If omitted, the port remains 465 for backward compatibility;
security is derived from the selected port. The explicit smtp-test client
requires the configured TLS mode and verifies the server certificate.
ONCE passes server, port, username, password and sender into the application;
it does not provide a universal SMTP TLS-mode switch. smtp-security controls
the test client, not application mail libraries. Configure each application's
supported TLS settings separately. For example, Basecamp Fizzy defaults to
STARTTLS; implicit TLS additionally requires its SMTP_TLS=true setting.
Do not infer application email success from the host-level probe alone.
DigitalOcean documents blocking standard SMTP ports by default; verify 2587
connectivity on the intended Droplet before relying on delivery.
Supply CLOUDFLARE_API_TOKEN and RESEND_API_KEY through your trusted shell.
Cloudflare needs zone read and DNS edit on the selected existing zone. Resend
needs domain and API-key management. Use authenticated gh with repository
administrator access for environment provisioning. These management credentials
are not sent to GitHub Actions or the VPS. A domain-scoped sending key is stored
privately in SQLite and transmitted to the host via SSH stdin.
On the VPS, ONCE deploy/update receives application environment values and SMTP credentials as command-line arguments. Processes permitted to inspect host process arguments can read them while ONCE runs; controller-side stdin transport does not prevent this host-side exposure.
The first implementation supports one Cloudflare zone and one sending domain per
deployment. Existing unowned DNS records, Resend domains and GitHub environments
require explicit recovery/ownership resolution; matching names never authorize
adoption. With Cloudflare enabled, every application host defaults to a managed
proxied A record (Auto TTL, 1); explicit manage-dns: false excludes a host.
The A record targets the deployment's verified public IPv4. SMTP records remain
DNS-only with TTL 300. With provider-dns: no-infra, application DNS remains
externally managed; an explicit manage-dns: true requires Cloudflare.
After applications are ready, HTTPS verification connects to the recorded origin
IP while retaining the application hostname for SNI and certificate validation.
It then checks public HTTPS through Cloudflare with a unique query and
Cache-Control: no-cache, no-store. Both checks require a successful response;
origin redirects stay on the same HTTPS hostname. Configure health endpoints to
bypass custom cache rules. PocketDeploy does not change zone-wide TLS settings,
WAF rules or cache rules; preserve Full (strict) for the origin connection.
GitHub applications with externally managed DNS also receive both checks.
Domain verification has its own DAG stage. For a domain that is not yet verified, it triggers verification once, then polls the recorded domain every ten seconds until verified. Already verified domains skip the trigger and polling. Convergence continues automatically without repeating compute, host setup or DNS reconciliation. SMTP credentials and applications wait for successful verification.
Use pocketdeploy converge --smtp-verification-timeout 900 to allow up to
15 minutes for verification. The timeout accepts integer seconds from 0 to 3600,
defaulting to 600; 0 performs one immediate check without polling. The option
requires converge without --dry-run and is not a colors.yml setting. Stage
start and completion, including elapsed duration, appear on stderr unless
--quiet is set. Add --verbose for periodic waiting updates. Authentication
or domain identity errors stop
immediately. Timeout returns smtp_verification_pending and preserves provisioned
resources and recovery state; interruption also preserves them. After resolving
the cause, a later convergence reconciles the existing resources.
A lost Resend domain/key creation response requires operator reconciliation, not
blind retries. Keys are reused; rotation is a separate operation.
GitHub environment names equal the durable deployment profile. Applications
sharing a GitHub repository form one ordered deployment group with one environment
and restricted SSH key. Each target retains its own volume and runtime bindings.
The controller installs the key, main-only branch policy, SSH_PRIVATE_KEY environment secret and
SERVER_IP, SERVER_USER, SSH_KNOWN_HOSTS, SITE_URL, SITE_URLS,
POCKETDEPLOY_PROFILE, POCKETDEPLOY_DEPLOYMENT_ID variables. Every converge
rotates every configured GitHub deployment key: it installs a new restricted public
key alongside the old one, updates the SSH_PRIVATE_KEY secret, then removes the
old host authority. The pending key is kept in ignored
.ssh/github-<repository-hash> files only while rotation is pending; an
interrupted rotation resumes with the same key. After the github step succeeds,
the final github-keys step deletes both local files, including files left by
older versions. If the github step fails, github-keys does not run and pending
keys stay for the retry. init creates no GitHub keys, and Vault snapshots exclude
them. plan, status and dry runs never generate, rotate or delete keys. Every
converge therefore updates GitHub secrets and host authorized_keys, repairs a
secret changed manually on GitHub, and invalidates a leaked deployment key. CI jobs
that read the old secret before a converge and connect after it may fail and need
a re-run. Provisioning an environment and its
settings is not atomic; a release during setup may fail until convergence completes.
The website workflow discovers all existing environments after publishing its
image. Every environment is an active target, without an opt-in flag. Missing
settings fail the target; no environments skips deployment. The current workflow
uses no-command SSH to update the configured image tag. The restricted handler
also supports deploy ghcr.io/owner/repo@sha256:DIGEST; other commands are rejected.
For a group this digest applies to every configured target, in configuration order.
SITE_URLS is a JSON array of their public origins; SITE_URL remains the first
origin for older workflows. The host lock spans the complete group. Images are
resolved once per reference before any writer pauses; each target finishes health
and volume verification before the next begins. A failure stops the group,
preserves completed targets and fences the failed target with its pending marker.
Narrowing the group's configured targets replaces its policy without touching
other repositories' restricted keys. No automatic group rollback is attempted.
Retiring an environment requires accounting for queued/running workflow snapshots.
pocketdeploy smtp-test --to YOUR_TEST_ADDRESSThis explicit command sends one message from smtp-from using s-nail on the VPS,
the configured Resend port and certificate-verified TLS. Credentials travel over SSH stdin
and use a temporary private configuration file, cleaned afterward. Success means
SMTP acceptance, not inbox delivery. Convergence never sends a test message.
A static website does not acquire email functionality from SMTP provisioning;
its enquiry backend must consume email settings separately. Outbound SMTP does
not create a receiving mailbox.
create has been removed without an alias; use converge for initial provisioning
and subsequent updates.
vault-id: YOUR_DEDICATED_VAULT_ID
vault-command: vaultcontext
# Optional known state document, also remembered locally:
# vault-state-document-id: DOCUMENT_IDAfter vaultcontext login and user-operated vaultcontext unlock --timeout 3600:
pocketdeploy vault-save
pocketdeploy vault-restore \
--document DOCUMENT_ID --version VERSION_ID --destination /absolute/path/to/recovery-checkRecovery files are .envrc.private, the SSH operator key pair, server key pair
and known hosts, plus a consistent SQLite snapshot. They must exist before a
complete recovery set can be saved. Create an empty private bindings file if the
deployment needs none. File versions upload first; the SQLite snapshot uploads
last and references their exact versions.
colors.yml, .envrc and package code come from Git. The snapshot records the
configuration content SHA256 and Git commit when available; the content hash
is authoritative, so unrelated Git commits do not prevent recovery. Preserve
uncommitted configuration separately or commit it before saving: a commit ID
alone cannot recover local edits. GitHub deployment keys are disposable and are
excluded from Vault; convergence recreates missing keys using the recovered
operator SSH authority and GitHub administrator access.
Before restoring, check out matching colors.yml in the destination directory.
If -f selects a different configuration filename, use that same filename in
the destination directory.
Both the selected configuration and destination configuration must match the
checkpoint hash. Restore never replaces configuration, even with --overwrite.
Older snapshots containing configuration and GitHub keys remain readable: their
configuration is staged only to verify its hash, and their GitHub keys are skipped.
Existing Vault documents and historical versions are not deleted.
The snapshot does not need to contain its own newly assigned Vault version ID.
Vault's file limit is 8 MiB. Vault operations are explicit: converge,
delete and failed workflows never save automatically or require Vault access.
The retired vault-save-after-run field is rejected, including when false; remove
it from existing configuration and remove COLORS_PAR_VAULT_SAVE_AFTER_RUN
from the operator environment. Older Git configurations may contain this retired field; migrate it before
creating a new recovery checkpoint.
Run vault-save after successful mutations and after failures that changed local
state while a complete recovery set exists. Successful deletion removes generated
keys, so a new complete checkpoint requires init to regenerate local authority
or converge to recreate the deployment first. Preserve the local deletion receipt
until then. Until that succeeds, Vault contains an older checkpoint and may lack new
resource identities or operation outcomes. Preserve local state and reconcile
cloud reality when recovering from an older snapshot. If saving fails, fix Vault
access and retry vault-save; do not rerun deployment just to retry its backup.
Restore stages and validates all files before publishing them, with the database
last. Existing destinations require --overwrite. A filesystem interruption
between file replacements requires repeating restore; multi-file publication is
not atomic. Restore never executes private bindings or starts deployments.
Review the Git configuration and run plan before taking over. If the
restored config contains machine-specific executable paths, update them locally.
Keep independent encrypted recovery material if VaultContext itself depends on the infrastructure being recovered. Vault encrypts remote copies, not the local working database. Passphrases are entered privately by the user, never retrieved from configuration or supplied in command arguments.
devenv shell
uv sync --locked --extra test
uv run pytest -q
uv build
uv run python scripts/test-launcher.pyTests use synthetic providers and state. They cover interrupted creates/deletes,
ownership mismatches, immutable drift, private snapshot recovery, secret-safe
errors, SSH host trust, stop-first delivery and idempotent convergence. Cloud
mutation is never part of the default suite. Live verification results are in
docs/verification.md when available; they are distinct from mocked coverage.
Publish and test package changes before updating the Git commit in the skill launcher. Pin a full published commit, never a moving branch. The root symlink keeps one launcher payload; commit the new pin after testing a copied launcher outside this checkout. The launcher integration check uses synthetic missing-state fixtures and never accesses a cloud account.
--help always prints ordinary help text, including with --json. A JSON
usage error has command: null when argument parsing cannot identify a command.
Running ./pocketdeploy with no arguments prints the same help as --help
to stdout and exits 0, without loading configuration or contacting providers.
Unknown commands and a lone --json remain usage errors (exit 2).
Use pocketdeploy plan --verbose or pocketdeploy converge --verbose to see
individual OCI and SSH requests start, finish, and report elapsed time on
stderr. Long-running requests periodically report that they are still waiting.
Labels are authored by PocketDeploy; arguments, credentials and raw provider
responses are never printed. --json --verbose keeps one JSON result on stdout.
--verbose and --quiet conflict and exit 2 before deployment work begins.
For oci-auth: security_token, cloud workflows check the selected profile's
local token expiry before invoking OCI. An expired token fails with a safe
renewal instruction instead of waiting for a provider request. This local check
does not prove a token is otherwise valid; OCI may still reject revoked or
invalid credentials. init and Vault workflows do not require OCI credentials.
adopt-dns transfers one existing application A record to this deployment.
Configure Cloudflare and the managed application first. Disable and drain the
previous DNS manager, checkpoint its state, and obtain the exact zone ID, record
ID and current IPv4. The address must match this deployment's recorded owned
compute. Multiple A records or AAAA/CNAME conflicts are refused.
Use pocketdeploy dns-evidence --dns-host app.example.com --json to obtain
the allowlisted current record evidence without changing state or providers.
The --dns-settings-sha256 fingerprint is computed from the current record's allowlisted fields
type, name, content, proxied, ttl, and comment (missing comment is null):
SHA-256 of UTF-8 JSON with sorted keys and separators (',', ':'). Filter provider
responses inside the inspecting process; never print raw provider metadata.
pocketdeploy adopt-dns --dns-host app.example.com \
--dns-zone-id EXACT_ZONE_ID --dns-record-id EXACT_RECORD_ID \
--dns-ipv4 CURRENT_OWNED_IPV4 --dns-settings-sha256 EXACT_FINGERPRINT \
--previous-dns-manager-disabledThe command journals exact evidence before changing only the record's ownership
comment. It preserves record identity, address, proxy setting and TTL. Retry an
interrupted transfer with the original evidence; do not clear its journal or
manually edit SQLite. Ordinary convergence and deletion refuse an incomplete
transfer. After adoption, plan and converge apply the normal proxied-A policy.
Keep the previous manager disabled and save a new Vault checkpoint before
removing its obsolete local state.
Applications can depend on a pinned PocketDeploy package and reuse its compute
adapters, state journal, host keys and SSH transport. Host.run_python(connection, source, payload, timeout=1200) runs trusted Python as root over pinned SSH,
passing the JSON payload through stdin and parsing JSON stdout. Put credentials
in the payload, never in source text. Remote errors suppress stdout/stderr.
Host.ssh(connection, command=None) provides an interactive or command session.
These interfaces share the same transport as ONCE deployment operations.
Application controllers can use pocketdeploy.runtime.session(config, read_only=False, allow_create=False) to open matching state and its provider/host
under a local deployment lock. Read-only plans with allow_create=True leave a
missing state absent. recorded_operation(state, command, desired_hash) records
success, failure and interruption. initialize(config, state, host, root) prepares
local authority. delete_compute(config, state, cloud, host, operation) is for
compute-only applications: it enforces destruction protection, refuses service
resources and removes keys only after all recorded resources are gone. Callers
must stop their application and finish data copying before deletion.
pocketdeploy.dns.CloudflareDNS(config, state, hosts) manages explicit A-record
hostnames independently of ONCE. Use preflight(), plan(connection),
converge(connection, operation_id), plan_delete() and delete(operation_id)
under the deployment lock. It shares record ownership, pagination, pending-write
reconciliation and deletion checks with ONCE DNS. Proxying defaults to true with
Auto TTL; constructor arguments proxied and ttl can select another policy.
Without cloudflare-zone-id, zone discovery requires one unambiguous common
zone. COLORS_PAR_CLOUDFLARE_API_TOKEN maps to the CLI token only in its child
environment; the operator environment remains unchanged.
For migration to new owned compute, evidence(host) returns allowlisted record
identity and a routing-settings fingerprint. adopt(connection, operation_id, evidence) requires exact evidence and previous_dns_manager_disabled: true.
It preserves the source address, record ID, proxy and TTL while transferring
the ownership comment. A separate convergence updates routing to the target.
Keep the previous manager disabled, preserve interrupted-operation state and
coordinate application writers before switching production traffic.
Host.prepare_transport(connection) prepares pinned SSH arguments once on the
controller thread. Its immutable transport supports concurrent run_python,
tunnel(local_port, remote_port) as a loopback-only context manager, and
rsync_shell() for the rsync remote-shell option. It never reopens controller
state from worker threads. pocketdeploy.ssh_config.update(payload, owner=...)
maintains application-owned SSH aliases under one shared local lock. Owners must
keep their stable uppercase marker (Walter uses WALTER). Aliases live in
~/.ssh/config.d/00-<owner-lowercase>-<profile>.conf; the main SSH configuration
is never rewritten. Configure an unconditional include before SSH options that
could override the managed connection settings, for example:
Host *
Include config.d/*
Host *
# Your default SSH options follow here.Installation requires that the include loads the managed file with its pinned
host trust and explicit identity settings taking precedence. Conflicting aliases,
unsafe files and ambiguous include layouts are refused. Existing managed blocks
in the main configuration require deliberate removal before installing a drop-in;
the updater does not silently migrate them. Removal deletes only the matching
owned drop-in and leaves the main configuration and other deployments intact.
check_only validates installation without writing files. Copied launchers and
applications pinned to older PocketDeploy commits retain their previous behavior
until their dependency is deliberately updated.
Structured diagnostics are available in pocketdeploy.diagnostics:
build_diagnostic(operation, category, returncode=..., elapsed_seconds=...) and
validate_diagnostic(value, allowed_operations=...). Use fixed operation names,
never arguments or credentials. Treat remote JSON as untrusted: pass an explicit
operation allowlist and explicit task/user allowlists if accepting attribution.
DeployError(diagnostic=...) and Reporter.failure(diagnostic=...) preserve this
allowlisted evidence. The diagnostics module uses only the standard library and
can be injected into standalone remote programs before their source executes.
Host.run_python and prepared transport accept optional progress(stage, event)
and allowed_stages arguments. Remote programs may emit newline-delimited JSON
objects containing exactly stage and event (started, completed, failed)
on stderr. Only allowlisted events are delivered, on the calling thread; raw
stderr is discarded and buffers are bounded. output.remote_progress(stages)
provides a callback that maps these events to verbose timed spans and heartbeats.
Final JSON and process status remain authoritative; events never complete durable
state. This allows one remote lock to cover all provisioning stages.
Signing infrastructure has a separate lifetime from applications and compute. Configure one authority per OCI scaffold:
oci-region: eu-frankfurt-1
oci-kms:
name: identitycontext
tenancy-id: ocid1.tenancy.oc1.example
binding-prefix: APP_IDENTITYCONTEXTProvide the dedicated runtime identity's existing mailbox/alias privately as
COLORS_PAR_APP_IDENTITYCONTEXT_OCI_USER_EMAIL (the prefix follows
binding-prefix). OCI identity-domain tenancies require a unique email for user
creation. Convergence validates this binding before making cloud changes; plans
do not require it. The mailbox is not an application signing credential.
After init (or restoring the existing deployment checkpoint), use
pocketdeploy kms-plan, then pocketdeploy kms-converge. These commands do not
converge applications, rotate GitHub keys, or require application environment
bindings. The controller creates a standard shared DEFAULT vault, an
HSM-protected RSA3072 key, and a dedicated IAM user/group. Its policy grants only
KEY_SIGN and KEY_READ on the exact key; the user has API-key capability only.
The controller rejects unexpected group members, user group memberships, API
keys and policy changes. Tenancy-wide policies remain the operator's authority;
these checks cannot prove that no other policy grants additional access.
The operator needs OCI permissions to create these KMS and IAM resources.
kms-converge writes an API authentication key to .ssh/kms-runtime.pem and
shell exports to .ssh/kms-runtime.env, with 0600 files in a 0700 directory.
Neither file contains the HSM signing private key, which stays in OCI. Do not
print or commit these files. Source the generated exports in a trusted shell,
then transfer the app-specific bindings into the scaffold's ignored
.envrc.private using a private-file operation. The prefix above produces
COLORS_PAR_APP_IDENTITYCONTEXT_ bindings with these suffixes:
OCI_TENANCY_ID,OCI_USER_ID,OCI_FINGERPRINT,OCI_REGION,OCI_PRIVATE_KEYKMS_KEY_ID,KMS_KEY_VERSION_ID,KMS_CRYPTO_ENDPOINT,KMS_MANAGEMENT_ENDPOINT
The service uses OCI SHA_256_RSA_PKCS1_V1_5 for JWT RS256 and the configured key
version as its JWKS kid. It must select an explicit version and retain the old
public keys throughout token expiry and verification-cache overlap. Runtime
metadata and public key retrieval use the management endpoint; signing uses the
crypto endpoint. Credentials grant no key rotation, export, encryption or
deletion authority. Save a vault-save checkpoint after provisioning; it includes
both runtime credential files. Missing local API credentials require checkpoint
recovery and are never silently replaced.
Provisioning records intent before each mutation and records verified ownership
using deployment UUID/role tags. A lost response is recovered only by matching
those tags and the pending operation. Names never establish ownership. A missing
recorded resource or an uncertain create with no matching resource blocks retries.
A still-provisioning vault/key is left pending; retry after OCI reports readiness.
For an older pending user creation rejected for missing email, use
kms-resolve-user-create --kms-event-id EVENT_ID. It re-reads OCI Audit, requires
the exact pending operation tag and runtime user name with HTTP 400/email
rejection, and verifies no matching user exists before atomically settling that
intent. It never retries the create. Supply the dedicated email binding and run
kms-converge afterward. Network failures and uncorrelated audit events remain
blocked. Audit read permissions are required only for this recovery command.
Existing untagged infrastructure is not adopted by name. An ownership transfer
requires a separately reviewed procedure; restoring this deployment's checkpoint
recovers its already-owned resources.
pocketdeploy kms-rotate creates one new key version and retains every old
version. It does not update exported bindings or running applications. Publish
both public keys, then run kms-converge, deliberately select the new version,
and deploy the service. If the rotation response is lost, inspect the exact
current version and use kms-resolve-rotation --kms-version-id OCID; this verifies
its key relationship and completes the pending intent without creating another
version. Never retry an uncertain rotation by editing controller state.
Ordinary delete retains all KMS and IAM signing resources, their private
credentials and their state. No KMS deletion command is provided. Retirement
requires a separate explicit OCI procedure after consumers and retained tokens
are accounted for; preserve recovery metadata and use OCI's scheduled-deletion
and cancellation process. Restoring a checkpoint never rotates or recreates
signing material. OCI KMS outage fails signing closed; application fallback is
an independently authorized application feature.
The implementation is validated with synthetic provider responses and local credential-generation tests. A live disposable provisioning/signing/rotation check remains required before claiming provider verification.
Provider references: vault types, key permissions, signing algorithms.