docs: strengthen operations guidance

docs/operations/deployment-basics.md

@@ -20,6 +20,8 @@ flowchart LR
 
 This page owns that path. The linked operations pages cover process topology, probes, metrics, backups, and security policy in more depth.
 
+This page uses `forj` while preparing source and `./bin/<app>` after an artifact has been built. Production supervisors and release checks should execute the exact binary being deployed, not source-aware development commands.
+
 ## Before You Start
 
 Decide which Apps belong in the release and where they will run. Provision the production database, queue, cache, storage, and other external services selected by those Apps. The deployment platform must also own DNS, TLS termination, network policy, process supervision, and secret delivery.
@@ -35,7 +37,7 @@ forj build &&
   test -x ./bin/app
 ```
 
-Expected result: `bin/app` exists and is executable. The build refreshes generated Project files, runs Wire, prepares the API index, and compiles the default App.
+Expected result: `bin/app` exists and is executable. The build refreshes Framework-managed Project files, runs Wire, prepares the API index, and compiles the default App.
 
 ### Apps with Web UI
 
@@ -51,17 +53,31 @@ Expected result: `cmd/app/frontend/dist` contains current frontend output and `b
 
 ### Additional Apps
 
-Build each independently deployable App:
+Build each independently deployable App. If it has frontend source, build that App's frontend first using its path under `cmd/<app>/frontend/`:
 
 ```bash
-forj admin build &&
+npm --prefix cmd/admin/frontend ci &&
+  npm --prefix cmd/admin/frontend run build &&
+  forj admin build &&
   test -x ./bin/admin
 ```
 
-Expected result: `bin/admin` contains the staff-facing App selected by the command prefix. Repeat the build for every App included in the release.
+Expected result: `cmd/admin/frontend/dist` contains the current staff frontend and `bin/admin` contains the staff-facing App selected by the command prefix. Omit the npm steps when that App has no frontend source. Repeat the applicable frontend build, App build, and artifact check for every App included in the release.
 
 Keep the artifact immutable after it has been tested. If a release is transferred to another host or registry, verify its checksum or image digest before activation.
 
+### Hand Off the Artifact
+
+The build system should hand the deployment system one identified, immutable release. Record at least:
+
+- the binary or image digest;
+- every App binary included in the release;
+- the target operating system and architecture;
+- the source revision and build time; and
+- any non-secret defaults or overrides compiled with `forj build`.
+
+Frontend output is already embedded in an App binary after the frontend build and `forj build`; do not deploy an unrelated `dist` directory beside it. Promote the same tested bytes between environments. If configuration or frontend assets require a rebuild, assign the result a new release identity and repeat artifact checks.
+
 ## Keep Configuration Outside the Artifact
 
 Supply production configuration through the process environment or the secret and configuration mechanism provided by the deployment platform.
@@ -81,6 +97,8 @@ APP_DIAG_TOKEN=<value-from-your-secret-store>
 
 The rendered App determines the rest. Configure its selected database, queue, cache, storage, event, mail, and observability drivers with production values. Do not allow a missing production dependency to silently fall back to a process-local driver.
 
+`forj build --env-defaults` and `--env-overrides` are explicit exceptions: they pin non-secret values into the binary. Defaults remain replaceable by deployment configuration; overrides do not. Use defaults only for artifact-level fallbacks and overrides only when every deployment of that artifact must use the same value. A rotated credential, environment endpoint, port, replica-specific identity, or retention setting belongs to the deployment system instead. See [Compiled Environment Values](/reference/configuration#compiled-environment-values) for exact precedence.
+
 Bind to `127.0.0.1` when a reverse proxy on the same host owns public traffic. Bind to `0.0.0.0` only when a container network, firewall, or host network policy controls access.
 
 If the App intentionally uses SQLite, local storage, uploads, or another writable filesystem path, place that data outside the versioned release directory and configure an absolute path. Replacing an immutable release must not replace or orphan durable application data.
@@ -110,6 +128,16 @@ Changing the command does not change application behavior or make in-memory driv
 
 Run one scheduler process unless the schedules use deliberate cross-process locking. See [Runtime Processes](/operations/runtime-processes) before introducing a split topology.
 
+A concrete split deployment might use the same immutable artifact in these supervised process groups:
+
+| Process group | Replicas | Traffic or work handoff | Scaling signal |
+| --- | ---: | --- | --- |
+| `./bin/app api` | Two or more | The platform sends HTTP traffic only to ready instances. | Request load, latency, and resource use. |
+| `./bin/app worker --queue emails` | One or more | A shared queue backend hands jobs to workers. | Queue depth, job age, failures, and resource use. |
+| `./bin/app scheduler` | One | The supervisor maintains singleton ownership unless schedules use a shared lock. | Availability and due-run outcomes, not HTTP load. |
+
+This is a topology example, not a required replica count. Each group receives its own environment, resource limits, probe configuration, and restart policy. If an App has another binary such as `bin/admin`, model its HTTP, worker, and scheduler roles independently rather than assuming the default App process owns them.
+
 ## Give the Process to a Supervisor
 
 The deployment platform should:
@@ -125,7 +153,21 @@ The deployment platform should:
 
 This contract applies equally to systemd, a container runtime, Kubernetes, Nomad, or another supervisor. The exact service unit or workload manifest belongs to that platform.
 
-Set the supervisor's stop grace period above the longest effective App shutdown path, with additional margin for the supervisor itself. In combined mode, runtime shutdown happens concurrently before the outer App lifecycle finishes. Test shutdown with real in-flight jobs instead of relying only on arithmetic.
+### Budget Timeouts as a System
+
+Timeouts protect different boundaries and should be planned together:
+
+| Boundary | Example control | What it bounds |
+| --- | --- | --- |
+| One unit of work | A job timeout or `SCHEDULER_COMMAND_TIMEOUT` | The handler or scheduled command execution. |
+| Runtime cleanup | `QUEUE_SHUTDOWN_TIMEOUT` | Queue drain and backend cleanup inside App shutdown. |
+| App shutdown | `APP_SHUTDOWN_TIMEOUT` | The HTTP or scheduler graceful-stop path and outer App lifecycle. |
+| Readiness request | Probe or `health --timeout-ms` | One operator or platform request, including sequential resource checks. |
+| Process supervision | Platform stop grace period | The complete interval before the platform may force termination. |
+
+The App resolves this policy once at startup. A queue or scheduler subprocess value larger than `APP_SHUTDOWN_TIMEOUT` is capped to the App budget and emits one structured warning with the configured and effective values. Run `./bin/app about` to inspect the effective Runtime settings before changing supervisor limits.
+
+Do not add every configured duration and assume that sum is the required supervisor value. Some shutdown work is concurrent, some limits are nested, and a handler that ignores cancellation can outlive its intended budget. Set the supervisor's stop grace period above the longest observed graceful-stop path with margin for traffic removal and supervisor overhead. In combined mode, runtime shutdown happens concurrently before the outer App lifecycle finishes. Test `SIGTERM` with real in-flight requests, jobs, and scheduled commands; make interrupted work safe to retry.
 
 The long-running command must be the deployed binary:
 
@@ -149,8 +191,30 @@ Expected result: the migration command exits successfully before processes from
 
 Run the command once per migration-owning App, not once per HTTP replica. During a rolling deployment, prefer additive schema changes that work with both the old and new binaries.
 
+For an additional migration-owning App, run that App's staged binary independently:
+
+```bash
+/srv/example/releases/2026-07-28/bin/admin migrate
+```
+
+Expected result: only the `admin` App's migration streams run. Repeat this once for each migration-owning App in the release; do not infer that the default App command migrated additional Apps.
+
 Use stable paths for backup output rather than writing backup sets inside a versioned release directory. [Backup and Restore](/operations/backups) covers discovery, verification, retention, and restore safeguards.
 
+## Hand Off Observability and Retention
+
+The App emits signals; the deployment platform and operators own their transport, access, and retention. Before activation, assign each signal to an operational destination:
+
+| Signal or data | App/process responsibility | Deployment responsibility |
+| --- | --- | --- |
+| Logs | Write structured runtime output to standard output and standard error. | Collect, index, redact, retain, and alert on it. |
+| Metrics | Expose the endpoint appropriate to combined or split topology. | Scrape it with bounded labels, retain time series, and define alerts. |
+| Health and readiness | Report process liveness and required dependency state. | Route probes correctly and remove unready HTTP instances from traffic. |
+| Inspects and Lighthouse | Capture and present bounded recent execution detail when enabled. | Restrict operator access and choose capture, sampling, and recent-window limits. |
+| Backups and durable data | Use the configured stable paths and backends. | Schedule backups, apply retention, verify restore, and keep data outside release directories. |
+
+Preserve the app name, runtime role, release identity, and instance identity in the surrounding platform metadata so an alert can be traced to the exact process and artifact. Set `APP_VERSION` and `APP_REVISION` while building framework-managed metrics discovery, and apply equivalent bounded labels in an external production scraper. Lighthouse's recent Inspect window is not a substitute for retained logs, metrics, or backups.
+
 ## Activate the Release
 
 A useful filesystem layout separates immutable releases from mutable state:
@@ -246,12 +310,14 @@ Queue payloads are another compatibility boundary. A previous worker binary must
 - Build every App for the deployment target.
 - Build frontend assets before `forj build` when the App has Web UI.
 - Store production configuration and secrets outside the artifact.
+- Record any non-secret defaults or overrides intentionally compiled into it.
 - Use production drivers for state shared across processes or hosts.
 - Keep writable data and backup sets outside immutable release directories.
 - Run the staged release's migrations once before it receives traffic.
 - Supervise each required process with the deployed binary.
 - Keep the scheduler singleton unless locking makes overlap safe.
 - Set and test graceful shutdown budgets.
+- Assign logs, metrics, Inspects, and backups explicit access and retention owners.
 - Verify liveness, readiness, metrics, and one application workflow.
 - Keep the previous artifact available for binary rollback.
 - Treat database migrations and queued payloads as separate rollback contracts.

docs/operations/http-server.md

@@ -77,6 +77,20 @@ For a staff operations App named `admin`, use its binary:
 
 Expected result: a complete, human-readable table of the staff-facing methods, paths, and handlers registered by `admin`. The command constructs the App route surface but does not start the HTTP listener.
 
+A small route table looks like this (terminal output uses color when supported):
+
+```text
++-------------------+---------+-----------------------+------------+
+| API Routes › (1)
++-------------------+---------+-----------------------+------------+
+| Path              | Methods | Handler               | Middleware |
++-------------------+---------+-----------------------+------------+
+| /api/v1/users/:id | GET     | users.Controller.Show |            |
++-------------------+---------+-----------------------+------------+
+```
+
+Long middleware names may be replaced by short codes with a `Middleware Legend` above the route title. The columns remain `Path`, `Methods`, `Handler`, and `Middleware`.
+
 ## Health, Readiness, and Verification
 
 Use liveness to answer whether the process is responding and readiness to decide whether it should receive traffic:

docs/operations/index.md

@@ -19,6 +19,7 @@ Use these guides when you need to run, split, observe, deploy, or recover an App
 | Configure probes | [Health and Readiness](/operations/health-readiness) |
 | Investigate runtime behavior | [Logging](/operations/logging), [Metrics](/operations/metrics), and [Inspects](/operations/inspects) |
 | Use the operator interface | [Lighthouse](/operations/lighthouse) |
+| Compare HTTP and infrastructure performance safely | [Performance Benchmarks](/operations/performance-benchmarks) |
 | Protect and recover durable state | [Backup and Restore](/operations/backups) |
 | Build, roll out, verify, and roll back a release | [Deploy an App](/operations/deployment-basics) |
 

docs/operations/logging.md

@@ -81,6 +81,22 @@ Each request event retains named fields for URI, method, status, latency, and cl
 
 Console output uses a compact value-oriented line with status-aware color. JSON output and registered log sinks retain the structured field names, so machine processing does not depend on console formatting.
 
+An App logger can route one structured entry to several destinations. `AddSink` preserves the simple synchronous callback API. Use `AddSinkWithOptions` when a destination needs error isolation or a bounded asynchronous queue:
+
+<!-- go-example: illustrative-fragment -->
+```go
+if err := appLogger.AddSinkWithOptions(auditSink, logger.SinkOptions{
+	Name:          "audit",
+	Async:         true,
+	QueueCapacity: 256,
+	Overflow:      logger.SinkOverflowDropOldest,
+}); err != nil {
+	return err
+}
+```
+
+Registrations append; they do not replace process output or an earlier sink. Synchronous sinks run in the logging path, so keep them fast. Asynchronous sinks preserve accepted entries in FIFO order and make overload behavior explicit with `block`, `drop-newest`, or `drop-oldest`. Flush managed sinks from an App shutdown hook with `appLogger.FlushSinks(ctx)` so accepted entries get the remaining shutdown budget.
+
 Disable access logs for a runtime where request volume would hide higher-signal events, then rely on metrics and inspects for the intended visibility.
 
 Keep access logs enabled during a new deployment until the request path, status, and latency are understood. If you disable them for steady-state volume, retain an explicit route-level metric and a safe Inspect sampling policy; otherwise a 5xx increase has no request-level path back to an operator.
@@ -97,7 +113,16 @@ APP_LOG_TIME
 
 Use structured fields that answer an operational question: App identity, Runtime source, route pattern, queue or schedule name, status, and latency. The request context can carry the `trace_id` correlation field used by Inspects, but the product surface is still called an Inspect.
 
-Never log authorization headers, cookies, credentials, raw queue payloads, or unredacted request bodies by default. Local HTTP error-response capture is intentionally local-environment behavior; do not rely on it as a production payload-dump mechanism.
+Never log authorization headers, cookies, credentials, raw queue payloads, or unredacted request bodies by default. Before deduplication, process output, or sink delivery, the App logger redacts common secret-bearing field names and high-confidence message forms such as bearer tokens and `password=...` assignments.
+
+Extend that mandatory policy for application-specific data:
+
+```dotenv
+APP_LOG_REDACT_KEYS=session_id,customer_reference
+APP_LOG_REDACT_MESSAGE_PATTERNS=["client_secret=[^ ]+","session=[^ ]+"]
+```
+
+The first value is a comma-separated list of additional case-insensitive field names. The second is a JSON array of regular expressions; invalid JSON or expressions fail during logger construction instead of silently weakening the policy. These controls supplement the framework defaults rather than disabling them. Redaction is a safety net, not permission to log whole payloads. Local HTTP error-response capture remains intentionally local-environment behavior.
 
 ## Failure Modes
 

docs/operations/metrics.md

@@ -113,6 +113,25 @@ environment=local
 
 This label set identifies a background staff-operations job from `admin`. Use `source` for logical runtime attribution and `process` for scrape topology.
 
+## Deployment Markers
+
+Set release identity in the Project-owned environment used to generate the deployment artifact:
+
+```dotenv
+APP_VERSION=v1.8.0
+APP_REVISION=8d5c20f
+```
+
+Then build the artifact:
+
+```bash
+forj build
+```
+
+When present, `APP_VERSION` becomes the bounded `release` target label and `APP_REVISION` becomes `revision`. Blank values are omitted. Keep both values low-cardinality: use one release version and one immutable commit or artifact revision, not a request ID or build timestamp. This lets a dashboard annotation or deployment comparison separate a real regression from ordinary traffic movement.
+
+Generation intentionally reads `.env.example` and `.env`, not unrelated ambient shell variables, so the discovery file and the artifact are derived from one reviewable Project snapshot. These labels are written into framework-managed local discovery files during generation. In an externally managed production scraper, apply the same release and revision labels in that platform's service-discovery configuration.
+
 Avoid user IDs, emails, raw URLs, raw SQL, cache keys, filenames, request IDs, and arbitrary error strings.
 
 ## Proving Path