Add compose v5 compatibility, legacy-field warnings, UI error fixes

[SohamKukreti]

Summary

Companion to the self-hosting docs update; all changes verified against a built 0.9.2 image.

docker-compose.yml:

• Express the PID cap as deploy.resources.limits.pids instead of pids_limit: Compose v5 normalizes the two together and rejected the file ("can't set distinct values on 'pids_limit' and 'deploy.resources.limits.pids'"), which also broke compose down. Identical behavior on Compose v2.x.

entrypoint.sh:

• When no CRAWL4AI_API_TOKEN is set, explain that the loopback bind is the container's own (published ports reset connections) and how to fix it. The old one-line notice read as "available at 127.0.0.1:11235" and sent users debugging the wrong thing.

server.py / schemas.py:

• /screenshot and /pdf: output_path (removed in 0.9.0) was silently dropped; requests still succeed but now return a warning that no file was written and results are fetched via GET /artifacts/{artifact_id}.
• Legacy hooks.code (removed in 0.9.0) parsed as an empty hook list and reported {"status": "success", "attached": []} with hooks enabled. The field is now captured (never executed) and the response reports status "ignored" plus a warning; /crawl/stream sends X-Hooks-Warning. With hooks disabled, any hooks payload still returns 403 (unchanged).
• Add "/" to the auth gate's exact-match public paths so the existing root -> /playground redirect is reachable; previously the middleware returned a bare 401 before the route ran. /monitor and all data routes remain gated.

static/playground/index.html:

• The streaming branch never checked response.ok and showed a green "Success" banner for 401 bodies; it now errors before reading the stream. All branches surface the server's detail message instead of a generic "Request failed", and 401s hint at the token bar (distinguishing "no token set" from "token rejected").

tests:

• Add deploy/docker/tests/test_legacy_compat.py covering the root redirect, output_path warning, legacy-hooks handling, and the compose PID-cap placement (13 behavioral tests, TestClient-based).

Fixes #2091

List of files changed and why

• deploy/docker/entrypoint.sh
• deploy/docker/schemas.py
• docker/server.py
• docker/static/playground/index.html
• deploy/docker/tests/test_legacy_compat.py
• docker-compose.yml

How Has This Been Tested?

Ran old tests and also created a new test file to test the changes. Manually verified as well.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[SetagGnaw]

Nice work on the legacy-compat warnings. One interaction seems worth a look.

The pre-existing gate in /crawl and /crawl/stream fires on the mere presence of a hooks object, without looking inside it:

if crawl_request.hooks and not HOOKS_ENABLED:
    raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")

crawl_request.hooks parses into a truthy HookConfig whether the payload carries declarative hooks, legacy code, or both, so all three shapes get the same 403 with the same advice. Before this PR the gate could not do better: pydantic dropped the unknown code field, so a code-only payload was indistinguishable from an empty one by the time it reached the gate. Your schema change is what makes the distinction expressible at all (hooks.code and hooks.hooks both survive parsing now), but the PR only uses it after the crawl, in the warning block.

Since CRAWL4AI_HOOKS_ENABLED defaults to false, that ordering means a legacy 0.8.x caller on a stock deployment never reaches the new hooks.status: "ignored" + warning path. They hit the 403 first, and its remedy is misleading for them twice over: enabling the flag won't make inline code run (it was removed entirely in 0.9.0), and it nudges operators into enabling the declarative-hooks surface just to appease code that can never execute. I see test_any_hooks_payload_403_when_disabled pins the 403-when-disabled behavior, so I take the gate itself as deliberate. The ask is only that the gate use the distinction your schema change just made possible.

Smallest version, no status codes change: keep firing on presence, but branch the error detail on what is actually inside, e.g.:

if crawl_request.hooks and not HOOKS_ENABLED:
    if crawl_request.hooks.code:
        raise HTTPException(
            403,
            "Inline hook code (hooks.code) was removed in 0.9.0 and cannot be "
            "enabled; it was not executed. Use declarative hook actions instead "
            "(GET /hooks/info), which are additionally disabled on this server "
            "(CRAWL4AI_HOOKS_ENABLED).",
        )
    raise HTTPException(403, "Hooks are disabled. Set CRAWL4AI_HOOKS_ENABLED=true to enable.")

Probably worth hoisting into a small helper, since the gate appears in both /crawl and /crawl/stream.

Related question: did you consider rejecting hooks.code payloads with a 400 in both cases (hooks enabled and hooks disabled), with a clear "removed in 0.9.0" error, instead of 200 + warning? A caller that never inspects the warning field reads the 200 as "my hook ran", which is the same silent-failure shape this PR is eliminating, just moved from the field level to the status level. Curious whether backward compatibility drove the accept-and-warn choice.

[SohamKukreti]

@SetagGnaw Good catch on the ordering. On a stock deployment (CRAWL4AI_HOOKS_ENABLED defaults to false) a legacy hooks.code caller hits the presence-based 403 and never reaches the "ignored" + warning path, and the 403's remedy is actively misleading for them: enabling the flag won't ever make inline code run.

I've adopted your suggested shape: keep firing on presence (the 403-when-disabled contract stays pinned by test_any_hooks_payload_403_when_disabled), branch only the detail on hooks.code, and hoist it into a small helper since the gate appears in both /crawl and /crawl/stream plus a test pinning the code-only 403 message so the two error texts don't drift back together.

On your 400 vs. 200 + warning question: I considered rejecting hooks.code payload with a 400, and backward compatibility did drive it, but with a specific shape in mind:

1. A 400 now would be a second breaking change for the same clients. Since 0.9.0 (two releases), a hooks.code payload with hooks enabled has returned 200 with complete, usable crawl results, the inline code never influenced the crawl in 0.9.x, so the results aren't degraded by its absence, only the hook side-effects are missing. Pipelines that adapted by simply not caring about hook effects (or that only ever used hooks for nice-to-haves like scroll/settle) currently work; a 400 would break them to protect callers who depend on hook effects, and those callers already see broken output, so they have a detection signal today.
2. Rejecting discards work the server can fully perform. The request minus code is a completely valid 0.9 crawl. That's also why mixed payloads argue against 400: declarative hooks + legacy code would be rejected wholesale even though everything except the dead field executes correctly.
3. It's consistent with the output_path decision in the same PR: serve the thing we can serve (crawl results / artifact), attach an explicit machine-readable signal about the part we can't (warning, status: "ignored", X-Hooks-Warning on the stream). One philosophy across both legacy fields.

Thanks for the review!

[ntohidi]

Nice work Soham — the #2091 diagnosis was thorough and these fixes are well-targeted. A few things to sort out before merge.

Blocking

1. The compose change silently disabled an existing security test.
deploy/docker/tests/test_security_container_posture.py:106 is assert "pids_limit" in compose against raw file text. You removed the key, but your new comment contains the words "not as pids_limit" — so the test still passes on a comment and no longer checks anything. Rewrite it to parse the YAML and assert deploy.resources.limits.pids == 512, or delete it in favour of your TestComposeFile test which already does this properly.

Also, compose config proves the key parses, not that it's applied. Please confirm once with docker inspect --format '{{.HostConfig.PidsLimit}}' <container> on a running stack — if that comes back 0, the PID cap is gone.

2. Issue item 2 is unaddressed, and it undercuts your own item-3 fix.
docker-compose.yml:7-8 still has env_file: - .llm.env with no required: false, and the environment: block (lines 9-19) is still commented out. A fresh clone still fails with "env file .llm.env not found", and the new entrypoint message tells users to set CRAWL4AI_API_TOKEN — which compose has no way to forward. Please add required: false, a CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} passthrough, and a commented token line in .llm.env.example.

While you're in that file: version: '3.8' is still line 1 (the issue flagged it — warns on every compose command).

3. pyyaml is missing from deploy/docker/tests/requirements.txt.
The new test file imports yaml; it only passes today via a transitive dep.

4. HookConfig.code: Optional[Dict[str, str]] is too strict.
MIGRATION.md only says "Python strings" — the exact 0.8.x shape was never pinned. A legacy caller whose payload doesn't match that type now gets a 422 on a field we document as ignored, which is worse than the silent drop it replaces. Use Optional[Any] so everything parses and reaches the removal message.

Should fix

output_path detection. (await request.json()).get("output_path") works, but it's duplicated across two handlers, invisible in OpenAPI, and sits inside the broad except Exception — so a surprise there becomes a 500 instead of a warning. It also only fires on the success path, so a legacy caller whose crawl fails never learns the field was dropped. Declaring it as a deprecated schema field on ScreenshotRequest/PDFRequest solves all three and matches what you did for hooks.code.

Hooks response shape. results.setdefault("hooks", {"attached": []}) — if the handler returns no hooks key and the request also carried declarative specs, the client gets a hooks object with no status. Seed it as {"status": "ignored", "attached": []}.

/monitor test. test_monitor_and_data_routes_stay_gated asserts /monitor → 401. I know the intent is proving the / change didn't widen the gate, but as written, adding the /monitor → /dashboard redirect the issue asks for would break this test. Assert "401 without a token" rather than "no redirect exists".

Playground. Two follow-ons: FastAPI 422s return detail as an array, so it renders as [object Object] — coerce non-strings. And only the streaming branch got the .catch(() => ({})) guard for non-JSON error bodies; the LLM and regular branches still call await response.json() bare, so a proxy HTML 502 throws a parse error there.

Entrypoint message. The bind guard at entrypoint.sh:27 accepts CRAWL4AI_API_TOKEN or CRAWL4AI_JWT_ENABLED=true, but the remedy line names only the token. Mention both.

Needs a call from @unclecode

Issue item 4 says "reject (or warn on)" output_path, so your warning satisfies it. Item 5 asks for an explicit 400 in both states for hooks.code, and this returns 200-with-warning when hooks are enabled. A warning key in a 200 body is still something an upgrading script won't read. I'd suggest warn this release, reject next, with the window written into MIGRATION.md — but let's decide deliberately.

Deferred / nits

• Issue item 3's healthcheck question (should the container report healthy while the published port is dead?) is untouched. Fine to defer — just note it in the PR body so it isn't lost.
• X-Hooks-Warning is header-only, and /crawl with stream: true diverts into stream_process before the body-level warning, so that path gets the header and nothing else. Worth documenting at minimum.
• Whitespace-only churn in server.py (~lines 892, 900, 966) adds diff noise.

If you'd rather keep this PR tight, narrow the "Fixes #2091" claim and open a follow-up for the rest — that works for me too.

[SohamKukreti]

@ntohidi thanks for the thorough review. Everything is addressed in the follow-up commit except item 2, which I held back deliberately; reasoning below.

Blocking 1 — done, both halves. test_pids_limit now parses the YAML (yaml.safe_load(compose)["x-base-config"]) and asserts deploy.resources.limits.pids == 512 plus pids_limit absent; my duplicate TestComposeFile is deleted so the check has one home, in the security suite. Runtime confirmation as requested: on a docker compose up -d stack (Compose v5.1.0), docker inspect --format '{{.HostConfig.PidsLimit}}' returns 512 — the cap is applied, not just parsed.

Blocking 2 — not addressed, on purpose; I'd like your read before touching it. Both one-liners have failure modes worse than the current annoyance:

• required: false needs the long-form env_file syntax, which only exists since Compose v2.24 (Jan 2024). Older Compose rejects the entire file — the same "nothing builds or starts" class of breakage item 1 of the issue was fixing for v5. It trades a clear error on fresh clones for a hard failure on older installs.
• CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-} is the dangerous one: environment: overrides env_file, and the :- default expands to empty string in any shell without the export (new terminal, cron, CI). That empty value clobbers a valid token in .llm.env → server silently falls back to loopback → the exact connection-reset failure this PR exists to explain. It breaks users whose setup currently works, silently. (The file's own comment on the commented-out block — "will overwrite .llm.env" — is warning about this.)

Given those trade-offs, I'd treat item 2 as its own change with its own decision (e.g. whether a Compose ≥2.24 floor is acceptable) rather than fold it in here. If you're comfortable accepting the trade-offs, I can pick it up in a follow-up PR.

Blocking 3 — done. pyyaml>=6.0 added; verified the file resolves standalone.

Blocking 4 — done. code: Optional[Any], plus a parametrized test that string / nested-dict / list shapes all parse and reach the removal message instead of 422ing.

output_path — done as suggested, with one heads-up: declaring the field required relaxing test_security_2026_04.py, which structurally asserted output_path absent from model_fields (that tripwire is why I'd gone the raw-body route). Those assertions now pin "deprecated, default None" instead — and to keep a behavioral tripwire, there's a new test that POSTs output_path into a tmp dir and asserts no file is created; I mutation-tested it (temporarily reintroduced the write — both parametrizations fail). Field shows as deprecated in OpenAPI; handlers read it via model_dump before the try (attribute access on a deprecated field emits a DeprecationWarning per request). One conscious trade: typed Optional[str], so a non-string value 422s — unlike hooks.code, the 0.8 shape here was definitionally a path string, and str keeps the OpenAPI honest.

Hooks seed, /monitor test, playground — all done as suggested. The playground error paths are covered by extracted-function assertions (422 array → body.urls: Field required-style text; non-JSON body → HTTP 502 fallback).

Entrypoint message — adjusted, with a finding you'll want to know: CRAWL4AI_JWT_ENABLED is read only by the bind guard — nothing maps it to security.jwt_enabled in config. Tested live: the env var alone gives a publicly-bound server whose only credential is the ephemeral token in the logs, with /token refusing issuance; config jwt_enabled: true without SECRET_KEY fail-fasts the worker (by design). So naming the flag as a bare remedy would mislead. The message now leads with the token and adds: "If you enabled security.jwt_enabled in a custom config.yml, set CRAWL4AI_JWT_ENABLED=true instead." The full recipe (config flag + SECRET_KEY + api_token) is verified working end-to-end.

[ntohidi]

Thanks for this — I went through all seven points in #2091 and verified each one against develop. They all reproduce, and the fixes here look right. Two things before this can go in.

1. Please rebase — there's one conflict

develop has moved. deploy/docker/server.py now conflicts, in the crawl() handler.

Cause: commit 130a382 ("fix(docker): preserve failed crawl results") removed these three lines from develop:

    # check if all of the results are not successful
    if all(not result["success"] for result in results["results"]):
        raise HTTPException(500, f"Crawl request failed: {results['results'][0]['error_message']}")

Your branch predates that commit, so it still carries them, and the new legacy-hooks block lands right next to them.

Resolution: keep your hooks block, drop those three lines. That's the whole conflict — nothing else in the PR clashes.

2. Point 2 of the issue isn't covered yet

The compose change here only moves the PID cap. Point 2 of #2091 is still open in docker-compose.yml:

• env_file: .llm.env has no required: false, so a fresh clone fails with "env file .llm.env not found".
• The environment: block is fully commented out, so export CRAWL4AI_API_TOKEN=... never reaches the container — which is exactly the loopback-only failure point 3 is about. The two fixes belong together.

Could you add both in the same push?

  env_file:
    - path: .llm.env
      required: false
  environment:
    - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:-}

While you're in that file: version: '3.8' on line 1 is obsolete and warns on every compose command. The issue mentions dropping it — worth doing here too.

Not blocking

Point 7's /monitor → /dashboard redirect is still missing. Your test comment already anticipates it, so I'm fine leaving it for a follow-up unless you want to fold it in.

Everything else — the Compose v5 PID fix, the entrypoint warning, the output_path and hooks.code warnings, the playground response.ok fix, and / in public_paths — all check out, and the new test_legacy_compat.py coverage is appreciated. CI is green on both checks.

[SohamKukreti]

Done, both the points have been resolved.
Rebase - done; resolved the server.py conflict exactly as you described (kept the hooks block, dropped the three lines).

Point 2 - added your snippet, plus required: false and dropped version: '3.8'.

One thing I hit while testing: environment: always overrides env_file:, even when the shell var is unset (it interpolates to an empty string). So with the passthrough in place, a token set in .llm.env never reaches the server — verified with docker inspect on Compose v2.20.3, v2.24.7, v2.32.4, and v5.1.0, all identical. Since the docs from #2091's update pointed people at .llm.env for the token (unreleased, so no users affected), I realigned them in the same push: token comes from the shell export or a project-root .env (compose reads it for interpolation; export wins), .llm.env stays for LLM keys. Also noted the v2.24+ floor the env_file long syntax needs.

Tested live against an image from this branch: export → reachable with auth working, .env-only → works, no token → fail-closed with the updated hint, fresh clone without .llm.env → starts fine, LLM keys still pass through, plain docker run flows unchanged. Full suite green locally.

Left the /monitor → /dashboard redirect for a follow-up as you suggested — the legacy-compat test already accepts the redirect.