Skip to content

Docs: Add markdown content filter example for REST API - #2201

Open
per1970 wants to merge 1 commit into
unclecode:developfrom
per1970:docs/content-filter-rest-api
Open

Docs: Add markdown content filter example for REST API#2201
per1970 wants to merge 1 commit into
unclecode:developfrom
per1970:docs/content-filter-rest-api

Conversation

@per1970

@per1970 per1970 commented Aug 25, 2026

Copy link
Copy Markdown

Summary

Adds a Markdown with Content Filter (Pruning / BM25) example to the Docker server README REST API section (deploy/docker/README.md), between the Simple Crawl and Streaming examples.

What it covers:

  • The working payload shape for content_filter, nested inside markdown_generator with the {"type": "ClassName", "params": {...}} serialization
  • The silent-drop pitfall: a top-level content_filter key in crawler_config is ignored by the untrusted config loader (HTTP 200, empty fit_markdown) — this cost real debugging time, and the failure mode is invisible unless you check fit_markdown
  • Response fields: fit_markdown / fit_html vs the always-present raw_markdown
  • BM25 variant (user_query) and a pointer to docs/examples/docker/demo_docker_api.py (Demo 2a–2c) for more configurations

Verification

Payload tested live against the 0.9.2 Docker server:

  • https://httpbin.org/html: raw 3598 → fit 3598 chars (pruning, threshold 0.45 fixed)
  • https://www.bbc.com/news: raw ~22K → fit ~9.9K chars (nav/cookie boilerplate stripped)

Transparency: Prepared with assistance from my Hermes AI agent (Muninn), based on live verification against the 0.9.2 server. The example payload in the PR is the exact payload that was tested. Happy to adjust wording or add the same note to demo_docker_api.py if preferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant