Auto-update doc build config

.tekton/doc-build.yaml

@@ -8,7 +8,8 @@ metadata:
     pipelinesascode.tekton.dev/max-keep-runs: "10"
     pipelinesascode.tekton.dev/on-cel-expression: |-
       (
-        event == "push" && (
+        event == "push" &&
+        !last_commit_title.contains("ci skip") && (
           source_branch.matches("^(main|master|release-.*)$") ||
           target_branch.matches("^(main|master|release-.*)$")
         )

docs/en/build-custom-kb.mdx

@@ -99,7 +99,7 @@ Field reference (full list in `builder/README.md`):
 | `sub_dirs` | Optional — only ingest these subdirectories of the repo. |
 | `split_type` | `title`, `chapter`, `chunk`, or any combination joined by `,`. The bundled ACP KB uses `title,chunk`. |
 | `doc_type` | `md`, `mdx`, or `md,mdx`. |
-| `kb_version` (top-level) | Stamped on every chunk under `metadata.version`; used to sanity-check that all chunks in a dump came from the same prepare run. |
+| `kb_version` | Stamped on every chunk under `metadata.version`; used to sanity-check that all chunks in a dump came from the same prepare run. Accepted as a top-level key in the JSON manifest *and* as a `--kb-version` CLI flag. |
 
 ## Step 2 — Set the builder environment
 
@@ -230,7 +230,10 @@ upgrade rule parser uses the file name as the collection's internal name when re
 #### B2. Ship the dump into the plugin image
 
 The bundled dumps live at `/workspace-smart-doc/dumps/` inside the smart-doc container, baked into the
-`mlops/smart-doc` image at build time (see `dumps/` in the smart-doc repository). To add a custom dump:
+`mlops/smart-doc` image at build time. The `dumps/` directory in the smart-doc repository tracks the
+per-version source dumps (e.g. `docvec_gte_acp_4_{1,2,3}_20260508.dump`); the cs2000 / cs3000 dumps that
+the install-form **Built-in KnowledgeBase File** selector points to are produced separately and added to the
+image during the build, so they will not be visible on a fresh checkout. To add a custom dump:
 
 1. Drop your `<your-collection-name>.dump` into `dumps/` in your smart-doc fork.
 2. Rebuild the smart-doc container image and push it to your registry.
@@ -278,24 +281,28 @@ If neither rebuilding the image nor reaching production from the workstation is
 restored by hand into the running PG and the chart told to query it directly:
 
 ```bash
+# Look up the postgres pod name (the bundled pgvector is a Deployment, so the pod
+# has a hashed suffix — there is no -0 ordinal to assume).
+POD=$(kubectl -n cpaas-system get pod -l app=postgre-vec -o jsonpath='{.items[0].metadata.name}')
+
 # Copy the dump into the postgres pod
 kubectl -n cpaas-system cp <your-collection-name>.dump \
-  postgre-vec-0:/tmp/<your-collection-name>.dump
+  "${POD}:/tmp/<your-collection-name>.dump"
 
 # Restore into docvec_sys_kb
-kubectl -n cpaas-system exec -it postgre-vec-0 -- bash -lc \
+kubectl -n cpaas-system exec -it "${POD}" -- bash -lc \
   "pg_restore -U postgres -d docvec_sys_kb /tmp/<your-collection-name>.dump"
-
-# Update the running collection name
-kubectl -n cpaas-system edit configmap smart-doc-config
-# Change PG_COLLECTION_NAME to <your-collection-name>
-
-# Roll the deployment to pick up the new collection
-kubectl -n cpaas-system rollout restart deployment smart-doc
 ```
 
-This path skips the chart-level idempotency stamp; if you re-roll the chart it will not re-apply your custom
-data. Switch to Mode B as soon as the constraint that forced this path is gone.
+> **IMPORTANT:** Do **not** edit the rendered `smart-doc-config` ConfigMap directly to point at the new
+> collection. The chart re-renders the ConfigMap from `pgconnect.pgCollectionName` on every Helm operation,
+> and the init container re-applies any rule in `acpKbUpgradeRules` on every pod restart — both will
+> overwrite a manual edit. Instead, push the new collection name through the chart by overriding
+> `pgconnect.pgCollectionName: <your-collection-name>` in your values, then re-roll. The init container will
+> see the existing collection in `docvec_sys_kb` and leave it alone (no rule fires).
+
+This path skips the dump-on-disk distribution and the multi-replica advisory lock. Switch to Mode B as soon
+as the constraint that forced this path is gone.
 
 ## Re-roll cadence
 
@@ -313,9 +320,10 @@ same version is a no-op even if the dump file on disk has been replaced.
 - The custom KB occupies the same `docvec_sys_kb` database as any bundled ACP dump. **Mode B replaces the
   bundled dump entirely.** If you want to keep ACP product knowledge alongside your internal docs, ingest both
   the ACP repos and your internal repos in the same `prepare` run so the resulting dump contains both.
-- The user-facing **BYO Knowledge Tool** (introduced in v1.3.1) writes to `docvec_user_kb` instead and is
-  unaffected by this workflow. Use BYO Knowledge for end-user document uploads, and use this guide for
-  admin-curated corpora baked into the deployment.
+- The user-facing **BYO Knowledge Tool** (introduced in v1.3.1) targets the user knowledge-base database
+  (`docvec_user_kb`), not the system KB this guide writes to, so it is unaffected by this workflow. Use BYO
+  Knowledge for end-user document uploads, and use this guide for admin-curated corpora baked into the
+  deployment.
 - Embedding-model mismatch is the most common failure mode: vectors built with a non-`gte-multilingual-base`
   model will be silently retrievable but always score near zero, so the answer quality collapses without an
   obvious error. If you see "I don't have enough information" answers across the board after a custom-KB

docs/en/install.mdx

@@ -87,7 +87,7 @@ The install form is grouped below by topic. Required fields are marked **(requir
 - **Enable builtin PGVector** — when enabled, the chart provisions a single PostgreSQL + ParadeDB instance for Alauda Hyperflux. Set:
     - **PGVector Storage Size** — the storage size for the PostgreSQL PVC (e.g. `10Gi`).
     - **PGVector StorageClass name** — the Kubernetes storage class for the PVC, e.g. `sc-topolvm`.
-- When **disabled**, create a `Secret` with the external PostgreSQL connection info instead. Hyperflux uses three logical databases on the same instance — `docvec_sys_kb` (built-in product knowledge base), `docvec_user_kb` (user-uploaded BYO Knowledge), and `docvec_kb` (chat history) — all created automatically by the init container if missing.
+- When **disabled**, create a `Secret` (referenced via **pg database secret name**) with the external PostgreSQL connection info instead. Hyperflux uses three logical databases on the same instance — `docvec_sys_kb` (built-in product knowledge base), `docvec_user_kb` (user-uploaded knowledge base), and the chat-history database (defaults to `docvec`, configurable via **PG database name**) — all created automatically by the init container if missing.
     ```yaml
     apiVersion: v1
     kind: Secret
@@ -105,9 +105,9 @@ The install form is grouped below by topic. Required fields are marked **(requir
     Then enter the secret name in **pg database secret name**.
 - **PG database name** — the chat-history database name (default `docvec`). Created on first start if missing.
 - **PG collection name** — the LangChain PGVector collection name used by the server. The default value matches the bundled gte-multilingual-base dump and **should not be changed** unless you are deploying a custom KB built with [Build a Custom Knowledge Base](build-custom-kb.mdx).
-- **Built-in KnowledgeBase File** — choose which built-in dump to restore on first start:
-    - `docvec_gte_cs2000_<date>.dump` (default) — chunk size 2000, balanced recall and answer focus.
-    - `docvec_gte_cs3000_<date>.dump` — chunk size 3000, slightly better recall on long-form documents at the cost of larger context per hit.
+- **Built-in KnowledgeBase File** — choose which built-in dump to restore on first start. The selector currently offers:
+    - `docvec_gte_cs2000_20260410.dump` (default) — chunk size 2000, balanced recall and answer focus.
+    - `docvec_gte_cs3000_20260410.dump` — chunk size 3000, slightly better recall on long-form documents at the cost of larger context per hit.
 - **Enable builtin Redis** — when enabled, the chart provisions a single Redis instance used by the rate limiter. When disabled, supply a Redis credentials secret in **redis database secret name**.
 
 ### Scheduling
@@ -135,6 +135,7 @@ The install form is grouped below by topic. Required fields are marked **(requir
 - **Enable Agent Mode (required)** — turn on multi-step reasoning so the agent can call MCP tools. **Recommended:** use a strong LLM (≥ GPT-4 / Qwen-72B class) when this is on; smaller models can loop or misuse tools.
 - **Enable MCP Tools** — load ACP MCP tools so the agent can read live cluster state. Only available when Agent Mode is on.
 - **Expose MCP** — expose the bundled `acp-mcp-server` over Ingress so external MCP clients (e.g. IDE-side coding agents) can reach it. Only available when Agent Mode is on.
+- The bundled `acp-mcp-server` deployment itself is controlled by `smartdoc.enableMCPServer` (defaults to `true`); leave it on whenever Agent Mode is on.
 
 > **NOTE:** Earlier versions required setting an "MCP K8s API Server Address" (the `erebus` URL). That field has been
 > removed in v1.4.0 — the bundled `acp-mcp-server` now talks to the cluster directly inside the global cluster, and
@@ -161,11 +162,11 @@ The install form is grouped below by topic. Required fields are marked **(requir
 - **Max Input Tokens Per Day** — per-user input token cap, default `200000`.
 - **Max Output Tokens Per Day** — per-user output token cap, default `1000000`.
 
-Click **Install** to start installation. The init container will:
+Click **Install** to start installation. On first start the init container will:
 
-1. Create the three logical databases (`docvec_sys_kb`, `docvec_user_kb`, `docvec_kb`) if missing.
-2. Restore the selected built-in dump into `docvec_sys_kb`.
-3. Apply schema migrations and ensure the BM25 index exists.
+1. Restore the selected built-in dump into `docvec_sys_kb` (the dump already includes the BM25 index).
+2. Create the chat-history database and `docvec_user_kb` if they don't exist.
+3. Apply the `doc_id` btree index and URL-backfill schema migrations to both KB databases.
 
 ## Troubleshooting
 

docs/en/overview/release_notes.mdx

@@ -12,7 +12,9 @@ weight: 40
 
 The bundled knowledge base has been re-embedded with Alibaba's `gte-multilingual-base` model, replacing
 `paraphrase-multilingual-mpnet-base-v2`. Cross-language retrieval (e.g., a Chinese question against English
-docs) is substantially better, and recall@5 on the bundled ACP corpora rises to ≈ 0.82 with hybrid retrieval.
+docs) is substantially better, and the per-repo macro recall@5 on the bundled ACP corpora rises to ≈ 0.91
+with hybrid retrieval (up from a 0.8185 baseline that previously needed Cohere reranking stacked on top to
+reach 0.8350).
 Existing v1.3.x deployments are migrated automatically — see the [Upgrade guide](../upgrade/index.mdx).
 
 #### **Hybrid retrieval (BM25 + dense vector)**
@@ -49,8 +51,10 @@ without losing earlier context.
 #### **Service-to-service API key authentication (HMAC-SHA256)**
 
 A new `apiKey` / `apiKeyHeader` / `apiKeyMaxTimeDiff` set lets external services call the smart-doc API
-without going through the OAuth proxy. Tokens are HMAC of `<username>:<unix_ts>`, so the shared key never
-travels on the wire. Disabled by default — leave `apiKey` empty unless you need it.
+without going through the OAuth proxy. The token sent in the `apiKeyHeader` is a three-part colon string
+`<username>:<unix_ts>:<hex_hmac_sha256(API_KEY, 'username:ts')>`, so the shared key never travels on the wire
+and replayed tokens expire after `apiKeyMaxTimeDiff` seconds (default 300). Disabled by default — leave
+`apiKey` empty unless you need it.
 
 #### **MCP server exposed via Ingress**
 

docs/en/upgrade/index.mdx

@@ -37,13 +37,17 @@ For most version upgrades (except for v1.2.0 → v1.2.1 and v1.3.x → v1.4.x),
 When the new chart rolls out, the init container scans `docvec_sys_kb` for any legacy mpnet collection that
 matches a rule in `smartdoc.acpKbUpgradeRules` (default: ACP 4.1 and ACP 4.2 mpnet collections), then:
 
-1. Records the in-flight swap into a `kb_swap_state` row (so it can recover if the pod is killed mid-swap).
-2. Drops the old collection's tables in `docvec_sys_kb`.
-3. `pg_restore` the new gte dump from `/workspace-smart-doc/dumps/`.
-4. Renames the new collection back to the old collection name so the running server keeps querying the same
+1. Acquires a PostgreSQL advisory lock keyed on `acpKbDataVersion` so a second replica's init exits cleanly.
+2. Records the in-flight swap into a `kb_swap_state` row (so it can recover if the pod is killed mid-swap).
+3. Drops the `langchain_pg_embedding` and `langchain_pg_collection` tables in `docvec_sys_kb`.
+4. `pg_restore` the new gte dump from `/workspace-smart-doc/dumps/` (warnings about pre-existing extensions/
+   schemas are tolerated, hard errors abort).
+5. Verifies the new collection name landed (`SELECT count(*) ... = 1`); aborts otherwise.
+6. Renames the new collection back to the old collection name so the running server keeps querying the same
    name. **You do not need to change `pgconnect.pgCollectionName`.**
-5. Re-applies the `doc_id` btree index and the ParadeDB BM25 index.
-6. Stamps `schema_migrations` with `acpKbDataVersion` so re-runs are no-ops.
+7. Rebuilds the ParadeDB BM25 index if the new dump didn't already ship one, and re-runs the `doc_id` btree
+   and URL-backfill schema migrations against the freshly restored table.
+8. Stamps `schema_migrations` with `acpKbDataVersion` and clears the inflight `kb_swap_state` row.
 
 The swap is:
 
@@ -72,7 +76,7 @@ kubectl -n cpaas-system cp <postgre-vec-xxx>:/tmp/docvec_sys_kb.backup.dump ./do
 kubectl -n cpaas-system cp <postgre-vec-xxx>:/tmp/docvec_user_kb.backup.dump ./docvec_user_kb.backup.dump
 ```
 
-`docvec_user_kb` (BYO Knowledge) and `docvec_kb` (chat history) are **not touched** by the swap — but a
+`docvec_user_kb` and the chat-history database are **not touched** by the swap — but a
 backup is cheap insurance.
 
 ### Execute the upgrade
@@ -92,9 +96,12 @@ A successful swap prints lines like:
 
 ```
 [upgrade] swap docvec_mpnet_acp_4_1_1215 (in docvec_sys_kb) -> dump docvec_gte_acp_4_1_20260508.dump; final name=gte-multilingual-base_20260410
-[upgrade] data_version=20260508-gte-acp-4-x stamped; kb_swap_state cleared
+[upgrade] 20260508-gte-acp-4-x applied.
 ```
 
+A re-roll without a `acpKbDataVersion` change prints `[upgrade] 20260508-gte-acp-4-x already applied, skipping.`
+instead.
+
 Once the smart-doc pod becomes Ready, run a representative question through the chat UI to confirm answers are
 grounded in the new corpus.