Skip to content

Commit c4e4075

Browse files
AuraMindNestAuraMindNest
authored
Add Boost Endpoint API Documentation and Update README.md (#52)
* Add Boost Endpoint API Documentation * Update README — WEBLATE_ADD_APPS, Routes, and Celery * Update due to coderabbitai review
1 parent d6c55ad commit c4e4075

2 files changed

Lines changed: 552 additions & 7 deletions

File tree

‎README.md‎

Lines changed: 158 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ SPDX-License-Identifier: BSL-1.0
1818
| ---------- | ------ | -------- |
1919
| QuickBook | `boost_weblate.formats.quickbook` | Implemented |
2020

21-
Additional formats should follow the same split: a thin class under `src/boost_weblate/formats/` that plugs into Weblate’s format APIs, with parsing and reconstruction under `src/boost_weblate/utils/`.
21+
Additional formats should follow the same split: a thin class under `src/boost_weblate/formats/` that plugs into Weblate's format APIs, with parsing and reconstruction under `src/boost_weblate/utils/`.
2222

2323
## Quickstart
2424

@@ -69,37 +69,43 @@ prek install
6969

7070
## Architecture
7171

72-
Weblate discovers formats by **import path** (see [WEBLATE_FORMATS config](#weblate_formats-configuration)). This repository keeps a clear boundary between “what Weblate sees” and “how a file format works.”
72+
Weblate discovers formats by **import path** (see [WEBLATE_FORMATS config](#weblate_formats-configuration)). This repository keeps a clear boundary between "what Weblate sees" and "how a file format works."
7373

7474
```mermaid
7575
flowchart TB
7676
subgraph weblate["Weblate"]
7777
WF["WEBLATE_FORMATS"]
7878
CF["ConvertFormat / store"]
79+
RP["real_patterns (URL list)"]
7980
end
8081
subgraph plugin["boost_weblate"]
8182
FMT["formats/ — format adapters"]
8283
UTL["utils/ — parse & serialize"]
84+
EP["endpoint/ — HTTP API + Celery"]
8385
TST["tests/ — mirrors src layout"]
8486
end
8587
WF --> FMT
8688
FMT --> CF
8789
FMT --> UTL
90+
EP -->|AppConfig.ready()| RP
8891
TST -.-> FMT
8992
TST -.-> UTL
93+
TST -.-> EP
9094
```
9195

92-
- **`src/boost_weblate/formats/`** — Weblate-facing **format classes** (subclasses of Weblate’s `BaseFormat` family, such as `weblate.formats.convert.ConvertFormat`). `QuickBookFormat` follows the same pattern as built-in convert formats (for example AsciiDoc): it turns a template file into a translation store and, on save, applies translations back using the template plus the store.
96+
- **`src/boost_weblate/formats/`** — Weblate-facing **format classes** (subclasses of Weblate's `BaseFormat` family, such as `weblate.formats.convert.ConvertFormat`). `QuickBookFormat` follows the same pattern as built-in convert formats (for example AsciiDoc): it turns a template file into a translation store and, on save, applies translations back using the template plus the store.
9397

9498
- **`src/boost_weblate/utils/`** — **Format-specific logic** with no Weblate import cycle: QuickBook parsing, segment extraction, translate-toolkit storage (`QuickBookFile` / `QuickBookUnit`), and reconstruction (`QuickBookTranslator`). New formats should add a sibling module (or package) here.
9599

96-
- **`tests/`** — **Pytest** layout mirrors `src/boost_weblate/` (`tests/formats/`, `tests/utils/`, `tests/endpoint/`). Shared fixtures live under `tests/fixtures/`. `tests/conftest.py` configures `sys.path`, sets `DJANGO_SETTINGS_MODULE` to `tests.django_qbk_format_settings`, and calls `django.setup()` so format tests can load Weblate’s Django stack without requiring PostgreSQL.
100+
- **`src/boost_weblate/endpoint/`** — **HTTP API** for Boost documentation project/component management. Exposes three routes under `/boost-endpoint/` (see [Routes](#routes)), uses Django REST Framework for auth and serialization, and hands off heavy work to a Celery task (see [Celery task](#celery-task)).
101+
102+
- **`tests/`** — **Pytest** layout mirrors `src/boost_weblate/` (`tests/formats/`, `tests/utils/`, `tests/endpoint/`). Shared fixtures live under `tests/fixtures/`. `tests/conftest.py` configures `sys.path`, sets `DJANGO_SETTINGS_MODULE` to `tests.django_qbk_format_settings`, and calls `django.setup()` so format tests can load Weblate's Django stack without requiring PostgreSQL.
97103

98104
## WEBLATE_FORMATS configuration
99105

100106
Weblate discovers formats from the `WEBLATE_FORMATS` setting (see `FileFormatLoader` in upstream `weblate.formats.models`). The official Docker image evaluates a single optional file after base settings: if `/app/data/settings-override.py` exists, it is compiled and executed with `exec()` in the **same namespace** as the rest of `weblate.settings_docker`.
101107

102-
Stock `weblate.settings_docker` does **not** always bind `WEBLATE_FORMATS` in that namespace before the hook runs, so a bare `WEBLATE_FORMATS += (...)` in the override can raise `NameError`. This repository ships ``src/boost_weblate/settings_override.py`` as the Docker ``exec()`` fragment: it assigns ``WEBLATE_FORMATS`` by **reading** upstream ``weblate/formats/models.py`` and regex-slicing ``FormatsConf.FORMATS`` (aligned with the installed Weblate version without importing ``weblate.formats.models`` during settings load, which can raise ``AppRegistryNotReady``). It appends the endpoint Django app via ``INSTALLED_APPS += ("boost_weblate.endpoint.apps.BoostEndpointConfig",)``. If you also set ``WEBLATE_ADD_APPS`` to the same app, remove one source to avoid duplicate ``INSTALLED_APPS`` entries.
108+
Stock `weblate.settings_docker` does **not** always bind `WEBLATE_FORMATS` in that namespace before the hook runs, so a bare `WEBLATE_FORMATS += (...)` in the override can raise `NameError`. This repository ships `src/boost_weblate/settings_override.py` as the Docker `exec()` fragment: it assigns `WEBLATE_FORMATS` by **reading** upstream `weblate/formats/models.py` and regex-slicing `FormatsConf.FORMATS` (aligned with the installed Weblate version, without importing `weblate.formats.models` during settings load, which can raise `AppRegistryNotReady`). It also appends the endpoint Django app to `INSTALLED_APPS` — see [`WEBLATE_ADD_APPS`](#weblate_add_apps) below.
103109

104110
**Operators:** ensure the plugin package is installed in the Weblate environment (`pip` / image layer), then install the override file where Weblate expects it. For the stock Docker layout:
105111

@@ -109,15 +115,160 @@ COPY settings-override.py /app/data/settings-override.py
109115

110116
That path is fixed; Weblate does not scan `DATA_DIR` for arbitrary override files. The override file is **not** the same as `WEBLATE_PY_PATH` / `python/customize` (importable customization on `sys.path`); for format registration, use this exec hook unless your image explicitly imports another settings module. See the comments in `settings_override.py` for the full distinction.
111117

112-
**Adding another format:** implement the class under `boost_weblate/formats/`, append its dotted class path in ``weblate_formats_with_quickbook()`` (or extend the tuple built there), redeploy, and restart Weblate. If upstream changes the layout of ``FormatsConf`` in ``models.py``, update the regex in ``settings_override.py`` accordingly.
118+
**Adding another format:** implement the class under `boost_weblate/formats/`, append its dotted class path in `weblate_formats_with_quickbook()` (or extend the tuple built there), redeploy, and restart Weblate. If upstream changes the layout of `FormatsConf` in `models.py`, update the regex in `settings_override.py` accordingly.
119+
120+
## WEBLATE_ADD_APPS
121+
122+
`WEBLATE_ADD_APPS` is a Weblate Docker environment variable that appends entries to `INSTALLED_APPS` before the container starts (handled by Weblate's own Docker entrypoint, not by this plugin).
123+
124+
This plugin registers the endpoint Django app in `settings_override.py` directly:
125+
126+
```python
127+
# excerpt from src/boost_weblate/settings_override.py
128+
_INSTALLED_APPS = globals().get("INSTALLED_APPS")
129+
if _INSTALLED_APPS is not None:
130+
if isinstance(_INSTALLED_APPS, tuple):
131+
globals()["INSTALLED_APPS"] = _INSTALLED_APPS + (_ENDPOINT_APP_CONFIG,)
132+
else:
133+
_INSTALLED_APPS += (_ENDPOINT_APP_CONFIG,)
134+
```
135+
136+
where `_ENDPOINT_APP_CONFIG = "boost_weblate.endpoint.apps.BoostEndpointConfig"`.
137+
138+
**Two approaches — pick one, not both:**
139+
140+
| Approach | How it works | When to use |
141+
|----------|-------------|-------------|
142+
| `settings_override.py` (this repo) | `exec()`'d fragment appends to `INSTALLED_APPS` directly and also sets `WEBLATE_FORMATS` | Recommended — one file covers both format registration and app installation |
143+
| `WEBLATE_ADD_APPS` env var | Weblate Docker entrypoint adds to `INSTALLED_APPS` before Django starts | Use only if you are not deploying `settings_override.py` at all |
144+
145+
> **Important:** if you set `WEBLATE_ADD_APPS=boost_weblate.endpoint.apps.BoostEndpointConfig` **and** deploy `settings_override.py`, the app will be added to `INSTALLED_APPS` twice, which raises a `django.core.exceptions.ImproperlyConfigured` error at startup. Remove one source.
146+
147+
Note that adding the app to `INSTALLED_APPS` (by either method) is **necessary but not sufficient** for HTTP routes to be active — see [Routes](#routes) below for why.
148+
149+
## Routes
150+
151+
The plugin exposes three HTTP endpoints, all under the `/boost-endpoint/` prefix on the Weblate site:
152+
153+
| Method | Path | Handler | Auth | Response |
154+
|--------|------|---------|------|----------|
155+
| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | `200 ok` (plain text) |
156+
| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | `200` JSON: `module`, `version`, `capabilities` |
157+
| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | `202` JSON: `status`, `task_id`, `detail` |
158+
159+
### Why routes need explicit registration
160+
161+
Weblate's `urls.py` does **not** auto-discover URLconfs from arbitrary `INSTALLED_APPS` entries. It builds a single `real_patterns` list by hand and only extends it for known built-in apps (legal, SAML, git-export, etc.) via explicit `if "app" in settings.INSTALLED_APPS:` guards — there is no generic plugin scan.
162+
163+
This plugin handles registration in `BoostEndpointConfig.ready()` (`src/boost_weblate/endpoint/apps.py`), which runs once at Django startup and appends to `weblate.urls.real_patterns`:
164+
165+
```python
166+
wl_urls.real_patterns.append(
167+
path(
168+
"boost-endpoint/",
169+
include(("boost_weblate.endpoint.urls", "boost_endpoint")),
170+
),
171+
)
172+
```
173+
174+
The operation is idempotent (guarded by a `_cppa_boost_weblate_urls_registered` attribute on the module). Routes sit under Weblate's `URL_PREFIX` handling because `real_patterns` is used before the prefix wrapper is applied.
175+
176+
### Request / response for `POST /boost-endpoint/add-or-update/`
177+
178+
**Request body (JSON):**
179+
180+
```json
181+
{
182+
"organization": "boostorg",
183+
"version": "boost-1.90.0",
184+
"add_or_update": {
185+
"zh_Hans": ["json", "unordered"],
186+
"ja": ["json"]
187+
},
188+
"extensions": [".adoc", ".md"]
189+
}
190+
```
191+
192+
| Field | Type | Required | Description |
193+
|-------|------|----------|-------------|
194+
| `organization` | string | Yes | GitHub organization that owns the Boost submodule repos |
195+
| `version` | string | Yes | Boost release tag, e.g. `"boost-1.90.0"` |
196+
| `add_or_update` | object | Yes | Map of language code → list of submodule names (non-empty list per key) |
197+
| `extensions` | array of strings | No | File extensions to scan (e.g. `[".adoc", ".md"]`); defaults to all Weblate-supported extensions |
198+
199+
**Response (202 Accepted):**
200+
201+
```json
202+
{
203+
"status": "accepted",
204+
"task_id": "d3b07384-d9a2-4f9b-a0cf-1234567890ab",
205+
"detail": "Boost add-or-update is running in the background; check Celery logs or task result for completion."
206+
}
207+
```
208+
209+
The view validates the request with `AddOrUpdateRequestSerializer`, dispatches the Celery task, and returns immediately. A `400` response with an `errors` object is returned if validation fails.
210+
211+
## Celery task
212+
213+
Heavy work (git clone, file scanning, Weblate project/component create-or-update) runs asynchronously in a Celery worker via `boost_add_or_update_task` (`src/boost_weblate/endpoint/tasks.py`). The view enqueues the task with `.delay()` and returns HTTP 202 immediately.
214+
215+
```text
216+
POST /boost-endpoint/add-or-update/
217+
│
218+
▼
219+
AddOrUpdateView.post()
220+
Validate body → AddOrUpdateRequestSerializer
221+
│ valid
222+
▼
223+
boost_add_or_update_task.delay(
224+
organization, add_or_update, version, extensions, user_id
225+
)
226+
│ │
227+
│ HTTP 202 + task_id │ (worker picks up)
228+
◄─────────────────── ▼
229+
for each lang_code → submodule_list:
230+
BoostComponentService(org, lang, version, extensions)
231+
.process_all(submodules, user, request)
232+
returns dict[lang_code → result]
233+
```
234+
235+
**Task signature:**
236+
237+
```python
238+
@app.task(trail=False)
239+
def boost_add_or_update_task(
240+
*,
241+
organization: str,
242+
add_or_update: dict[str, list[str]],
243+
version: str,
244+
extensions: list[str] | None,
245+
user_id: int,
246+
) -> dict[str, Any]:
247+
```
248+
249+
- Uses Weblate's own Celery `app` instance (`weblate.utils.celery.app`), so it runs in the same worker pool as all other Weblate tasks with no extra broker configuration.
250+
- `user_id` is passed instead of the `User` object because Celery serializes task arguments to JSON; the task re-fetches the user from the database inside the worker.
251+
- Exceptions propagate unhandled so Celery marks the task as `FAILURE` and monitoring/alerting can act on it.
252+
- `trail=False` suppresses Celery's default task-result trail to avoid unbounded result-backend growth.
253+
254+
**`BoostComponentService`** (`src/boost_weblate/endpoint/services.py`) performs the actual work for each language:
255+
256+
1. Clone the GitHub submodule repository for the given organization, version, and language.
257+
2. Scan the cloned tree for files matching the requested (or all supported) extensions.
258+
3. Build Weblate `Project` and `Component` configurations from the scan results.
259+
4. Call `get_or_create` on each `Project`/`Component` via the Weblate ORM; update existing ones.
260+
5. Add the target language to each component via `add_new_language`.
261+
6. Delete stale components no longer present in the scan, commit, and push.
262+
263+
The service has no plugin-owned models; it operates entirely through Weblate's Django ORM.
113264

114265
## Contributing
115266

116267
- **Hooks:** use prek (or classic pre-commit) with `.pre-commit-config.yaml` so local runs match CI (Ruff, YAML/TOML checks, REUSE, actionlint, pytest).
117268

118269
- **Tests:** add tests next to the code you touch (`tests/formats/`, `tests/utils/`, or `tests/endpoint/`). Keep `django.setup()`-friendly patterns; heavy DB or migration suites are intentionally avoided in the bundled Django test settings.
119270

120-
- **CI coverage:** the *Lint and format* workflow runs a **Tests and coverage** job that prints `term-missing` output, runs `coverage report`, writes `coverage.xml` and `htmlcov/`, and uploads those plus `.coverage` as a workflow artifact (download from the run’s *Artifacts* section on GitHub). Coverage is configured in `pyproject.toml` (`[tool.coverage.*]`); the job uses `uv sync --frozen --group dev --group pre-commit` so `pytest-cov` and `coverage[toml]` match the lockfile.
271+
- **CI coverage:** the *Lint and format* workflow runs a **Tests and coverage** job that prints `term-missing` output, runs `coverage report`, writes `coverage.xml` and `htmlcov/`, and uploads those plus `.coverage` as a workflow artifact (download from the run's *Artifacts* section on GitHub). Coverage is configured in `pyproject.toml` (`[tool.coverage.*]`); the job uses `uv sync --frozen --group dev --group pre-commit` so `pytest-cov` and `coverage[toml]` match the lockfile.
121272

122273
- **Pull requests:** open PRs against the default branch on GitHub. Keep changes focused; ensure CI is green (build/wheel checks, lint, tests). Respond to review feedback on the PR thread; for design questions or bug reports, use [Issues](https://github.com/cppalliance/cppa-weblate-plugin/issues).
123274

0 commit comments

Comments
 (0)