Add Datasets module for managing geospatial and tabular data uploads - #45
Merged
Merged
Conversation
…l datasets
Adds a new ``gis_datasets`` module that lets users upload arbitrary GIS
files (GeoJSON, Shapefile, KML, GeoTIFF, CSV, etc.), browse them with
extracted metadata (CRS, bbox, feature/band counts), and download them.
* Local-filesystem storage abstracted behind ``LocalDatasetStorage`` so
an S3 backend can drop in later without touching the service.
* Best-effort metadata extraction (stdlib JSON for GeoJSON; optional
``fiona``/``rasterio`` paths for Shapefile/KML/raster) — never blocks
an upload when a parser is missing.
* REST API (``/api/gis_datasets``) + Inertia views (Browse/Create/Edit/Show),
permissions group, sidebar menu item, locale namespace, and a health
check that probes the storage dir.
* Alembic migration for the new ``gis_datasets_dataset`` table with
``branch_labels=("gis_datasets",)``.
Framework changes:
* ``i18n_manifest._serialize`` emits single-quoted TS string literals to
match the project's biome ``quoteStyle: "single"``, so the generated
``keys.generated.ts`` is biome-clean without a follow-up format pass.
* ``test_health.test_ready_no_checks_is_healthy`` now tolerates module-
registered checks (the old assertion broke whenever a new module
registered one).
https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Renames the module so it can catalog any dataset kind (not only GIS).
Mechanical changes only — no behaviour changes:
* ``modules/gis_datasets/`` → ``modules/datasets/`` (and inner
``gis_datasets/`` package → ``datasets/``).
* Class names: ``GisDatasetsModule`` → ``DatasetsModule``,
``GisDatasetsServices`` → ``DatasetsServices``,
``GisDatasetsSettings`` → ``DatasetsSettings``.
* Module meta: ``"GisDatasets"`` → ``"Datasets"``; routes flip from
``/api/gis_datasets`` and ``/gis_datasets`` to ``/api/datasets`` and
``/datasets``; Inertia namespace ``Datasets/Browse`` etc.
* Permissions: ``gis_datasets.{view,upload,edit,delete}`` →
``datasets.*`` (group label "Datasets").
* Settings env prefix: ``SM_GIS_DATASETS_*`` → ``SM_DATASETS_*``;
default storage dir ``./var/datasets``.
* Locale namespace ``gis_datasets`` → ``datasets``; sidebar label
"Datasets".
* Health check name ``datasets.storage``.
Database:
* Drops the ``gis_datasets_dataset`` migration and replaces it with
``add_datasets_table`` creating ``datasets_dataset`` (table prefix
follows the new module name; SQLite parity intact). Migration carries
``branch_labels=("datasets",)``.
All 585 tests still pass; lint and ``make doctor`` clean.
https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Makes the Datasets module easy to depend on. Downstream modules now have a single stable import path (``datasets.contracts``) and a one-line FastAPI dependency. Service protocol ``IDatasetService`` gains: * ``get_by_slug(slug)`` — slug is the URL-friendly handle other modules are most likely to persist as a foreign reference. * ``list_by_kind(kind, *, limit=None)`` — lets consumers filter to just the kinds they care about (e.g. a tiling module only fetches rasters). * ``get_file(dataset_id)`` → ``DatasetFile`` — opaque handle bundling the on-disk path, filename, mime type, and full ``DatasetOut`` metadata so a consumer can open/parse bytes without reaching into ``datasets.storage`` (which would trip SM009). Public value types: * ``DatasetFile`` — ``path`` + ``metadata`` + ``open()`` helper; designed to survive a future S3 storage backend swap. * URL builders ``download_url`` / ``detail_url`` / ``show_url`` plus ``API_PREFIX`` / ``VIEW_PREFIX`` constants, so dependent UIs don't hard-code ``/api/datasets/...``. Events carry slug: * ``DatasetUploaded`` and ``DatasetDeleted`` now include ``slug`` so subscribers can route/index without a round-trip to the service. Dependency ergonomics: * ``DatasetServiceDep`` ``Annotated`` alias — consuming endpoints write ``datasets: DatasetServiceDep`` instead of ``Depends(get_dataset_service)``. ``datasets.contracts.__init__`` now re-exports the whole surface with a docstring pointing consumers at the right entry points. Contract tests live in ``tests/test_public_surface.py`` — anything that fails there is an API-break for dependent modules. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Without its own ``tsconfig.json`` the datasets module's .tsx pages compiled only indirectly via the host's ``moduleGlobs``, so the ``ci-js-typecheck`` Makefile loop skipped them entirely. Type errors in Browse/Create/Edit/Show would have slipped past CI. Added: * ``package.json`` — declares the workspace name and UI/Inertia peers (mirrors ``modules/products/package.json``). * ``tsconfig.json`` — extends ``@simple-module/tsconfig/base.json`` with the standard ``@/*`` and ``@simple-module/ui/*`` path aliases. * ``pyproject.toml`` hatch ``force-include`` so the wheel ships the module-root ``package.json`` (host discovers JS deps via ``importlib.resources`` after a pip install). ``make ci-js-typecheck`` now runs ``tsc -p modules/datasets/tsconfig.json``. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
…nagement-YB2Wm # Conflicts: # .gitignore # framework/hosting/simple_module_hosting/i18n_manifest.py # host/pyproject.toml # packages/i18n/src/generated-resources.ts # packages/i18n/src/keys.generated.ts # pyproject.toml
…kend Datasets no longer owns its own filesystem. Bytes live in whatever ``file_storage`` backend is configured (local FS today, S3 tomorrow), so swapping the host's ``SM_FILE_STORAGE_BACKEND`` to ``s3`` gets every dataset upload/download/delete S3-native for free. Module wiring: * ``meta.depends_on = ["FileStorage"]`` — ensures file_storage's ``register_settings`` runs first, so ``app.state.file_storage.backend`` is populated before any datasets endpoint fires. * Dropped ``LocalDatasetStorage`` and the datasets-owned storage health check (file_storage has its own). * Dropped ``SM_DATASETS_STORAGE_DIR`` — file_storage's settings are the single source of truth. Service layer: * ``DatasetService.__init__`` now takes a ``StorageBackend`` directly. * Upload path spools to a tempfile (for fiona/rasterio/stdlib extractors that need a path), then streams bytes to ``backend.put()``. * Keys live under a ``datasets/`` prefix in the backend namespace so they don't collide with generic ``file_storage`` uploads. * Delete cleans up the backend object after the DB row is gone and tolerates already-missing objects via ``StorageNotFoundError``. Public contract: * ``DatasetFile`` is now backend-agnostic — no ``path`` attribute (that only made sense for local FS). Consumers call ``await file.stream()``, ``await file.read()``, or ``await file.materialize_to_tempfile()`` to get a local ``Path`` for GIS libraries that insist on one. * Download endpoint streams via ``StreamingResponse`` and will 302 to a presigned URL when the backend supports it. Tests updated to inject a ``FilesystemBackend(root=tmp_path)`` directly; public-surface tests cover the async ``stream`` / ``materialize_to_*`` pathway that downstream modules will rely on. All 7 CI items green locally: 741 Python tests, 8 JS tests, lint, typecheck (Python + TS), file-size, hardcoded-strings, biome. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
…nagement-YB2Wm # Conflicts: # package-lock.json # packages/i18n/src/generated-resources.ts # packages/i18n/src/keys.generated.ts # pyproject.toml
Follows the convention main #39 established: ship a Protocol only for real extension points (e.g. ``file_storage.StorageBackend`` with a registry and multiple backends). Datasets has one implementation and consumers already type-hint against the concrete ``DatasetService`` via the ``DatasetServiceDep`` annotated alias — the Protocol was ceremony. * Delete ``datasets/contracts/service.py`` and drop the re-export. * Tighten docstrings so nothing still points at the old Protocol. Also regenerates ``keys.generated.ts`` / ``generated-resources.ts`` to include the new feature_flags keys alongside datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Sidebar menu items render their icon via ``NavIcon``'s static SVG map. My datasets module's ``register_menu_items`` set ``icon="layers"`` but the map had no entry for that name, so the sidebar showed "Datasets" as text only while every other module got a glyph. Added a Heroicons ``layers`` path — the sidebar now renders consistently. Verified visually by running the dev stack and walking through the full flow: upload a GeoJSON, land on Browse with a populated row, follow into Show (dl/dt/dd detail card), then Edit (Card + form). All four datasets pages (Browse, Create, Edit, Show) use the same PageShell / Card / Table / Empty primitives as Products, Files, and Users. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Metadata extraction is the only expensive bit of the upload path — GeoJSON is fast but ``fiona`` (shapefile/KML) and ``rasterio`` can block for tens of seconds on large uploads, which is far too long to hold an HTTP request open. Moved extraction off the request path into a Celery task that the ``background_tasks`` module's worker picks up automatically. Flow on upload: 1. Endpoint spools bytes to a temp file for size-validation purposes. 2. Service persists a ``Dataset`` row with ``extraction_status="pending"`` — no bbox/CRS/feature_count yet — and uploads the bytes to ``file_storage.StorageBackend``. 3. Endpoint enqueues ``datasets.extract_metadata`` via Celery (task name exported as ``EXTRACT_METADATA_TASK``). 4. Worker calls ``extract_metadata_task(dataset_id)``: pulls bytes from the backend to a local tempfile, runs ``extract_metadata``, updates the row via the ``background_tasks.sync_db`` session. New files: * ``datasets/tasks.py`` — ``extract_metadata_task`` shared-task. Uses ``asyncio.run`` to drive the async file_storage backend from inside the sync Celery worker. Autodiscovered by ``background_tasks.celery_app`` — no framework hook. * ``tests/test_tasks.py`` — unit tests with ``sync_db`` + file_storage pointed at tmp paths via ``monkeypatch`` env vars. Verifies the task fills in bbox/CRS/feature_count and degrades to ``not_found`` when the row has been deleted between enqueue and run. * ``tests/conftest.py`` — autouse fixture that stubs ``app.state.background_tasks.celery`` with a ``MagicMock`` (mirrors the admin suite's setup) so the upload endpoint can call ``send_task`` without a live Redis broker in CI. Module wiring: * ``meta.depends_on = ["FileStorage", "BackgroundTasks"]``. * ``datasets/pyproject.toml`` picks up ``background-tasks`` + ``celery`` as explicit deps so ``datasets`` can be installed standalone. * New ``get_celery`` dep in ``datasets/deps.py`` reads from ``app.state.background_tasks.celery``. Refactored service: * ``DatasetService.register_upload`` no longer calls ``extract_metadata`` — it persists the row, uploads bytes, and leaves ``extraction_status="pending"``. The endpoint does the enqueue. The background_tasks admin UI at ``/admin/background-tasks`` now shows every dataset upload as a task row, walks it through ``pending → running → success/failed``, and makes it retryable. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Introduces ``datasets/constants.py`` as the single source of truth for every module identifier — permissions, menu, routes, pages, Celery task name, env prefix, table/schema names, dependency module names, kinds, extraction-status values, defaults. Every other file imports from it instead of embedding string literals, mirroring the pattern used by ``file_storage``, ``products``, ``background_tasks``, etc. Files touched: * ``module.py`` — ``MODULE_PASCAL``, ``ROUTE_PREFIX_*``, ``MODULE_FILE_STORAGE``/``MODULE_BACKGROUND_TASKS`` (for ``depends_on``), ``PERMISSION_GROUP``, ``ALL_PERMISSIONS``, menu constants. * ``endpoints/api.py`` — ``PERM_DATASETS_*`` via ``RequiresPermission(...)``, ``TASK_EXTRACT_METADATA`` for the Celery enqueue, ``ALL_KINDS`` for the kind allow-list. * ``endpoints/views.py`` — module-local ``_PAGE_*`` string constants (Name-only literals, required so ``simple_module_core.diagnostics`` SM003 orphan-page check can resolve them via its AST walker); ``constants.PERM_DATASETS_*`` for permissions. * ``service.py``, ``tasks.py`` — ``ExtractionStatus.*``, ``DEFAULT_*``, ``STORAGE_KEY_PREFIX``, ``TASK_EXTRACT_METADATA``. * ``models.py``, ``settings.py``, ``extractors.py``, ``schemas.py`` — ``SCHEMA_NAME``, ``TABLE_DATASET``, ``ENV_PREFIX``, ``DatasetKind`` enum, ``DEFAULT_GEOJSON_CRS``, ``DEFAULT_MAX_UPLOAD_MB``. Also drops the remaining "GIS"/"geospatial" framing from user-visible strings and docstrings — the catalog now describes itself as a generic dataset catalog with optional geospatial metadata extraction. The ``fiona``/``rasterio`` library references stay in code comments because those are the actual library names. * ``locales/en.json`` — "geospatial datasets" → "datasets" across browse description, empty state, create description. * ``models.py`` — "A geospatial dataset ..." → "A dataset ...". * ``extractors.py`` — module docstring drops "GIS" framing. * ``contracts/files.py`` — "GIS library" → "parser library" / named libraries. All 7 CI items green locally: 816 python tests, 8 JS tests, ruff format/check, ty, biome, tsc x 11 workspaces, 300-line cap, ``check_hardcoded_strings`` clean, ``make doctor`` shows no SM003 warnings against datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
Rounds out the module's integration with the framework's governance primitives. Every identifier lives in ``datasets/constants.py``. Feature flags (registered via ``register_feature_flags``): * ``datasets.auto_extract`` (default on) — guards the Celery ``send_task`` call on the upload endpoint. Admins can freeze auto-extraction during incidents without redeploying; Dataset rows stay ``pending`` and can be re-triggered later. * ``datasets.allow_raster_uploads`` (default on) — 422s GeoTIFF uploads with a clear "raster uploads are disabled" message when turned off, useful on instances without ``rasterio`` installed. Permission role mapping (via ``registry.map_role``): * The ``user`` role now inherits ``datasets.view`` + ``datasets.upload`` so non-admins can browse and self-upload. ``edit`` / ``delete`` stay admin-only through the framework wildcard. Settings registration (registered via ``on_startup`` against the ``settings`` module's registry, skipped gracefully if the settings module isn't installed): * ``datasets.max_upload_mb`` — runtime override for the per-dataset upload size. ``get_max_upload_bytes`` reads this on every upload request, falling back to the env-backed default on the module's ``DatasetsSettings``. * ``datasets.default_kind`` — the kind to assign when filename-based detection falls back to ``other``. Resilience fix: * ``send_task`` wrapped in a logger-suppressed try/except. A dead broker (Redis down, network blip) no longer cascades into a failed upload — the dataset row is already persisted with ``extraction_status="pending"`` and the failure is logged for operator retry. Tests: * ``tests/test_wiring.py`` covers flag registration, permission role mapping, setting registration (via ``getattr(app.state, "settings", None)`` so it skips gracefully), and the flag-toggle behaviours on upload (auto-extract off → no ``send_task``; raster flag off → 422). Verified visually with a live dev run — see ``.playwright-cli/video-2026-04-20T17-02-55-633Z.webm`` for the full walk-through (login → upload → bbox/CRS extraction → browse → show → feature flags admin → module settings admin). All 7 CI items green: ruff/ty/biome/tsc, 823 python tests, 8 JS tests, file-size cap, ``check_hardcoded_strings`` clean, ``make doctor`` 0 errors on datasets. https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
CI's ``make install-js`` runs ``npm ci`` which fails when the lock file
disagrees with any workspace ``package.json``. The
``@simple-module/datasets`` workspace was added a while back but its
entry never made it into ``package-lock.json``, so every JS job
(lint, typecheck, tests, build) was failing during install with::
npm error Missing: @simple-module/datasets@0.1.0 from lock file
Local ``npm install`` had been silently fixing this each time I ran it
which is why CI was the only place that surfaced the gap. Re-ran
``npm install`` and committed the resulting lock-file delta (17
lines, just the new workspace's metadata).
https://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new Datasets module that provides a complete system for uploading, storing, and managing geospatial and tabular datasets. The module integrates with the file storage backend for bytes persistence and uses Celery for asynchronous metadata extraction.
Key Changes
Core Service Layer
DatasetService: Orchestrates dataset CRUD operations, file uploads, and metadata extraction
Metadata Extraction (
extractors.py):rasteriofionaAPI Endpoints
REST API (
endpoints/api.py):GET /api/datasets/- List all datasetsGET /api/datasets/{id}- Get dataset detailsPOST /api/datasets/- Upload new dataset with multipart form dataPATCH /api/datasets/{id}- Update dataset metadataDELETE /api/datasets/{id}- Delete datasetGET /api/datasets/{id}/download- Download with presigned URL supportView Endpoints (
endpoints/views.py):Background Tasks
tasks.py):extract_metadata_task: Asynchronously extracts metadata from uploaded filesData Models & Contracts
Datasettable with soft-delete support, audit timestamps, and spatial metadata fieldscontracts/):DatasetOut: DTO for API responsesDatasetFile: Handle for file access abstractionDatasetUploaded/DatasetDeleted: Domain events for subscribersFrontend
Configuration & Wiring
FLAG_AUTO_EXTRACT: Toggle automatic metadata extraction on uploadFLAG_ALLOW_RASTER_UPLOADS: Control raster file acceptancedatasets.view,datasets.upload,datasets.edit,datasets.deleteDatabase
datasets_datasettable with columns for metadata, spatial info, and extraction statusNotable Implementation Details
file_storage.StorageBackend, enabling S3/GCS/Azure support without module changeshttps://claude.ai/code/session_01W384SpjDjwzuZmQq2WV4Qv