Skip to content

Epic: ability to append data to an existing dataset #138

@rudokemper

Description

@rudokemper

Feature Request

Currently, for one-time uploads (e.g. via the Dataset Uploader app or by running one of the connector scripts like GeoJSON to Postgres), we do not provide a robust way to append data to an existing dataset:

  • The Dataset Uploader app explicitly prevents you from uploading a new dataset if the database table already exists. This is by design for now to keep scope for an MVP of this app, but there is a TODO in the app's README to make it possible to modify this app to update an existing dataset.
  • The connector scripts, eventually calling StructureDBWriter._safe_insert, will try to update existing records based on primary key. But sometimes we need to generate a unique ID to use as primary key if there is none, and it becomes tricky to do that deterministically across multiple uploads, e.g. if the user uploads different versions of the same dataset to add new data that was created since the last upload. For this, see discussion in Generate unique ID if not present in GeoJSON feature #136.

For now, this is just a placeholder issue for a future epic, to ensure we don't forget about this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions