Patch endpoints #744

rhysrevans3 · 2024-08-21T07:44:51Z

Description:
Adds PATCH endpoints to transaction extension. Adds support for RFC 6902 and RFC 7396. Pivots on header Content-Type value.

Related pull request: stac-api-extensions/transaction#14

PR Checklist:

pre-commit hooks pass locally
Tests pass (run make test)
Documentation has been updated to reflect changes, if applicable, and docs build successfully (run make docs)
Changes are added to the CHANGELOG.

…endpoints

rhysrevans3 · 2024-08-23T13:56:16Z

I've had to switch from stac-pydantic Item/Collection to Dict to allow for partial updates. Not sure if this is the best method or if you can switch off pydantic validation. Another option would be having separate models for partial items/collections in stac-pydantic where all attributes.

vincentsarago · 2024-08-26T13:39:51Z

I've had to switch from stac-pydantic Item/Collection to Dict to allow for partial updates. Not sure if this is the best method or if you can switch off pydantic validation. Another option would be having separate models for partial items/collections in stac-pydantic where all attributes.

Would a typedDict make more sense?

…endpoints

stac_fastapi/types/stac_fastapi/types/transaction.py

stac_fastapi/types/stac_fastapi/types/core.py

stac_fastapi/types/stac_fastapi/types/transaction.py

Adding default for content_type.

rhysrevans3 · 2025-04-02T13:30:24Z

@vincentsarago you're right I had missed that. I've updated json_patch_item and json_patch_collection to raise an NotImplementedError as a default. Is that enough or does it need to be in the registration of the endpoint?

Happy to use this pull request to move everything to the transaction to the extension module if required.

rhysrevans3 · 2025-04-02T13:45:43Z

I could also add a NotImplemented default to the merge-patch to not break existing implementations? I'm unsure if the Transaction extension requires a PATCH endpoint and if it requires merge-patch or if you could specify a different Content-Type. For example if you wanted to implement json-patch and not merge-patch or even a non-JSON PATCH like XML PATCH RFC 5261

stac_fastapi/types/stac_fastapi/types/core.py

vincentsarago · 2025-04-02T18:27:15Z

stac_fastapi/types/stac_fastapi/types/core.py

+        collection_id: str,
+        item_id: str,
+        patch: Any,
+        content_type: Optional[str],


Suggested change

content_type: Optional[str],

content_type: Optional[str] = None,

Should the default be None or application/json? I've set it to None for now.

I think None is fine.

we could also totally remove it from the arg list and use request: Request

That was my attempt to make it configurable in the api.html page but it doesn't seem like it actually sets the Content-Type so I'd be happy to go back to using request if that's better for stac-fastapi.

fastapi/fastapi#7786 (comment)

I think you can achieve this with openapi customization

def register_patch_item(self): """Register patch item endpoint (PATCH /collections/{collection_id}/items/{item_id}).""" self.router.add_api_route( name="Patch Item", path="/collections/{collection_id}/items/{item_id}", response_model=Item if self.settings.enable_response_models else None, responses={ 200: { "content": { MimeTypes.geojson.value: {}, }, "model": Item, } }, openapi_extra={ "requestBody": { "content": { "application/json-patch+json": { "schema": ..., }, "application/merge-patch+json": { "schema": ..., }, "application/json": { "schema": ..., } }, "required": True, }, }, response_class=self.response_class, response_model_exclude_unset=True, response_model_exclude_none=True, methods=["PATCH"], endpoint=create_async_endpoint( self.client.patch_item, PatchItem, ), ) def register_patch_collection(self): """Register patch collection endpoint (PATCH /collections/{collection_id}).""" self.router.add_api_route( name="Patch Collection", path="/collections/{collection_id}", response_model=Collection if self.settings.enable_response_models else None, responses={ 200: { "content": { MimeTypes.geojson.value: {}, }, "model": Collection, } }, openapi_extra={ "requestBody": { "content": { "application/json-patch+json": { "schema": ..., }, "application/merge-patch+json": { "schema": ..., }, "application/json": { "schema": ..., } }, "required": True, }, }, response_class=self.response_class, response_model_exclude_unset=True, response_model_exclude_none=True, methods=["PATCH"], endpoint=create_async_endpoint( self.client.patch_collection, PatchCollection, ), )

I'll let you figure out the schemas 😅

I think if would be fine to advertise support for the 3 content-type application/json-patch+json, application/merge-patch+jsonandapplication/jsonand then let the application raiseNotImplementedErrorwithin the client method... or we make this configurable... or we just document how people could add themerge` content-type

Thanks that's just what I was looking! I've added openapi_extra for patch item and collection and removed the content_type parameter.

stac_fastapi/types/stac_fastapi/types/core.py

rhysrevans3 · 2025-04-03T11:07:50Z

I've written a conversion from merge to patch for the elasticsearch-opensearch backend if this is useful to other backends? I could move it to the core utils?

https://github.com/stac-utils/stac-fastapi-elasticsearch-opensearch/blob/2b1381821ec978ea254df0c8fae87420186a8789/stac_fastapi/core/stac_fastapi/core/utilities.py#L145-L171

vincentsarago · 2025-04-03T11:41:50Z

@rhysrevans3 yeah that would be great. It will lower the barrier for people to support both patch and merge 🙏

Maybe it could be a method on the PartialItem/PartialCollection model

☝️ while writing this I realized that they are typedDict not pydantic model 🤔

rhysrevans3 · 2025-04-03T11:54:15Z

@rhysrevans3 yeah that would be great. It will lower the barrier for people to support both patch and merge 🙏

Maybe it could be a method on the PartialItem/PartialCollection model

☝️ while writing this I realized that they are typedDict not pydantic model 🤔

Yes I copied the Item/Collection models in the same file. I think the reason for using TypeDict over Pydantic models was performance (avoid validation) I remember there was some discussion when we moved to pydantic v2 #625 which suggested that's no longer an issue? Should I switch to using Pydantic models?

vincentsarago · 2025-04-03T11:56:52Z

@rhysrevans3 I'm sorry it's taking longer 😅

I now think (with the addition of a merge to patch method) that by default we should make both patch/merge supported by default 😅.

On think I'm still not sure is how the merging/patching is made.

I'm looking at https://github.com/rhysrevans3/stac-fastapi-elasticsearch/blob/2b1381821ec978ea254df0c8fae87420186a8789/stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py#L940-L1031 to try to understand what's going on.

We should write some pseudo code in the doc (in the docstring first) to show how the patch method will look like

    def patch_item(
        self,
        collection_id: str,
        item_id: str,
        patch: Union[stac.PartialItem, List[stac.PatchOperation]],
        **kwargs,
    ) -> Optional[Union[stac.Item, Response]]:
        """Update an item from a collection.

        Called with `PATCH /collections/{collection_id}/items/{item_id}`

        Args:
            item_id: id of the item.
            collection_id: id of the collection.
            patch: either the partial item or list of patch operations.

        Returns:
            The patched item.
        """
        # convert patch item to list of merge operations
        if isinstance(patch, PartialItem):
            patch = merge_to_operations(patch)

        # Get Original Item
        item = backend.get_item(collection_id, item_id)

        # Update Item body from merge op
        item = update_item(item, patch)

        # Push Item to the backend
        _ = backend.create_item(collection_id, item_id, item)

        return item

vincentsarago · 2025-04-03T11:57:50Z

Should I switch to using Pydantic models?

Yes we should use pydantic model for input validation. It's only for the output that we have optional validation

rhysrevans3 · 2025-04-03T12:26:55Z

@rhysrevans3 I'm sorry it's taking longer 😅

I now think (with the addition of a merge to patch method) that by default we should make both patch/merge supported by default 😅.

On think I'm still not sure is how the merging/patching is made.

I'm looking at https://github.com/rhysrevans3/stac-fastapi-elasticsearch/blob/2b1381821ec978ea254df0c8fae87420186a8789/stac_fastapi/elasticsearch/stac_fastapi/elasticsearch/database_logic.py#L940-L1031 to try to understand what's going on.

We should write some pseudo code in the doc (in the docstring first) to show how the patch method will look like

    def patch_item(
        self,
        collection_id: str,
        item_id: str,
        patch: Union[stac.PartialItem, List[stac.PatchOperation]],
        **kwargs,
    ) -> Optional[Union[stac.Item, Response]]:
        """Update an item from a collection.

        Called with `PATCH /collections/{collection_id}/items/{item_id}`

        Args:
            item_id: id of the item.
            collection_id: id of the collection.
            patch: either the partial item or list of patch operations.

        Returns:
            The patched item.
        """
        # convert patch item to list of merge operations
        if isinstance(patch, PartialItem):
            patch = merge_to_operations(patch)

        # Get Original Item
        item = backend.get_item(collection_id, item_id)

        # Update Item body from merge op
        item = update_item(item, patch)

        # Push Item to the backend
        _ = backend.create_item(collection_id, item_id, item)

        return item

No worries I'd rather it takes a bit longer and we get it right!

For Elasticsearch you can update records using Java style scripts so for each operation type I've written an equivalent Elasticsearch script. I'm not sure if you could do the same for PostgreSQL or the other backends.

A merge is equivalent to a group of add and remove operations (null values are used to remove). so you can convert from merge to operations to Elasticsearch.

stac_fastapi/types/stac_fastapi/types/stac.py

stac_fastapi/extensions/tests/test_transaction.py

rhysrevans3 · 2025-04-04T09:42:21Z

I've not tested this but this is an example of running a list patch operations on an item:

from stac_fastapi.types.stac import PatchOperation
from stac_pydantic import Item

def patch_item(item: Item, operations: PatchOperation) -> Item:
    item_dict = item.model_dump()

    for operation in operations:
        path_parts = operation.path.split('/')
 
       if operation.op == "test":
            test_value = item_dict.copy()
            for path_part in path_parts:
                test_value = test_value[path_part]

            assert test_value == operation.value
            continue

        if operation.op == "replace":
            nest = item_dict.copy()
            for path_part in path_parts:
                assert path_part in nest
                nest = nest[path_part]

        update = {}

        if operation.op in ["add", "copy", "replace", "move"]:
            if operation.hasattr("from_"):
                from_parts = operation.from_.split('/')

                value = item_dict.copy()
                for path_part in from_parts:
                    value = value[path_part]

            else:
                value = item.value

            update = value
            for path_part in path_parts.reverse():
                update = {path_part: update}

        if operation.op in ["remove", "move"]:
            if operation.op == "move":
                path_parts = from_parts
 
            last_part = path_parts.pop(-1)

            nest = item_dict
            for path_part in path_parts:
                nest = nest[path_part]

            del nest[last_part]

        return Item.model_validate(item_dict | update)

rhysrevans3 added 9 commits August 20, 2024 12:43

Adding patch endpoints.

0cddff7

Merge branch 'main' of github.com:stac-utils/stac-fastapi into patch_…

14d18f9

…endpoints

Adding annotated from main.

0c5de64

Fixing and adding tests.

bf2ddbb

Updating changelog.

632d5a5

Fixing ruff errors.

5f2b4fa

Ruff format.

010e2cb

Switching to List for python 3.8.

0ccded0

Updating docs make file.

1b46754

rhysrevans3 marked this pull request as ready for review August 22, 2024 08:37

jonhealy1 requested a review from vincentsarago August 23, 2024 12:45

rhysrevans3 added 3 commits August 23, 2024 14:32

Switching from Item/Collection to Dict to allow partial updates.

79c769c

Ruff format fix.

68a65a0

Fixing broken tests.

88b40d4

rhysrevans3 added 2 commits August 23, 2024 15:18

Adding missing asyncs for patchs.

b7bcbd5

Moving request to kwargs for patch item and collection.

cfc31c6

rhysrevans3 added 2 commits August 28, 2024 11:13

Switching to TypedDict.

81dbcad

Merge branch 'main' of github.com:stac-utils/stac-fastapi into patch_…

f053f07

…endpoints