Skip to content

feature: artifact-helper #750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

pvbouwel
Copy link

Allow uploading artifacts using:

from openeo.extra.artifacts import ArtifactHelper

artifact_helper = ArtifactHelper.from_openeo_connection(connection)
storage_uri = artifact_helper.upload_file(object_name, src_file_path)
presigned_uri = artifact_helper.get_presigned_url(storage_uri)

Where object_name is a logical string name that will be used to give a unique name in the backend storage.
And src_file_path points to a local file that needs to be stored remotely.

This MVP version still has the sts and s3 URIs hard coded. This is temporary. It is possible to expand the links that are exposed at the root_url (https://openeo.dataspace.copernicus.eu/openeo/1.2/) to include rels like 'artifacts-s3-endpoint' and 'artifacts-sts-endpoint'. That way different artifact backends can be chosen based on the OpenEO environment.

@pvbouwel pvbouwel requested review from soxofaan and jdries March 13, 2025 14:39
Copy link
Member

@soxofaan soxofaan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is quite a large diff 😄 and I haven't gone deep yet, but here are some initial notes

Copy link
Member

@soxofaan soxofaan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit short on resources to go deep, so just some superficial notes:

This PR adds a lot of new things (lot of new modules, lot of classes) to the public API of the openeo package. Is it necessary to make all of this public? Once something is public, it's hard to change things because you have to care about compatibility. It's also overwhelming to the user I think.

I also think this feature should be documented in the docs, e.g. somewhere under https://open-eo.github.io/openeo-python-client/cookbook/index.html . Doing that documentation exercise will also make it more clear what the minimal public user-facing part of the API is.
I see you already did some docs on ArtifactHelper, so it should be easy to bootstrap initial docs from that like done at

.. autoclass:: openeo.extra.job_management.MultiBackendJobManager
:members:

@pvbouwel pvbouwel force-pushed the feature/openeo-client-artifact-uploader branch from d78de33 to f6b3809 Compare June 27, 2025 12:30
@pvbouwel
Copy link
Author

@soxofaan provided docs as requested. I did move around code but mostly renames of files to start with _ to discourage users from entering.

Other changes:

  • introduced feature_flags for s3sts such that backend can advertise what is supports
  • now decided on advertised feature flag whether to enable/disable checksumming
  • more tests
  • Added JSON lexer for sphinx as JSON would not render properly otherwise

@pvbouwel pvbouwel requested a review from soxofaan June 27, 2025 12:37
Peter Van Bouwel added 2 commits July 16, 2025 10:09
…sible during processing

See `API: openeo.extra.artifacts` in the documentation for details.
@pvbouwel pvbouwel force-pushed the feature/openeo-client-artifact-uploader branch from d3d610d to 2dd009f Compare July 16, 2025 08:10
@pvbouwel
Copy link
Author

Unittests are failing because of another change: #788

…ent of advertised config unless a required parameter is missing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants