Skip to content

Pluggable transfers and multipart transfer implementation #1705

Pluggable transfers and multipart transfer implementation

Pluggable transfers and multipart transfer implementation #1705

Triggered via pull request March 9, 2025 12:42
Status Failure
Total duration 3m 56s
Artifacts

tests.yml

on: pull_request
Matrix: Tests / Tests
Fit to window
Zoom out
Zoom in

Annotations

20 errors and 20 warnings
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L128
test_record_files_operations assert 2 == 1 + where 2 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e4d03e4c0>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e4d03e4c0> = Bucket.query
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L150
test_record_files_clear AssertionError: assert 4 == 3 + where 4 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e4d021070>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e4d021070> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L188
test_record_files_teardown_full AssertionError: assert 7 == 3 + where 7 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e5523b400>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e5523b400> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L226
test_record_files_teardown_partial AssertionError: assert 10 == 3 + where 10 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e4b15f760>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e4b15f760> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L269
test_record_files_soft_delete AssertionError: assert 13 == 3 + where 13 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e4c94d0a0>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e4c94d0a0> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.9, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L341
test_record_files_copy assert 10 == 1 + where 10 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f4e4b489640>.count + where <flask_sqlalchemy.query.Query object at 0x7f4e4b489640> = ObjectVersion.query
Tests / Tests (3.9, postgresql14, opensearch2): tests/services/files/s3/conftest.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py 2025-03-09 12:43:08.514964+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py 2025-03-09 12:44:39.902799+00:00 @@ -39,10 +39,11 @@ app_config["S3_ACCESS_KEY_ID"] = os.environ["S3_ACCESS_KEY_ID"] app_config["S3_SECRET_ACCESS_KEY"] = os.environ["S3_SECRET_ACCESS_KEY"] return app_config + @pytest.fixture() def s3_location(app, db): """Creates an s3 location for a test.""" from invenio_files_rest.models import Location
Tests / Tests (3.9, postgresql14, opensearch2): tests/services/files/s3/conftest.py#L1
pydocstyle-check /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py:1 at module level: D100: Missing docstring in public module
Tests / Tests (3.9, postgresql14, opensearch2): tests/services/files/s3/test_service_s3_backend.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/test_service_s3_backend.py 2025-03-09 12:43:08.514964+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/test_service_s3_backend.py 2025-03-09 12:44:40.279086+00:00 @@ -10,11 +10,11 @@ file_service, s3_location, example_s3_file_record, identity_simple, ): - + recid = example_s3_file_record["id"] key = "dataset.bin" total_size = 17 * 1024 * 1024 # 17MB part_size = 10 * 1024 * 1024 # 10MB @@ -90,6 +90,6 @@ result = file_service.get_file_content(identity_simple, recid, key) assert result.file_id == key # get the content from S3 and make sure it matches the original content sent_file = result.send_file() - assert content == requests.get(sent_file.headers['Location']).content + assert content == requests.get(sent_file.headers["Location"]).content
Tests / Tests (3.9, postgresql14, opensearch2): invenio_records_resources/services/files/tasks.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/invenio_records_resources/services/files/tasks.py 2025-03-09 12:43:08.504964+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/invenio_records_resources/services/files/tasks.py 2025-03-09 12:45:36.495832+00:00 @@ -90,11 +90,13 @@ if not checksum.startswith("multipart:"): return # multipart checksum looks like: multipart:<s3 multipart checksum>-part_size # s3 multipart checksum is the etag of the multipart object and looks like # hex(md5(<md5(part1) + md5(part2) + ...>))-<number of parts> - original_checksum_hex, _number_of_parts_str, part_size_str = checksum[10:].rsplit("-") + original_checksum_hex, _number_of_parts_str, part_size_str = checksum[ + 10: + ].rsplit("-") part_size = int(part_size_str) storage = current_files_rest.storage_factory(fileinstance=file_instance) with storage.open("rb") as f: object_checksum = hashlib.md5()
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L128
test_record_files_operations assert 2 == 1 + where 2 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05c65c70>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05c65c70> = Bucket.query
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L150
test_record_files_clear AssertionError: assert 4 == 3 + where 4 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05c278c0>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05c278c0> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L188
test_record_files_teardown_full AssertionError: assert 7 == 3 + where 7 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05c718e0>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05c718e0> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L226
test_record_files_teardown_partial AssertionError: assert 10 == 3 + where 10 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05c73530>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05c73530> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L269
test_record_files_soft_delete AssertionError: assert 13 == 3 + where 13 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05027800>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05027800> = <class 'mock_module.models.FileRecordMetadata'>.query + where <class 'mock_module.models.FileRecordMetadata'> = models.FileRecordMetadata
Tests / Tests (3.12, postgresql14, opensearch2): tests/records/test_systemfield_files.py#L341
test_record_files_copy assert 10 == 1 + where 10 = count() + where count = <flask_sqlalchemy.query.Query object at 0x7f8d05c27350>.count + where <flask_sqlalchemy.query.Query object at 0x7f8d05c27350> = ObjectVersion.query
Tests / Tests (3.12, postgresql14, opensearch2): tests/services/files/s3/conftest.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py 2025-03-09 12:43:08.162519+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py 2025-03-09 12:45:04.657264+00:00 @@ -39,10 +39,11 @@ app_config["S3_ACCESS_KEY_ID"] = os.environ["S3_ACCESS_KEY_ID"] app_config["S3_SECRET_ACCESS_KEY"] = os.environ["S3_SECRET_ACCESS_KEY"] return app_config + @pytest.fixture() def s3_location(app, db): """Creates an s3 location for a test.""" from invenio_files_rest.models import Location
Tests / Tests (3.12, postgresql14, opensearch2): tests/services/files/s3/conftest.py#L1
pydocstyle-check /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/conftest.py:1 at module level: D100: Missing docstring in public module
Tests / Tests (3.12, postgresql14, opensearch2): tests/services/files/s3/test_service_s3_backend.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/test_service_s3_backend.py 2025-03-09 12:43:08.162519+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/tests/services/files/s3/test_service_s3_backend.py 2025-03-09 12:45:05.217098+00:00 @@ -10,11 +10,11 @@ file_service, s3_location, example_s3_file_record, identity_simple, ): - + recid = example_s3_file_record["id"] key = "dataset.bin" total_size = 17 * 1024 * 1024 # 17MB part_size = 10 * 1024 * 1024 # 10MB @@ -90,6 +90,6 @@ result = file_service.get_file_content(identity_simple, recid, key) assert result.file_id == key # get the content from S3 and make sure it matches the original content sent_file = result.send_file() - assert content == requests.get(sent_file.headers['Location']).content + assert content == requests.get(sent_file.headers["Location"]).content
Tests / Tests (3.12, postgresql14, opensearch2): invenio_records_resources/services/files/tasks.py#L1
Black format check --- /home/runner/work/invenio-records-resources/invenio-records-resources/invenio_records_resources/services/files/tasks.py 2025-03-09 12:43:08.152519+00:00 +++ /home/runner/work/invenio-records-resources/invenio-records-resources/invenio_records_resources/services/files/tasks.py 2025-03-09 12:46:23.053853+00:00 @@ -90,11 +90,13 @@ if not checksum.startswith("multipart:"): return # multipart checksum looks like: multipart:<s3 multipart checksum>-part_size # s3 multipart checksum is the etag of the multipart object and looks like # hex(md5(<md5(part1) + md5(part2) + ...>))-<number of parts> - original_checksum_hex, _number_of_parts_str, part_size_str = checksum[10:].rsplit("-") + original_checksum_hex, _number_of_parts_str, part_size_str = checksum[ + 10: + ].rsplit("-") part_size = int(part_size_str) storage = current_files_rest.storage_factory(fileinstance=file_instance) with storage.open("rb") as f: object_checksum = hashlib.md5()
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/jsonresolver/contrib/jsonschema.py#L31
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/jsonresolver/contrib/jsonschema.py#L31
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/invenio_records/resolver.py#L14
jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/invenio_records/resolver.py#L14
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/invenio_records/resolver.py#L14
jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/invenio_records/resolver.py#L14
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/marshmallow_utils/fields/generated.py#L12
The '__version_info__' attribute is deprecated and will be removed in in a future version. Use feature detection or 'packaging.Version(importlib.metadata.version("marshmallow")).release' instead.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/marshmallow_utils/fields/generated.py#L12
The '__version_info__' attribute is deprecated and will be removed in in a future version. Use feature detection or 'packaging.Version(importlib.metadata.version("marshmallow")).release' instead.
Tests / Tests (3.9, postgresql14, opensearch2): invenio_records_resources/services/files/schema.py#L133
The 'default' argument to fields is deprecated. Use 'dump_default' instead.
Tests / Tests (3.9, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.9.21/x64/lib/python3.9/site-packages/marshmallow/fields.py#L1218
The 'missing' argument to fields is deprecated. Use 'load_default' instead.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/jsonresolver/contrib/jsonschema.py#L31
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/jsonresolver/contrib/jsonschema.py#L31
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/invenio_records/resolver.py#L14
jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/invenio_records/resolver.py#L14
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/invenio_records/resolver.py#L14
jsonschema.exceptions.RefResolutionError is deprecated as of version 4.18.0. If you wish to catch potential reference resolution errors, directly catch referencing.exceptions.Unresolvable.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/invenio_records/resolver.py#L14
jsonschema.RefResolver is deprecated as of v4.18.0, in favor of the https://github.com/python-jsonschema/referencing library, which provides more compliant referencing behavior as well as more flexible APIs for customization. A future release will remove RefResolver. Please file a feature request (on referencing) if you are missing an API for the kind of customization you need.
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/fs/__init__.py#L4
Deprecated call to `pkg_resources.declare_namespace('fs')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/fs/opener/__init__.py#L6
Deprecated call to `pkg_resources.declare_namespace('fs.opener')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages
Tests / Tests (3.12, postgresql14, opensearch2): opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/pkg_resources/__init__.py#L2559
Deprecated call to `pkg_resources.declare_namespace('fs')`. Implementing implicit namespace packages (as specified in PEP 420) is preferred to `pkg_resources.declare_namespace`. See https://setuptools.pypa.io/en/latest/references/keywords.html#keyword-namespace-packages