Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pybind improvements and cleanups #2243

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

pybind improvements and cleanups #2243

wants to merge 1 commit into from

Conversation

jamesmunro
Copy link
Collaborator

@jamesmunro jamesmunro commented Mar 13, 2025 •
edited
Loading

Add tests for pybind interfaces using pybind11_stubgen. Remove ignore from docs builds.
The only functional changes is moving arcticdb_ext.version_store.SortedValue to arcticdb_ext.storage.SortedValue,

Errors

pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.DynamicFieldBuffer.__init__ : Invalid expression 'arcticdb::entity::TypeDescriptor'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.DynamicFieldBuffer.as_field : Invalid expression 'arcticdb::ColumnData'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.FieldDecodingResult.shape_buffer. : Invalid expression 'arcticdb::Buffer'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.FieldDecodingResult.values_buffer. : Invalid expression 'arcticdb::Buffer'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.FieldEncodingResult.buffer. : Invalid expression 'arcticdb::Buffer'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.Segment.header. : Invalid expression 'arcticdb::SegmentHeader'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.decode_segment : Invalid expression 'arcticdb::SegmentInMemory'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.encode_segment : Invalid expression 'arcticdb::SegmentInMemory'
pybind11_stubgen - [  ERROR] In arcticdb_ext.codec.encode_segment : Invalid expression 'arcticdb::EncodingVersion'
pybind11_stubgen - [  ERROR] In arcticdb_ext.column_store.ColumnData.type. : Invalid expression 'arcticdb::entity::TypeDescriptor'
pybind11_stubgen - [  ERROR] In arcticdb_ext.column_store.StringPool.as_buffer_info : Can't find/import 'buffer_info'
pybind11_stubgen - [  ERROR] In arcticdb_ext.metrics.prometheus.MetricsConfig.__init__ : Invalid expression 'arcticdb::MetricsConfig::Model'
pybind11_stubgen - [  ERROR] In arcticdb_ext.storage.LibraryManager.get_library : Invalid expression '<arcticdb_ext.storage.StorageOverride object at 0x7f289b4d3430>'
pybind11_stubgen - [  ERROR] In arcticdb_ext.storage.LibraryManager.get_library_config : Invalid expression '<arcticdb_ext.storage.StorageOverride object at 0x7f289b4c31f0>'
pybind11_stubgen - [  ERROR] In arcticdb_ext.storage.LibraryManager.write_library_config : Invalid expression '<arcticdb_ext.storage.StorageOverride object at 0x7f289b4d2e70>'
pybind11_stubgen - [  ERROR] In arcticdb_ext.storage.create_library_index : Invalid expression 'arcticdb::storage::LibraryIndex'
pybind11_stubgen - [  ERROR] In arcticdb_ext.storage.create_mem_config_resolver : Invalid expression 'arcticdb::storage::ConfigResolver'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.rollback_row : Invalid expression 'bm::bvector<bm::mem_alloc<bm::block_allocator'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.rollback_row : Invalid expression ':ptr_allocator'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.rollback_row : Invalid expression ':alloc_pool<bm::block_allocator'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.rollback_row : Invalid expression ':ptr_allocator> > >'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.row_builder. : Invalid expression 'arcticdb::stream::RowBuilder<arcticdb::stream::TimeseriesIndex, arcticdb::stream::FixedSchema, arcticdb::stream::Aggregator<arcticdb::stream::TimeseriesIndex, arcticdb::stream::FixedSchema, arcticdb::stream::RowCountSegmentPolicy, arcticdb::stream::DenseColumnPolicy> >'
pybind11_stubgen - [  ERROR] In arcticdb_ext.stream.FixedTimestampAggregator.start_row : Invalid expression 'arcticdb::stream::RowBuilder<arcticdb::stream::TimeseriesIndex, arcticdb::stream::FixedSchema, arcticdb::stream::Aggregator<arcticdb::stream::TimeseriesIndex, arcticdb::stream::FixedSchema, arcticdb::stream::RowCountSegmentPolicy, arcticdb::stream::DenseColumnPolicy> >'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.batch_key_exists : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.batch_key_exists : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.find_keys : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.find_keys : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.find_keys_for_id : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.find_keys_for_id : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.get_key_path : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.get_key_path : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.key_exists : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.key_exists : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.overwrite_append_data : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.overwrite_append_data : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.overwrite_segment_in_memory : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.overwrite_segment_in_memory : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_descriptor : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_descriptor : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_metadata : Invalid expression 'google::protobuf::Any'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_metadata : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_metadata : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_timeseries_descriptor : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_timeseries_descriptor : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_to_read_result : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_to_read_result : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_to_segment : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.read_to_segment : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.remove : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.remove : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.write : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.tools.LibraryTool.write : Invalid expression 'arcticdb::entity::RefKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.types.IndexDescriptor.__init__ : Invalid expression 'arcticdb::IndexDescriptor::Type'
pybind11_stubgen - [  ERROR] In arcticdb_ext.types.IndexDescriptor.kind : Invalid expression 'arcticdb::IndexDescriptor::Type'
pybind11_stubgen - [  ERROR] In arcticdb_ext.types.StreamDescriptor.sorted : Invalid expression 'arcticdb::SortedValue'
pybind11_stubgen - [  ERROR] In arcticdb_ext.types.TimeseriesDescriptor.next_key. : Invalid expression 'arcticdb::entity::AtomKeyImpl'
pybind11_stubgen - [  ERROR] In arcticdb_ext.types.TimeseriesDescriptor.sorted. : Invalid expression 'arcticdb::SortedValue'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.FilterClause.__init__ : Invalid expression 'arcticdb::ExpressionContext'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.FilterClause.__init__ : Invalid expression 'arcticdb::PipelineOptimisation'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.FilterClause.set_pipeline_optimisation : Invalid expression 'arcticdb::PipelineOptimisation'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.FrameSlice.col_range. : Invalid expression 'arcticdb::pipelines::ColRange'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.FrameSlice.row_range. : Invalid expression 'arcticdb::pipelines::RowRange'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.ProjectClause.__init__ : Invalid expression 'arcticdb::ExpressionContext'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.PythonVersionStore.list_incompletes : Invalid expression 'arcticdb::pipelines::SliceAndKey'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.ResampleClauseLeftClosed.__init__ : Invalid expression 'arcticdb::ResampleBoundary'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.ResampleClauseRightClosed.__init__ : Invalid expression 'arcticdb::ResampleBoundary'
pybind11_stubgen - [  ERROR] In arcticdb_ext.version_store.read_dataframe_from_file : Invalid expression 'arcticdb::pipelines::ReadQuery'
pybind11_stubgen - [WARNING] Raw C++ types/values were found in signatures extracted from docstrings.
Please check the corresponding sections of pybind11 documentation to avoid common mistakes in binding code:
 - https://pybind11.readthedocs.io/en/latest/advanced/misc.html#avoiding-cpp-types-in-docstrings
 - https://pybind11.readthedocs.io/en/latest/advanced/functions.html#default-arguments-revisited

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

@jamesmunro
Cleanup pybind declarations, reordering and moving as needed.
fd5d2f8 
Add tests for pybind interfaces using pybind11_stubgen.
Remove ignore from docs builds.
@jamesmunro jamesmunro added the minor Feature change, should increase minor version label Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexowens90 alexowens90 Awaiting requested review from alexowens90 alexowens90 will be requested when the pull request is marked ready for review alexowens90 is a code owner

@willdealtry willdealtry Awaiting requested review from willdealtry willdealtry will be requested when the pull request is marked ready for review willdealtry is a code owner

@poodlewars poodlewars Awaiting requested review from poodlewars poodlewars will be requested when the pull request is marked ready for review poodlewars is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees
No one assigned
Labels
minor Feature change, should increase minor version
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@jamesmunro