feat: add versioned job objects with syft-migration registry#9432
Merged
Conversation
Split JobState and JobSubmissionMetadata into versioned V1 MigratableObjects, register them in a package-local migration registry with a pinned protocol schema, and re-export current-version aliases from the models package.
… package models/ now has one folder per versioned object (job_state/, job_submission_metadata/) with v1.py files and current-version aliases in each folder's __init__. The migration machinery (registry, protocol schema) moves to syft_job/migrations/. Old syft_job.models.state / .config import paths are removed and all consumers updated.
syft-migration: new ProtocolSchema class listing all supported versions per canonical name, computed per protocol by MigrationRegistry.compute_protocol_schemas() and exported via MigrationService.export_protocol_schemas(). syft-job tests: simulate a test-scope V2 of JobState and JobSubmissionMetadata with up/down migrations, loading v1 from disk and upgrading, and downgrading a v2-only object.
… from registry objects The registry now derives the supported versions directly from its objects dict instead of walking package schemas, returns one ProtocolSchema for the package's protocol, and stamps it with the current package version. MigrationService.export_protocol_schemas is renamed to export_protocol_schema accordingly.
MigrationRegistry now takes protocol_name/package_name/package_version at construction. current_protocol_schema is a computed property pinning the latest registered version of each object, register_protocol_schema becomes register_historic_protocol_schema (history only, usually loaded from release-artifact files), and compute_protocol_schema reads identity from the registry so it no longer raises. syft-job's schema.py registration module is removed since the current schema is now derived.
MigrationError and the _identity/_has_identity helpers move to syft_migration/identity.py, breaking the registry<->schema import cycle so registry.py imports the schema classes at the top instead of inside methods. default_registry is removed: a concrete MigratableObject subclass must now receive registry= (or inherit it); defining one without a registry raises MigrationError.
PackageProtocolSchema now lists all supported versions per canonical name (supported_versions: dict[str, list[str]]) instead of pinning one version via object_versions, sharing a BaseVersionsSchema with ProtocolSchema. Both gain current_schema(canonical_name) returning the latest supported version, and migrate_to_schema targets that.
Replaces the hardcoded __version__ (which had drifted to 0.1.25 while pyproject was at 0.1.39) with importlib.metadata in a dedicated version.py; the job registry now uses the same value.
job_state -> JobState, job_submission_metadata -> JobSubmissionMetadata
…t helpers test_versioned_objects_registered_and_aliased now checks that a version exists rather than pinning '1'. New test imports every syft_job module, collects all MigratableObject subclasses defined in the package, and asserts each is registered in job_registry. Helpers renamed to create_mock_submission and _mock_submission_config_path.
tests/migrations/ now holds test_objects_registered.py (registry coverage incl. the package scan), test_serialization.py (save/load identity round trips and service loading), test_migrations.py (V2 upgrade/downgrade flows), and mocks.py with the shared mock submission helpers.
MigrationRegistry.has_upgradeable_path_to_latest(canonical_name, from_version) checks a migration path exists to the latest registered version (trivially true for a single version). Tested in syft-migration with mock registries, and syft-job gains test_upgrade_paths.py asserting every registered job object version can upgrade to its latest.
test_migrations.py now runs against job_registry itself: every registered version of JobState and JobSubmissionMetadata is written to disk, loaded, and upgraded to the latest version, and the latest version is downgraded to every registered version and round-tripped through disk. No-ops today with only V1, but automatically exercises real migration paths once newer versions are registered.
tests/migrations/fixtures/<canonical_name>/v<version>.yaml holds a full serialized sample of each registered object version. The real-registry tests load fixtures and upgrade to latest, and downgrade the latest fixture to every version, replacing the _instance_of reference-copy helper. A missing fixture for a registered version fails the test with a pointer to add one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
JobStateandJobSubmissionMetadatainto versionedJobStateV1/JobSubmissionMetadataV1classes built onsyft_migration.MigratableObjectsyft_job/models/<model_name>/— one folder per versioned object (v1.py+ current-version alias in__init__.py)syft_job/migrations/— migration machinery:registry.py(job_registry),schema.py(pinnedPackageProtocolSchema), and future migration functionsJobState/JobSubmissionMetadataremain as current-version aliases importable fromsyft_job.modelssyft_job.models.state/.models.configpaths removed; all consumers (syft-job, syft-bg, syft-enclave, root tests) updatedsyft-migrationas a syft-job dependencyTest plan
packages/syft-job/tests/test_job_migration.pyjust test-unit-fast(406 passed) anduv run pytest packages/syft-job/tests(9 passed)pre-commit run --all-filespasses