[dvc][compat] Add blob transfer support for DaVinciRecordTransformer #1471
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jingy-li
reviewed
Jan 24, 2025
internal/venice-common/src/main/resources/avro/PartitionState/v14/PartitionState.avsc
Outdated
Show resolved
Hide resolved
sixpluszero
reviewed
Jan 24, 2025
| @@ -23,5 +23,3 @@ _site/ | |||
| Gemfile.lock | |||
| .bundles_cache | |||
| docs/vendor/ | |||
| clients/da-vinci-client/classHash*.txt | |||
There was a problem hiding this comment.
Why do we remove this? classHash should not be included.
There was a problem hiding this comment.
This file will no longer get generated as the classHash is now stored in the offset record.
internal/venice-common/src/main/resources/avro/PartitionState/v14/PartitionState.avsc
Outdated
Show resolved
Hide resolved
internal/venice-common/src/main/resources/avro/PartitionState/v14/PartitionState.avsc
Outdated
Show resolved
Hide resolved
kvargha
changed the title
jingy-li
reviewed
Jan 24, 2025
| String snapshotPath = RocksDBUtils.composeSnapshotDir(dvcPath1 + "/rocksdb", storeName + "_v1", i); | ||
| Assert.assertTrue(Files.exists(Paths.get(snapshotPath))); | ||
| } | ||
|
|
There was a problem hiding this comment.
Do we need to verify that the RecordTransformerClassHash info in the offset record is correct?
There was a problem hiding this comment.
It will be difficult to do that in an integration test as I don't see a good way to access the storageEngine to fetch the offsetRecord. But I did add an extremely trivial validation in a unit test.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[dvc][compat] Add blob transfer support for DaVinciRecordTransformer
This PR makes DVRT compatible with blob transfer. Here are the changes made:
classHashin theoffsetRecordso it can be transported during blob transfer. This is to ensure that the blob that was transferred is compatible with the current transformer logic.classHashpersistence as it is now stored in theoffsetRecord.classHashwas being calculated based on the wrapper class not the user's class.How was this PR tested?
Add unit and integration tests.
Does this PR introduce any user-facing changes?
DVRT is no longer compatible with database checksum verification, but this is fine since DVRT is not currently being used.