[feat] Add distributed cache trait, type transformation, serialization, compression, encryption by schgoo · Pull Request #346 · microsoft/oxidizer

schgoo · 2026-03-27T21:32:50Z

No description provided.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…chgoo/distributedcache

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…s/encrypt pipeline Switch the cachet crate's serialize, compress, and encrypt codecs from Vec<u8> to BytesView (from the bytesbuf workspace crate). This makes cloning in the fallback/promotion path O(1) (refcount bump) instead of O(n) (full byte copy). Changes: - Add bytesbuf dependency to cachet - Update BincodeEncoder/BincodeCodec to produce/consume BytesView - Update ZstdCodec to compress/decompress BytesView - Update AesGcmCodec to encrypt/decrypt BytesView - Update builder transform.rs serialize/compress/encrypt methods - Update serialize_pipeline example to use BytesView - Re-export BytesView from cachet::lib - Add Hash impl for BytesView in bytesbuf crate - Add bytesbuf::view::BytesView to allowed_external_types Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-03-27T21:39:42Z

⚠️ Breaking Changes Detected


--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.46.0/src/lints/enum_variant_added.ron

Failed in:
  variant CacheOp:Len in /home/runner/work/oxidizer/oxidizer/crates/cachet_tier/src/testing.rs:33

If the breaking changes are intentional then everything is fine - this message is merely informative.

Remember to apply a version number bump with the correct severity when publishing a version with breaking changes (1.x.x -> 2.x.x or 0.1.x -> 0.2.x).

geeknoid · 2026-04-06T17:25:33Z

+impl ZstdCodec {
+    /// Creates a new Zstd codec with the given compression level.
+    ///
+    /// Levels typically range from 1 (fastest) to 22 (best compression).


Should be a newtype constraining the value from 1 to 22.

Thanks for the feedback, @geeknoid! This draft PR is just a bytesbuf sanity check/reference and I will close it once I get the relevant code checked in in phases. You're definitely right on a lot of it. The bytesbuf portions were AI generated, and I haven't dug through them deeply yet. I am starting the work on the serialization feature today, so I'll incorporate your comments there.

The ultimate goal is zero-copy from serialization-I/O. Compression and Encryption do write new output, but maybe we can avoid full copy of the input.

geeknoid · 2026-04-06T17:26:33Z

+            dctx.reset(zstd::zstd_safe::ResetDirective::SessionOnly)
+                .map_err(|code| Error::from_message(format!("failed to reset zstd decompression context: error code {code}")))?;
+            let mut decoder = zstd::stream::read::Decoder::with_context(value.clone(), dctx);
+            let mut output = Vec::new();


This should be a BytesBuf, right? This would avoid temp allocations.

geeknoid · 2026-04-06T17:27:35Z

+            .cipher
+            .decrypt(nonce, ciphertext)
+            .map_err(|e| Error::from_message(format!("AES-GCM decryption failed: {e}")))?;
+        Ok(plaintext.into())


Does this do a full data copy?

I think the issue here is that some encryption algorithms require contiguous input, whereas BytesView/BytesBuf is multi-span. So basically we have two options here:

Add something like BytesView::as_contiguous() -> Cow<[u8]> that returns the slice if the BytesView is single-span, or copies when multi-span

Use something like https://github.com/RustCrypto/AEADs/tree/master/aead-stream to implement chunked encryption. This has security risks, and this particular crate is not very widely used and has no tests.

geeknoid · 2026-04-06T17:28:15Z

+
+impl<T: serde::Serialize + Send + Sync> Encoder<T, BytesView> for BincodeEncoder {
+    fn encode(&self, value: &T) -> Result<BytesView, Error> {
+        let bytes = bincode::serialize(value).map_err(Error::from_source)?;


Can we serialize into a BytesBuf to avoid an alloc?

geeknoid · 2026-04-06T17:29:09Z

+
+impl<T: serde::Serialize + serde::de::DeserializeOwned + Send + Sync> Codec<T, BytesView> for BincodeCodec {
+    fn decode(&self, bytes: &BytesView) -> Result<T, Error> {
+        bincode::deserialize(bytes.first_slice()).map_err(Error::from_source)


Is this right? Don't you need to go through all the slices?

geeknoid · 2026-04-06T17:29:46Z

+
+impl Encoder<BytesView, BytesView> for ZstdCodec {
+    fn encode(&self, value: &BytesView) -> Result<BytesView, Error> {
+        let compressed = zstd::encode_all(value.clone(), self.level).map_err(Error::from_source)?;


Can we decompress into a BytesBuf to avoid a copy?

geeknoid · 2026-04-06T17:31:17Z

+
+impl Codec<BytesView, BytesView> for AesGcmCodec {
+    fn decode(&self, value: &BytesView) -> Result<BytesView, Error> {
+        let slice = value.first_slice();


I think this isn't right, you need to go through all the slices, right?

schgoo and others added 17 commits March 25, 2026 06:39

Added MappingAdapter. Needs more work

9187184

Split builder.rs into builder module

a48c41a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Rename map module to transform

533c925

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Make TransformAdapter generic over codec types

53534f7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add TransformBuilder with phase markers

c9c8e88

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add serialize feature with BincodeEncoder/BincodeDecoder

a86ba3f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add compress and encrypt features with ZstdEncoder/AesGcmEncoder

772f53b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add serialize pipeline and type mapping examples

460ed7f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Simplify TransformBuilder to 6 generics with boxed codecs

2fb11f5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Merge branch 'main' of https://github.com/microsoft/oxidizer into u/s…

0453c0e

…chgoo/distributedcache

Refactor to Encoder/Codec trait split with bidirectional value codecs

342fd4d

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Reuse thread-local zstd DCtx for decompression

75ba1cc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add comprehensive transform integration tests

463e313

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Add tests for 100% line coverage on transform module

44ad47a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Catch all mutants in transform module

9934edc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Clean up async unit tests

9978f69

geeknoid reviewed Apr 6, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feat] Add distributed cache trait, type transformation, serialization, compression, encryption#346

[feat] Add distributed cache trait, type transformation, serialization, compression, encryption#346
schgoo wants to merge 17 commits intomainfrom
u/schgoo/distributedcache

schgoo commented Mar 27, 2026

Uh oh!

github-actions Bot commented Mar 27, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

schgoo Apr 8, 2026

Uh oh!

schgoo Apr 8, 2026 •

edited

Loading

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

schgoo Apr 8, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

geeknoid Apr 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

schgoo commented Mar 27, 2026

Uh oh!

github-actions Bot commented Mar 27, 2026

⚠️ Breaking Changes Detected

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

schgoo Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

schgoo Apr 8, 2026 •

edited

Loading