Adding support for sharing memory between the module and the engine #1804

yairgott · 2025-03-01T08:42:51Z

Overview

Sharing memory between the module and engine reduces memory overhead by eliminating redundant copies of stored records in the module. This is particularly beneficial for search workloads that require indexing large volumes of documents.

Vectors

Vector similarity search requires storing large volumes of high-cardinality vectors. For example, a single vector with 512 dimensions consumes 2048 bytes, and typical workloads often involve millions of vectors. Due to the lack of a memory-sharing mechanism between the module and the engine, ValkeySearch currently doubles memory consumption when indexing vectors, significantly increasing operational costs. This limitation introduces adoption friction and reduces ValkeySearch's competitiveness.

Implementation Details

Memory Allocation Strategy

At a fundamental level, there are two primary allocation strategies:

[Chosen] Module-allocated memory shared with the engine.
Engine-allocated memory shared with the module.

For ValkeySearch, it is crucial that vectors reside in cache-aligned memory to maximize SIMD optimizations. Allowing the module to allocate memory provides greater flexibility for different use cases, though it introduces slightly higher implementation complexity.

Shared SDS

Shared SDS, a new data type, facilitates module-engine memory sharing with thread-safe intrusive reference counting. It preserves SDS semantics and structure while adding ref-counting and a free callback for statistics tracking.

A core component that enables thread-safe buffer sharing could be beneficial for use cases beyond modules. One notable advantage is avoiding deep copies of buffers when IO threading is enabled.

Module API

New Module APIs

VM_CreateSharedSDS:
- Creates a new Shared SDS.
- Accepts an allocation function for fine-grained control (e.g., cache alignment).
- Accepts a free callback function to track deallocations.
VM_SharedSDSPtrLen: Retrieves the raw buffer pointer and length of a Shared SDS.
VM_ReleaseSharedSDS: Decreases the Shared SDS ref-count by 1.

Extended Module APIs

VM_HashSet: Supports setting a shared SDS in the hash.
VM_HashGet: Retrieves a shared SDS from the hash and increments its ref-count by 1.

Engine Hash Data-Type

ValkeySearch indexes documents which reside in engine as t_hash data-type records. While JSON is also supported, it is out of scope for this discussion. The t_hash implementation is based on either list-pack for small datasets or hashtable for larger ones.

Since list-pack performs deep copies, it cannot support intrusive ref-counting semantics. As a result, if list-pack is used as the underline data-type while setting a shared SDS, .e.g. by calling VM_HashSet, it is converted to hashtable. Additionally, for the same reason, a shared SDS is never stored as embedded value in a hashtable entry.

yairgott · 2025-03-01T08:44:04Z

Planning to add unit tests soon. Publishing early to start the discussion rolling.

Sharing memory between the module and engine reduces memory overhead by eliminating redundant copies of stored entries in the module. This is particularly beneficial for search workloads that require indexing large volumes of stored data. Shared SDS, a new data type, facilitates module-engine memory sharing with thread-safe intrusive reference counting. It preserves SDS semantics and structure while adding ref-counting and a free callback for statistics tracking. New module APIs: - VM_CreateSharedSDS: Creates a new Shared SDS. - VM_SharedSDSPtrLen: Retrieves the raw buffer pointer and length of a Shared SDS. - VM_ReleaseSharedSDS: Decreases the Shared SDS ref-count by 1. Extended module APIs: - VM_HashSet: Now supports setting a shared SDS in the hash. - VM_HashGet: Retrieves a shared SDS and increments its ref-count by 1.

codecov · 2025-03-01T09:39:37Z

Codecov Report

Attention: Patch coverage is 22.11538% with 81 lines in your changes missing coverage. Please review.

Project coverage is 70.93%. Comparing base (3f6581b) to head (47a9487).
Report is 8 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/module.c	6.52%	43 Missing ⚠️
src/sds.c	25.64%	29 Missing ⚠️
src/sds.h	0.00%	5 Missing ⚠️
src/t_hash.c	69.23%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1804      +/-   ##
============================================
- Coverage     70.99%   70.93%   -0.06%     
============================================
  Files           123      123              
  Lines         65651    65749      +98     
============================================
+ Hits          46609    46642      +33     
- Misses        19042    19107      +65

Files with missing lines	Coverage Δ
src/server.h	`100.00% <ø> (ø)`
src/t_zset.c	`96.88% <100.00%> (+0.04%)`	⬆️
src/t_hash.c	`95.71% <69.23%> (-0.52%)`	⬇️
src/sds.h	`78.04% <0.00%> (-4.85%)`	⬇️
src/sds.c	`82.69% <25.64%> (-3.98%)`	⬇️
src/module.c	`9.60% <6.52%> (-0.01%)`	⬇️

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

madolson · 2025-03-03T15:26:28Z

Consensus is that we don't want to rush the implementation and commit to a specific API, after we are technically past the new API cutoff for the release. If we can converge offline about the design quickly, we'll merge it, otherwise we'll wait until 9.0.

yairgott · 2025-03-03T20:05:34Z

@zuiderkwast , @ranshid , happy discuss further with you!

It would be valuable for ValkeySearch 1.0 to have such interface in 8.1, after all there is no second chance to make a first time good impression ;).

Signed-off-by: yairgott <[email protected]>

ranshid

Some initial highlevel comments.

I think the main issue I have is that this new API implementation has some unclear limitations.
For example it can only support up sds32 headers (which is probably 99.99% O.K, but still...) and it will not be O.K to use in all cases were we might have to embed the field (eg keys, hash fields etc...)

I think all embedding flows will go though sdswrite, but we need a way to make sure we assert in all flows that might embed the sds.
Also we need to make sure we document this correctly for module users to understand these limitations as they are not trivial and can easily be changed following future core changes that might embed some fields for optimizations.

ranshid · 2025-03-06T07:55:33Z

src/sds.h

@@ -150,6 +164,9 @@ static inline size_t sdsavail(const_sds s) {
        SDS_HDR_VAR(64, s);
        return sh->alloc - sh->len;
    }
+    case SDS_TYPE_32_SHARED: {
+        return 0;


why is that?

ranshid · 2025-03-06T07:57:05Z

src/t_hash.c

@@ -97,7 +97,7 @@ hashTypeEntry *hashTypeCreateEntry(sds field, sds value) {
    size_t value_len = sdslen(value);
    size_t value_size = sdsReqSize(value_len, SDS_TYPE_8);
    sds embedded_field_sds;
-    if (field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE) {
+    if (sdsType(value) != SDS_TYPE_32_SHARED && field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE) {


nit

Suggested change

if (sdsType(value) != SDS_TYPE_32_SHARED && field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE) {

bool embed_value = sdsType(value) != SDS_TYPE_32_SHARED && field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE;

if (embed_value) {

Also theoretically field can also be a shared sds right? I know in your VSS implementation only values are, but in case we would like to generalize this new API we should at-least assert in case the field is shared.

ranshid · 2025-03-20T12:09:12Z

src/sds.c

+ * - `sh`: A pointer to the `sdshdrshared` structure whose reference count
+ *   should be increased.
+ */
+void sdsRetain(sdshdrshared *sh) {


Is this function accepting sdshdrshared meant in order to enforce type correctness by the compiler?
I wonder if it would not be better to allow this function to accept any sds type and just return in case the sds type is not shread.
this way we can avoid external applications errors by force casting wrong sds type and also using the SDS_HDR_VAR macro which AFAIK is mainly used internally in the sds implementation.

ranshid · 2025-03-20T12:10:28Z

src/sds.h

@@ -84,10 +96,11 @@ struct __attribute__((__packed__)) sdshdr64 {
 #define SDS_TYPE_16 2
 #define SDS_TYPE_32 3
 #define SDS_TYPE_64 4
+#define SDS_TYPE_32_SHARED 5


Why we use shared only up to 32 size? we could generalize better by supporting SDS_TYPE_64 and avoid this limitation

zuiderkwast

Not a full review. Just a few thoughts.

I acknowledge the need to share strings between module and core.
I think this shared string type should be called ValkeyModuleSharedString. There is nothing SDS-specific in the API so the term SDS has no meaning in the module perspective. SDS is an internal implementation detail.
This new type increases the complexity of the module API. We will have one more string type with its own set of functions. I imagine we want to allow shared strings for more things in the future, such as set elements, string values, sorted set elements, etc. and conversion between this and other kinds of strings, so the size and complexity of the API might grow significantly.
Internally in the core, we have been moving towards embedding SDS strings into other structures. Shared strings can't be embedded and this adds new complexity in the core.
Accessing strings without copying them has been discussed before, but more often about modules accessing values from the core. I imagine we can allow some read-only access as const char * in a way similar to ValkeyModule_StringDMA but for hash values, set elements, etc.

yairgott force-pushed the engine_module_shared_memory branch from 104a4dd to f9aad1a Compare March 1, 2025 09:24

PingXie requested review from zuiderkwast and ranshid March 3, 2025 19:54

yairgott closed this Mar 3, 2025

yairgott reopened this Mar 3, 2025

yairgott force-pushed the engine_module_shared_memory branch 6 times, most recently from dd9a9d2 to d601ba1 Compare March 5, 2025 08:13

adding TCL

Loading
Loading status checks…

47a9487

Signed-off-by: yairgott <[email protected]>

yairgott force-pushed the engine_module_shared_memory branch from d601ba1 to 47a9487 Compare March 5, 2025 19:51

ranshid reviewed Mar 20, 2025

View reviewed changes

zuiderkwast reviewed Mar 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding support for sharing memory between the module and the engine #1804

Adding support for sharing memory between the module and the engine #1804

yairgott commented Mar 1, 2025 •

edited

Loading

yairgott commented Mar 1, 2025 •

edited

Loading

codecov bot commented Mar 1, 2025 •

edited

Loading

madolson commented Mar 3, 2025

yairgott commented Mar 3, 2025 •

edited

Loading

ranshid left a comment

ranshid Mar 6, 2025

ranshid Mar 6, 2025

ranshid Mar 20, 2025

ranshid Mar 20, 2025

ranshid Mar 20, 2025

zuiderkwast left a comment

	if (sdsType(value) != SDS_TYPE_32_SHARED && field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE) {
	bool embed_value = sdsType(value) != SDS_TYPE_32_SHARED && field_size + value_size <= EMBED_VALUE_MAX_ALLOC_SIZE;
	if (embed_value) {

Adding support for sharing memory between the module and the engine #1804

Are you sure you want to change the base?

Adding support for sharing memory between the module and the engine #1804

Conversation

yairgott commented Mar 1, 2025 • edited Loading

Overview

Implementation Details

Memory Allocation Strategy

Shared SDS

Module API

Engine Hash Data-Type

yairgott commented Mar 1, 2025 • edited Loading

codecov bot commented Mar 1, 2025 • edited Loading

Codecov Report

madolson commented Mar 3, 2025

yairgott commented Mar 3, 2025 • edited Loading

ranshid left a comment

Choose a reason for hiding this comment

ranshid Mar 6, 2025

Choose a reason for hiding this comment

ranshid Mar 6, 2025

Choose a reason for hiding this comment

ranshid Mar 20, 2025

Choose a reason for hiding this comment

ranshid Mar 20, 2025

Choose a reason for hiding this comment

ranshid Mar 20, 2025

Choose a reason for hiding this comment

zuiderkwast left a comment

Choose a reason for hiding this comment

yairgott commented Mar 1, 2025 •

edited

Loading

yairgott commented Mar 1, 2025 •

edited

Loading

codecov bot commented Mar 1, 2025 •

edited

Loading

yairgott commented Mar 3, 2025 •

edited

Loading