Enabling both cache_mode and SSD Data Tiering in dragonfly

[jahnavilatha589]

Can I enable both cache_mode and SSD Data Tiering..? As cache_mode evict keys if maxmemory is reached.

I enabled both cache_mode and SSD data tiering, but once the max memory limit reached, I encountered the following error:
" tiered_storage.cc:163] Should not reach here"

With cache_mode enabled, it should remove keys from memory, correct?

Shared some info and configuration details below.
`
### info memory
used_memory:270940752
used_memory_human:258.39MiB
used_memory_peak:286962880
used_memory_peak_human:273.67MiB
fibers_stack_vms:229264
fibers_count:7
used_memory_rss:340791296
used_memory_rss_human:325.00MiB
used_memory_peak_rss:340942848
maxmemory:272629760
maxmemory_human:260.00MiB
used_memory_lua:0
object_used_memory:2624
type_used_memory_string:2624
table_used_memory:266950352
num_buckets:495780
num_entries:6185616
inline_keys:6185616
listpack_blobs:0
listpack_bytes:0
small_string_bytes:2624
pipeline_cache_bytes:0
dispatch_queue_bytes:0
dispatch_queue_subscriber_bytes:0
dispatch_queue_peak_bytes:0
client_read_buffer_peak_bytes:5376
tls_bytes:0
snapshot_serialization_bytes:0
cache_mode:cache
maxmemory_policy:eviction

### info tiered
tiered_entries:6185575
tiered_entries_bytes:426804675
tiered_total_stashes:12896849
tiered_total_fetches:134168
tiered_total_cancels:0
tiered_total_deletes:0
tiered_total_uploads:0
tiered_total_stash_overflows:0
tiered_heap_buf_allocations:130621
tiered_registered_buf_allocations:261484
tiered_allocated_bytes:506957824
tiered_capacity_bytes:805306368
tiered_pending_read_cnt:0
tiered_pending_stash_cnt:0
tiered_small_bins_cnt:123769
tiered_small_bins_entries_cnt:6185575
tiered_small_bins_filling_bytes:3321
tiered_cold_storage_bytes:0
tiered_offloading_steps:3332329858
tiered_offloading_stashes:2930
tiered_ram_hits:0
tiered_ram_cool_hits:0
tiered_ram_misses:0

configurations -

--cache_mode=true
--log_dir=/abc/logs
--pidfile=/abc/hypermem/pid.txt
--aclfile=/abc/hypermem/aclfile.txt
--dir=/abc/storage/data
--version_check=false
--masteruser=repl
--masterauth=hypermemreplpwd
--tiered_prefix=/abc/storage/flash/flash
--proactor_threads=1

Linux version

"Debian GNU/Linux 11 (bullseye)"
`

[Athinivas]

@dranikpg @adiholden any possible ETA for the fix ?

PS: We are facing similar issues. Kindly suggest any workaround for same.

[adiholden]

I would say that a workaround right now would be not to run ssd tiering in cache_mode.
We will fix this but I dont have ETA for this

[romange]

@Athinivas can you provide "info all" stats when this happens again?

[diwu-sf]

I just tested cache_mode + ssd tiering, with max memory set to 768MB and used memtier_benchmark to try to fill it up and stress test, when I run save in redis-cli simultaneously, sometimes it would also segfault.

I feel like the combination of cache_mode + ssd tiering should be tested more.
The ideal behavior is that:

• keys are held in memory, 768MB can hold a lot of keys
• values (ours are embedding vectors that are roughly 2 to 4KB each) are held in SSD cache
• with cache_mode on, when both SSD is full, keys are evicted
• snapshotting every 30 mins should never segfault

The behavior that we're seeing is unstable snapshotting, and we get error trying to write keys in before the SSD is anywhere close to full

[romange]

Can you please expand on how do you stress test it? I really want to find these segfault bugs

[romange]

Specifically, can you provide the arguments to both dragonfly and memtier_benchmark?

[romange]

Closing as we have not got any additional info.