The value of used_memory_rss_human is significantly greater than that of used_memory_human and it is increasing continuosly and not resetting.

[vijul]

Even latest versions are behaving same and this usage is continuosly increasing in my scenario around 2.3Mb per minute. Can you please look into it and confirm if there is any parameter which will help us recover the memory because memory in use by keys are less and keys are expiring but memory is not being freed.
Hashing is using way too much memory here.
127.0.0.1:6379> info

Server
redis_version:7.4.0
dragonfly_version:df-v1.32.0
redis_mode:standalone
arch_bits:64
process_id:1
os:Linux 5.14.0-427.37.1.el9_4.x86_64 x86_64
thread_count:1
multiplexing_api:epoll
tcp_port:6379
uptime_in_seconds:1599559
uptime_in_days:18
hz:100
executable:dragonfly
config_file:

Clients
connected_clients:235
max_clients:64000
client_read_buffer_bytes:1281792
blocked_clients:0
pipeline_queue_length:0
send_delay_ms:0
timeout_disconnects:0

Memory
used_memory:2327045016
used_memory_human:2.17GiB
used_memory_peak:5324299280
used_memory_peak_human:4.96GiB
fibers_stack_vms:12841280
fibers_count:314
used_memory_rss:2649477120
used_memory_rss_human:2.47GiB
used_memory_peak_rss:5324299280
maxmemory:7516192768
maxmemory_human:7.00GiB
used_memory_lua:847760
object_used_memory:2322730768
type_used_memory_string:433584
type_used_memory_set:12464
type_used_memory_zset:334959600
type_used_memory_hash:1987325120
table_used_memory:4042944
prime_capacity:80640
expire_capacity:33600
num_entries:7539
inline_keys:11
small_string_bytes:246000
pipeline_cache_bytes:0
dispatch_queue_bytes:0
dispatch_queue_subscriber_bytes:0
dispatch_queue_peak_bytes:21672
client_read_buffer_peak_bytes:2641664
tls_bytes:6184
snapshot_serialization_bytes:0
commands_squashing_replies_bytes:18446744073655919026
psync_buffer_size:1024
psync_buffer_bytes:340154
cache_mode:store
maxmemory_policy:noeviction
replication_streaming_buffer_bytes:0
replication_full_sync_buffer_bytes:0

Stats
total_connections_received:337260
total_handshakes_started:0
total_handshakes_completed:0
total_commands_processed:1808463611
instantaneous_ops_per_sec:1365
total_pipelined_commands:113196854
pipeline_throttle_total:0
pipelined_latency_usec:35632534329
total_net_input_bytes:426470769312
connection_migrations:0
connection_recv_provided_calls:0
total_net_output_bytes:11491731755399
rdb_save_usec:0
rdb_save_count:0
big_value_preemptions:146
compressed_blobs:7
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:5536203
evicted_keys:0
total_heartbeat_expired_keys:4952275
total_heartbeat_expired_bytes:245888816
total_heartbeat_expired_calls:818750842
hard_evictions:0
garbage_checked:7615
garbage_collected:53
bump_ups:0
stash_unloaded:704
oom_rejections:0
traverse_ttl_sec:0
delete_ttl_sec:0
keyspace_hits:1233540553
keyspace_misses:94758769
keyspace_mutations:174958592
total_reads_processed:1196207406
total_writes_processed:1185217062
huffenc_attempt_total:0
huffenc_success_total:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
reply_count:1185217062
reply_latency_usec:24789351
blocked_on_interpreter:0
lua_interpreter_cnt:10
lua_blocked_total:585141
lua_interpreter_return:127164065
lua_force_gc_calls:0
lua_gc_freed_memory_total:0
lua_gc_duration_total_sec:0

Replication
role:master
connected_slaves:1
slave0:ip=10.233.157.47,port=6379,state=online,lag=28
master_replid:08af91f8741e11b03b2471013cb3adc2ed4ac46e

Modules
module:name=ReJSON,ver=20000,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20000,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]

Keyspace
db0:keys=0,expires=0,avg_ttl=-1
db12:keys=7377,expires=7375,avg_ttl=-1
db22:keys=68,expires=2,avg_ttl=-1
db26:keys=10,expires=2,avg_ttl=-1
db27:keys=47,expires=47,avg_ttl=-1
db114:keys=2,expires=2,avg_ttl=-1
db118:keys=2,expires=0,avg_ttl=-1
db155:keys=33,expires=27,avg_ttl=-1

Cpu
used_cpu_sys:67391.719379
used_cpu_user:21758.509033
used_cpu_sys_children:0.8139
used_cpu_user_children:0.263
used_cpu_sys_main_thread:67391.594745
used_cpu_user_main_thread:21758.478270

Cluster
cluster_enabled:0
migration_errors_total:0
total_migrated_keys:0

Latencystats
127.0.0.1:6379> exit

date
Mon Dec 8 10:44:47 UTC 2025

[romange]

300MB discrepancy between the RSS usage and used memory is not unreasonable.
RSS usage includes all the non-data related allocations - connections, pipelining buffers, reply buffers etc.

[vijul]

Main concern here is keys are expiring soon but memory is not being freed as hashing is keep on growing and i did not find any parameter which will reset it.

[eaglesssss]

dragonfly info

Server

redis_version:7.4.0
dragonfly_version:df-v1.35.1
redis_mode:standalone
arch_bits:64
process_id:7
os:Linux 5.15.126-1-pve x86_64
thread_count:96
multiplexing_api:iouring
tcp_port:6379
uptime_in_seconds:85209
uptime_in_days:0
hz:100
executable:dragonfly
config_file:

Clients

connected_clients:6
max_clients:64000
client_read_buffer_bytes:132096
blocked_clients:0
pipeline_queue_length:0
send_delay_ms:0
timeout_disconnects:0

Memory

used_memory:36847632
used_memory_human:35.14MiB
used_memory_peak:37634064
used_memory_peak_human:35.89MiB
fibers_stack_vms:44564384
fibers_count:683
used_memory_rss:7094706176
used_memory_rss_human:6.61GiB
used_memory_peak_rss:37634064
maxmemory:1419425959116
maxmemory_human:1.29TiB
used_memory_lua:30211168
object_used_memory:7348752
type_used_memory_string:7340032
type_used_memory_hash:224
type_used_memory_key:48
type_used_memory_stream:8448
table_used_memory:27069696
prime_capacity:645120
expire_capacity:80640
num_entries:4
inline_keys:2
small_string_bytes:48
pipeline_cache_bytes:0
dispatch_queue_bytes:0
dispatch_queue_subscriber_bytes:0
dispatch_queue_peak_bytes:0
client_read_buffer_peak_bytes:132096
tls_bytes:71416
snapshot_serialization_bytes:0
commands_squashing_replies_bytes:0
psync_buffer_size:0
psync_buffer_bytes:0
cache_mode:store
maxmemory_policy:noeviction
replication_streaming_buffer_bytes:0
replication_full_sync_buffer_bytes:0

Stats

total_connections_received:8243
total_handshakes_started:0
total_handshakes_completed:0
total_commands_processed:33002
instantaneous_ops_per_sec:0
total_pipelined_commands:14
pipeline_throttle_total:0
pipelined_latency_usec:7237
total_net_input_bytes:3853518376
connection_migrations:0
connection_recv_provided_calls:0
total_net_output_bytes:7733815273
rdb_save_usec:0
rdb_save_count:0
big_value_preemptions:0
compressed_blobs:0
instantaneous_input_kbps:-1
instantaneous_output_kbps:-1
rejected_connections:-1
expired_keys:0
evicted_keys:0
total_heartbeat_expired_keys:0
total_heartbeat_expired_bytes:0
total_heartbeat_expired_calls:0
hard_evictions:0
garbage_checked:0
garbage_collected:0
bump_ups:0
stash_unloaded:0
oom_rejections:0
traverse_ttl_sec:0
delete_ttl_sec:0
keyspace_hits:11315
keyspace_misses:0
keyspace_mutations:4264
total_reads_processed:89530
total_writes_processed:31034
huffenc_attempt_total:0
huffenc_success_total:0
defrag_attempt_total:0
defrag_realloc_total:0
defrag_task_invocation_total:0
reply_count:31034
reply_latency_usec:14382570
blocked_on_interpreter:0
lua_interpreter_cnt:2
lua_blocked_total:0
lua_interpreter_return:712
lua_force_gc_calls:471
lua_gc_freed_memory_total:3402508288
lua_gc_duration_total_sec:0.433144

Replication

role:master
connected_slaves:0
master_replid:2dcd89617a1076c94b4a7f1125b011447d207448

Modules

module:name=ReJSON,ver=20000,api=1,filters=0,usedby=[search],using=[],options=[handle-io-errors]
module:name=search,ver=20000,api=1,filters=0,usedby=[],using=[ReJSON],options=[handle-io-errors]

Keyspace

db0:keys=4,expires=0,hits=11315,misses=0,hit_ratio=100.00,avg_ttl=-1

Cpu

used_cpu_sys:9847.832912
used_cpu_user:9855.913221
used_cpu_sys_children:0.2845
used_cpu_user_children:0.2527
used_cpu_sys_main_thread:96.630712

used_cpu_user_main_thread:102.490199

Cluster

cluster_enabled:0
migration_errors_total:0
total_migrated_keys:0

Latencystats

We got similar issue after upgrading dragonfly image version from 1.27.1 to 1.35.1
Before upgrading mean usage was around 300-500MB depending on machine.
After upgrading, as can be seen on the screenshot, it went up from starting 150MB to 6GB, tho it started to decrease on one of the machines few hours ago.

Both instances are used the same way:
We store single compressed protobuf string with max size around 8MB, which gets updated every 2 minutes. Each instance has 2 clients that every minute check for new version and fetch it if its new. We also store short stream of versions and hset with metadata. Updates are done via lua script calls.

Memory increase seems to follow the pattern of increasing every 2 minutes by ~11MB.

[abustany]

Checking in, did you have a chance to catch up?

What I observe in our case is an instance with a tiny dataset (25MB) on a server with maxmemory set to 400MB. Almost all operations involve running Lua scripts. At some point, used_memory_rss climbs up slowly all the way to maxmemory, at which point the server starts returning "out of memory" errors when running Lua scripts. Looking at info memory, psync_buffer_bytes is almost 350MB. I wonder why the buffer grows that much?

[vijul]

Thanks @eaglesssss and @romange for actively looking into it.

[abustany]

I'm now running into this very often, both with 1.31 and 1.33. My dataset is tiny and yet:

# Memory
used_memory:10817608
used_memory_human:10.32MiB
used_memory_peak:10818520
used_memory_peak_human:10.32MiB
fibers_stack_vms:4356720
fibers_count:111
used_memory_rss:526307328
used_memory_rss_human:501.93MiB
used_memory_peak_rss:528908288
maxmemory:419430400
maxmemory_human:400.00MiB

I tried setting the environment variables above, here's more debug info:

memory arena show

mimalloc: arena 0: 32 blocks of size 32MiB (in 1 fields)
mimalloc:   inuse blocks:
mimalloc:     xxxxxxxxxxxxxxxxx...............
mimalloc:     total ('x'): 17
mimalloc:   committed blocks:
mimalloc:     xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
mimalloc:     total ('x'): 32
mimalloc:   abandoned blocks:
mimalloc:     ................................
mimalloc:     total ('x'): 0
mimalloc:   purgeable blocks:
mimalloc:     ................................
mimalloc:     total ('x'): 0
mimalloc: total inuse blocks    : 17
mimalloc: total abandoned blocks: 0
mimalloc: total purgeable blocks: 0

memory malloc-stats

___ Begin malloc stats ___
arena: 9543680, ordblks: 39, smblks: 0
hblks: 0, hblkhd: 0, usmblks: 0
fsmblks: 0, uordblks: 4538768, fordblks: 5004912, keepcost: 3648
___ End malloc stats ___

___ Begin mimalloc stats ___
heap stats:     peak       total       freed     current        unit       count
  reserved:     1.0 GiB     1.0 GiB     0           1.0 GiB
 committed:     1.0 GiB     1.0 GiB     1.6 GiB  -640.9 MiB                          ok
     reset:     0
    purged:     1.6 GiB
   touched:    64.2 KiB     2.4 MiB     4.2 GiB    -4.2 GiB                          ok
  segments:    17          39          22          17                                not all freed
-abandoned:     0           0           0           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           5.3 Ki     -5.3 Ki                           ok
-abandoned:     0           0           0           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     1
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:     1.0 Ki
   threads:     1           1           0           1                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:   598.971 s
   process: user: 4.788 s, system: 2.361 s, faults: 0, rss: 504.1 MiB, commit: 1.0 GiB
___ End mimalloc stats ___

Command line:

      --alsologtostderr
      --break_replication_on_master_restart=true
      --primary_port_http_enabled=false
      --admin_port=9999
      --admin_nopass
      --logtostderr
      --maxmemory
      400MiB
      --conn_io_threads
      1
      --proactor_threads
      1
      --epoll_file_threads
      1
      --break_replication_on_master_restart
      true
      --no_tls_on_admin_port
      --tls
      --tls_cert_file=/etc/dragonfly-tls/tls.crt
      --tls_key_file=/etc/dragonfly-tls/tls.key
      --tls_ca_cert_file=/etc/dragonfly/tls/ca.crt

The workload is mostly Lua scripts processing relatively "large" objects (~3MB) using cjson from/into a hash. The server goes into OOM mode in a matter of minutes.

[abustany]

The same seems to happen with 1.36.0 as well

[abustany]

And 1.27 is much more stable in terms of memory use. I'm happy to provide more info if needed to help debug this.