OpenCloud process grows to ~6 GB anonymous RSS, hits container memory limit and becomes unresponsive

[Surfer2010]

Description

I am seeing recurring severe memory growth of the OpenCloud server process.

Over time, /usr/bin/opencloud server grows from a normal memory footprint of roughly 100–300 MB to approximately 6 GB RSS.

Almost the entire RSS consists of anonymous memory (RssAnon).

Once this happens, the container reaches its configured 6 GiB memory limit, memory and CPU pressure become extremely high, OpenCloud stops responding properly on port 9200, and eventually the instance has to be restarted.

This has happened repeatedly and is reproducible across multiple OpenCloud versions.

I have collected detailed 5-second monitoring data from outside the affected container.

---

Environment

OpenCloud is running in Docker inside a Proxmox LXC container.

Host:

• Proxmox
• Kernel: 6.8.12-2-pve
• LXC container memory limit: 6 GiB
• OpenCloud exposed internally on TCP port 9200

OpenCloud versions where the problem has been observed:

• 7.2.4
• newer versions were tested as well
• the installation has since been upgraded further while investigating the issue

The issue is not limited to the public reverse proxy path. We have also observed direct internal failures when connecting to the OpenCloud backend at 192.168.16.13:9200.

---

Normal state

After restarting OpenCloud, typical memory usage is approximately:

VmRSS:    ~100–250 MB
RssAnon:  ~80–120 MB

There are normally no CLOSE-WAIT connections on port 9200.

---

Failure state

During one captured failure:

memory.current = 6395109376
memory.max     = 6442450944

OpenCloud process:

RSS: ~6004348 kB

The OpenCloud process alone therefore consumes approximately 5.7 GiB RSS.

Almost all of this memory is anonymous memory.

Previous captures showed values such as:

VmRSS:   ~6000 MB
RssAnon: ~6000 MB

This is not page cache growth.

It appears to be anonymous userspace memory retained by the OpenCloud Go process.

---

cgroup pressure during the failure

Example:

memory.pressure:

some avg10=99.13 avg60=99.11 avg300=99.09
full avg10=80.84 avg60=80.87 avg300=80.66

CPU pressure:

some avg10=99.05 avg60=99.08 avg300=99.05
full avg10=5.80 avg60=5.60 avg300=5.70

I/O pressure:

some avg10=46.34 avg60=47.66 avg300=47.13

No OOM kill occurs:

memory.events:

low 0
high 803034594
max 0
oom 0
oom_kill 0
oom_group_kill 0

The system therefore appears to spend a very large amount of time under memory reclaim/pressure without actually killing OpenCloud.

---

Network behaviour once OpenCloud is stalled

Another repeat of this failure produced more than 3,300 TCP connections on port 9200:

tcp=3320
LISTEN=1
CLOSE-WAIT=3319

At the same time:

oc_rss_kb=6004348
oc_anon_kb=6004220
ct_mem≈6397 MB

The CLOSE-WAIT connections are mostly connections from the reverse proxy to OpenCloud.

Further investigation showed something interesting:

The OpenCloud process itself did NOT have thousands of corresponding socket file descriptors.

Instead, the OpenCloud port 9200 listener had a very large accept queue.

In a previous occurrence we captured:

LISTEN Recv-Q=1226 Send-Q=4096 *:9200

and approximately 1,186 port-9200 CLOSE-WAIT sockets.

Many of these entries had:

Recv-Q=330
inode=0

This suggests that the connections are waiting in the kernel accept queue while OpenCloud is no longer servicing accept() fast enough.

Therefore I currently consider the CLOSE-WAIT accumulation to be a consequence of OpenCloud starvation rather than the original trigger.

---

Important timeline from another captured incident

We have 5-second monitoring data showing that memory exhaustion happens BEFORE the major I/O storm.

Example:

11:36:58
OpenCloud RSS: ~5.59 GiB
CT memory:     ~6 GiB
CT I/O PSI:   0

11:38:20
OpenCloud RSS: ~5.72 GiB
OpenCloud anonymous memory: ~5.71 GiB
CT I/O PSI: 2.72

11:38:45
CT I/O PSI: 27.32

11:39:26
CT I/O PSI: 50.11

During the following minutes, enormous amounts of physical reads were observed while OpenCloud remained pinned near 6 GiB.

The sequence therefore appears to be:

OpenCloud anonymous memory growth
        ↓
OpenCloud reaches ~6 GiB
        ↓
LXC reaches memory limit
        ↓
memory reclaim / page fault / I/O storm
        ↓
OpenCloud becomes unable to service requests
        ↓
accept queue grows
        ↓
CLOSE-WAIT connections accumulate
        ↓
port 9200 eventually times out

---

Memory growth behaviour

The growth is not always gradual.

During earlier monitoring, OpenCloud RSS increased in discrete steps.

Examples:

~4.9 GB
→ ~5.0 GB
→ ~5.1 GB
→ ~5.25 GB

Some of these jumps happened without corresponding increases in the process read_bytes counter.

This makes the behaviour look like periodic/internal allocation rather than simply file cache growth.

---

Components considered during investigation

Possible areas we investigated include:

• search / Bleve indexing or reindexing
• eventhistory / postprocessing
• embedded NATS / JetStream
• reverse proxy behaviour
• desktop client requests
• mobile client requests
• WireGuard path
• Docker/container storage

The Bleve/index files we inspected were not unusually large.

Normal access through the reverse proxy did not immediately reproduce the memory growth.

Opening OpenCloud from LTE, WireGuard and a browser also did not immediately reproduce it.

The memory growth can occur while the number of TCP connections is very low.

Therefore the reverse proxy / CLOSE-WAIT behaviour does not currently look like the original cause of the memory allocation.

---

NATS observations

During some stalls OpenCloud has logged internal NATS errors such as:

write tcp 127.0.0.1:<port>->127.0.0.1:9233: i/o timeout
nats: stale connection
Disconnected from NATS
context deadline exceeded

We have also seen postprocessing consumer failures during these periods.

However, these messages often appear after the system is already under severe pressure, so it is unclear whether NATS is involved in the original trigger or is simply another victim of the stall.

---

What I would like help with

The main unresolved question is:

Which OpenCloud component could allocate and retain approximately 6 GiB of anonymous Go heap over time?

I would especially appreciate advice on how to capture useful diagnostics before restarting the instance.

For example:

• Is there a supported pprof/debug endpoint?
• Can heap profiling be enabled for the main OpenCloud process?
• Is there a way to obtain a Go heap profile from a running OpenCloud server?
• Which services/components would be the most useful to trace?
• Are there known issues involving search/Bleve, eventhistory, postprocessing, NATS/JetStream or another service that could explain this amount of anonymous memory?

The failure is intermittent, but I have persistent monitoring in place and can capture additional metrics during the next occurrence.

I can provide detailed 5-second cgroup/memory/I/O/TCP logs if useful.

[n-at-han-k]

This may be related to the search-service memory blowup fixed in #3485: on folder delete/move/restore/purge, the bleve backend expands a Path:<folder>/* wildcard into one term searcher per descendant, all live at once. On our instance a routine folder delete held 2.24 GB of a 2.29 GB live heap in that one query stack, and the observable state matches yours closely: almost pure anonymous memory, the process pinned at the cgroup limit under near-100% memory pressure with memory.events high climbing but oom_kill 0, and the server too starved to service requests (in our case even its own debug endpoints stopped answering).

One caveat before assuming it's the same bug: our trigger is a folder operation (delete, move, restore, purge — including trash emptying and sync clients deleting folders), producing a fast spike that then stays pinned; your description reads as gradual growth, and the 5-second capture starts after RSS is already ~5.6 GiB, so the onset isn't visible. A fast spike pinned against the limit looks identical after the fact.

Two ways to verify on your instance:

1. Heap profile during an incident — enable pprof on any one service's debug endpoint (one process serves all services), e.g.:

   SEARCH_DEBUG_PPROF=true
   SEARCH_DEBUG_ADDR=127.0.0.1:9224
   

   then while memory is high:

   go tool pprof -top http://127.0.0.1:9224/debug/pprof/heap
   

   If it's this bug, the top entries are vellum.(*FST).Reader, zapx.(*SegmentBase).dictionary and scorch.(*IndexSnapshot).TermFieldReader under WildcardQuery.Searcher / bleve.(*Batch).Delete. If the endpoint won't answer while stalled, briefly lift the memory limit (systemctl set-property --runtime <unit> MemoryHigh=infinity or the LXC equivalent) and retry.

2. Correlate onset — check whether each incident follows a folder delete/move/restore or trash emptying (server logs, or a sync client removing a directory tree). The spike size scales with the number of files under the folder.

A quicker but weaker signal: OC_EXCLUDE_RUN_SERVICES=search — if growth stops, it's search-side (though in #1269 one reporter still had issues with search disabled, so multiple causes may exist).

If the profile shows that stack, #3485 is the fix — it returns the same results with bounded memory (1194 MB → 102 MB peak for a 100k-file folder in the benchmark there).