Description
While investigating a separate OpenCloud memory-growth issue, I discovered another recurring failure mode which appears to be independent of memory exhaustion.
The whole container experiences extremely high I/O pressure for an extended period.
During these events OpenCloud itself is using very little memory.
The event can eventually cause OpenCloud internal services, NATS communication and monitoring requests to time out.
This behaviour has now been captured multiple times.
Environment
OpenCloud runs in Docker inside a Proxmox LXC container.
Host:
Proxmox
kernel 6.8.12-2-pve
OpenCloud has been tested with multiple versions.
One captured occurrence was with:
opencloudeu/opencloud:7.2.4
The same low-memory I/O-stall behaviour has subsequently been reproduced after upgrading OpenCloud, including with OpenCloud 7.5.0.
This makes it unlikely that the problem is specific to 7.2.4.
Important distinction from another memory issue
I also have another failure mode where OpenCloud grows to approximately 6 GiB anonymous RSS.
This issue is NOT about that condition.
During the I/O stall described here, memory usage is completely normal.
For example:
OpenCloud RSS: ~83 MB
OpenCloud RssAnon: ~81 MB
entire LXC memory: ~298 MB
TCP connections: 1
while simultaneously:
Therefore memory exhaustion cannot explain this particular event.
Example captured on 2026-09-06
At 04:15:
ct_io=100.00
ct_mem=298303488
oc_rss_kb=82960
oc_anon_kb=81296
tcp=1
LISTEN=1
At the same time:
Five seconds later:
ct_io=100.00
dockge_io=98.71
ct_mem=302305280
oc_rss_kb=82960
oc_anon_kb=81296
tcp=1
This state continued for many minutes.
Examples:
04:16:54
ct_io=100.00
dockge_io=94.81
OpenCloud RSS ~82 MB
04:17:50
ct_io=100.00
dockge_io=95.44
OpenCloud RSS ~82 MB
04:19:43
ct_io=100.00
dockge_io=96.54
OpenCloud RSS ~82 MB
04:20:03
ct_io=100.00
dockge_io=95.56
OpenCloud RSS ~83 MB
There was essentially no network load:
So this does not look like a request storm.
Previous reproduction on 2026-09-03
A very similar event occurred several days earlier.
Example:
04:20:04
ct_io=99.99
oc_io=93.26
dockge_io=98.39
ct_mem=351735808
OpenCloud RSS=119988 kB
OpenCloud anon=119860 kB
tcp=1
LISTEN=1
The values remained close to:
ct_io ≈ 100%
oc_io ≈ 85–97%
dockge_io ≈ 85–99%
for an extended period.
Again, OpenCloud memory was only approximately 120 MB.
NATS failures during the I/O stall
During the September 3 occurrence, OpenCloud later logged:
write tcp 127.0.0.1:<port>->127.0.0.1:9233: i/o timeout
nats: stale connection
Disconnected from NATS
context deadline exceeded
Shortly afterwards it reconnected.
We also observed a postprocessing error similar to:
failed to get consumer
context deadline exceeded
The NATS connection involved localhost:
127.0.0.1 -> 127.0.0.1:9233
Therefore the reverse proxy or external network cannot explain that timeout.
The NATS error also appears relatively late in the event.
The I/O stall had already existed before the NATS timeout occurred.
I therefore currently consider the NATS timeout a consequence of the system-wide stall rather than proof that NATS initiated it.
Other processes are affected
During the event, I/O pressure is not limited to the OpenCloud process.
For example, another Docker workload in the same LXC shows very high I/O wait:
This is important because cgroup I/O PSI represents time tasks are stalled waiting for I/O; it does not necessarily mean that Dockge itself generated the I/O.
It suggests that multiple workloads inside the LXC are blocked by a shared storage/I/O bottleneck.
Host scheduling investigation
I checked for scheduled host tasks around the recurring early-morning window.
No obvious Proxmox job matched the start of the event.
Checked items included:
- systemd timers
/etc/crontab- cron jobs
- ZFS scrub/trim schedules
- fstrim
- Proxmox backup jobs
No scheduled task was found around the relevant ~04:00–04:30 period that explains the behaviour.
There were also no obvious kernel/ZFS/I/O errors in the host journal during the investigated window.
Monitoring services also stall
Inside the LXC, simple monitoring services start timing out while this is happening.
For example:
opencloud-resource-monitor.service:
start operation timed out
Other small periodic jobs also take unusually long to complete.
Once the I/O pressure disappears, those jobs immediately return to normal.
This suggests they are victims of the I/O blockade rather than its cause.
Interesting read behaviour
During these events the OpenCloud process read_bytes counter continues increasing.
However, process/cgroup accounting during the most severe stalls is difficult to interpret because many processes appear to accumulate physical reads.
During another captured I/O storm we observed very large read_bytes deltas attributed to:
- opencloud
- dockerd
- containerd
- Dockge/node
- systemd
- other processes
I therefore do not want to claim that OpenCloud itself is necessarily generating all of the physical I/O.
It may involve page faults, reclaim, container filesystem behaviour or the underlying storage layer.
Why I am reporting this here
Because multiple processes inside the LXC are affected, I cannot currently prove that OpenCloud is the root cause.
However:
- the problem has repeatedly affected an OpenCloud installation,
- OpenCloud performs continuous reads during the affected period,
- embedded NATS communication eventually times out,
- postprocessing/consumer operations have failed during the stall,
- the event has been reproduced on different OpenCloud versions,
- there is no corresponding memory exhaustion or network request storm.
I would therefore appreciate help determining whether OpenCloud has any scheduled/background operation that could correlate with these early-morning I/O events.
Questions
Are there OpenCloud background tasks that could periodically perform significant storage activity, for example:
- search indexing/reindexing
- Bleve maintenance
- NATS/JetStream maintenance
- eventhistory
- postprocessing
- cleanup jobs
- metadata/database maintenance
- scheduled scans
Is there a supported way to enable detailed tracing for these components so that the next occurrence can be correlated with a specific OpenCloud service?
I have persistent 5-second monitoring of:
container I/O PSI
OpenCloud I/O PSI
memory usage
OpenCloud RSS/RssAnon
process read_bytes
TCP states
port 9200 queues
and can collect additional diagnostics during the next occurrence if developers can suggest which OpenCloud-specific metrics or debug endpoints would be most useful.
Description
While investigating a separate OpenCloud memory-growth issue, I discovered another recurring failure mode which appears to be independent of memory exhaustion.
The whole container experiences extremely high I/O pressure for an extended period.
During these events OpenCloud itself is using very little memory.
The event can eventually cause OpenCloud internal services, NATS communication and monitoring requests to time out.
This behaviour has now been captured multiple times.
Environment
OpenCloud runs in Docker inside a Proxmox LXC container.
Host:
OpenCloud has been tested with multiple versions.
One captured occurrence was with:
The same low-memory I/O-stall behaviour has subsequently been reproduced after upgrading OpenCloud, including with OpenCloud 7.5.0.
This makes it unlikely that the problem is specific to 7.2.4.
Important distinction from another memory issue
I also have another failure mode where OpenCloud grows to approximately 6 GiB anonymous RSS.
This issue is NOT about that condition.
During the I/O stall described here, memory usage is completely normal.
For example:
while simultaneously:
Therefore memory exhaustion cannot explain this particular event.
Example captured on 2026-09-06
At 04:15:
At the same time:
Five seconds later:
This state continued for many minutes.
Examples:
There was essentially no network load:
So this does not look like a request storm.
Previous reproduction on 2026-09-03
A very similar event occurred several days earlier.
Example:
The values remained close to:
for an extended period.
Again, OpenCloud memory was only approximately 120 MB.
NATS failures during the I/O stall
During the September 3 occurrence, OpenCloud later logged:
Shortly afterwards it reconnected.
We also observed a postprocessing error similar to:
The NATS connection involved localhost:
Therefore the reverse proxy or external network cannot explain that timeout.
The NATS error also appears relatively late in the event.
The I/O stall had already existed before the NATS timeout occurred.
I therefore currently consider the NATS timeout a consequence of the system-wide stall rather than proof that NATS initiated it.
Other processes are affected
During the event, I/O pressure is not limited to the OpenCloud process.
For example, another Docker workload in the same LXC shows very high I/O wait:
This is important because cgroup I/O PSI represents time tasks are stalled waiting for I/O; it does not necessarily mean that Dockge itself generated the I/O.
It suggests that multiple workloads inside the LXC are blocked by a shared storage/I/O bottleneck.
Host scheduling investigation
I checked for scheduled host tasks around the recurring early-morning window.
No obvious Proxmox job matched the start of the event.
Checked items included:
/etc/crontabNo scheduled task was found around the relevant ~04:00–04:30 period that explains the behaviour.
There were also no obvious kernel/ZFS/I/O errors in the host journal during the investigated window.
Monitoring services also stall
Inside the LXC, simple monitoring services start timing out while this is happening.
For example:
Other small periodic jobs also take unusually long to complete.
Once the I/O pressure disappears, those jobs immediately return to normal.
This suggests they are victims of the I/O blockade rather than its cause.
Interesting read behaviour
During these events the OpenCloud process
read_bytescounter continues increasing.However, process/cgroup accounting during the most severe stalls is difficult to interpret because many processes appear to accumulate physical reads.
During another captured I/O storm we observed very large
read_bytesdeltas attributed to:I therefore do not want to claim that OpenCloud itself is necessarily generating all of the physical I/O.
It may involve page faults, reclaim, container filesystem behaviour or the underlying storage layer.
Why I am reporting this here
Because multiple processes inside the LXC are affected, I cannot currently prove that OpenCloud is the root cause.
However:
I would therefore appreciate help determining whether OpenCloud has any scheduled/background operation that could correlate with these early-morning I/O events.
Questions
Are there OpenCloud background tasks that could periodically perform significant storage activity, for example:
Is there a supported way to enable detailed tracing for these components so that the next occurrence can be correlated with a specific OpenCloud service?
I have persistent 5-second monitoring of:
and can collect additional diagnostics during the next occurrence if developers can suggest which OpenCloud-specific metrics or debug endpoints would be most useful.