[Bug] be内存持续增加，直至无响应

[xzjslc]

Search before asking

• I had searched in the issues and found no similar issues.

Version

3.1.4

What's Wrong?

be节点运行一段时间发现内存不足，重启后恢复正常，一天左右再次内存不足。服务器负载也都是在一个较低水平。
查看日志发现UntrackedMemory占了90%以上的内存。
导出jeheap如下图，butil ResourcePool add_block在持续增加，且无回落趋势。怀疑存在内存泄露问题

What You Expected?

内存可被正常回收

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[xzjslc]

导出jeheap时并未在峰值，中间也导出过几次，butil ResourcePool add_block这部分的占比是一直在提升的状态

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_a501243ab2e4
This content is generated by AI for reference only.

Initial triage for maintainers:

The attached heap graph is actionable, but it does not prove a classic unreleased-pointer leak by itself. The largest visible branch is about 3.0 GB under butil::ResourcePool::add_block out of a 5.49 GB focused heap. The visible caller chain is:

AsyncResultWriter::process_block -> VTabletWriterV2::open -> _open_streams/_open_streams_to_backend -> LoadStreamStubs::open -> LoadStreamStub::open -> brpc::Channel::CallMethod -> bthread_timer_add -> TimerThread::Bucket::schedule -> butil::ResourcePool::add_block.

For Doris 3.1.4, this maps to the load stream open path. VTabletWriterV2::open() opens load streams to destination BEs, and LoadStreamStub::open() creates a brpc stream and calls open_load_stream with open_load_stream_timeout_ms (default 60000 ms). brpc schedules an RPC timeout task via bthread_timer_add. In brpc 1.4.0, canceled timer tasks are recycled by the timer thread later, and butil::ResourcePool retains allocated blocks as a high-water cache rather than returning them to the OS immediately.

So the current evidence is most consistent with a large brpc timer-task/resource-pool high-water mark driven by load stream opens. This can still be a real production problem because it is charged as UntrackedMemory and can keep RSS high, but the screenshot alone is not enough to distinguish:

1. high load-stream open rate or many concurrent loads/backends/streams with a 60s timeout causing a large timer-task high-water mark;
2. repeated failed/slow open_load_stream RPCs causing many timeout tasks to remain pending until the timer thread catches up;
3. an actual lifecycle bug where failed/canceled load stream opens are not being completed/closed normally.

I checked the 3.1.4 load-stream code path and did not see an obvious permanent LoadStreamMapPool retention in the normal close/cancel paths: successful close calls release() and the last sink erases the load stream map, and cancel also cancels streams and releases the map. That makes the brpc timer path the first place to verify rather than assuming scanner block memory is leaking.

Missing information needed to confirm the root cause:

• Raw heap profiles from at least two timestamps, preferably shortly after restart and when UntrackedMemory is high, not only the rendered image.
• BE memory tracker or memory overview output at the same timestamps, plus BE RSS.
• BE logs covering the growth window, especially lines containing open load stream, failed to open stream, Failed to connect to backend, releasing streams, closing olap table sink, and canceled olap table sink, with the related load_ids.
• Load workload shape: load type, load QPS, concurrent load count, number of BEs, tablet/replica scale, load_stream_per_node, open_load_stream_timeout_ms, and whether failures/retries happen during the period.
• brpc/bthread vars if available, especially bthread_timer_*, collected while memory is growing and after load traffic stops.
• Whether memory plateaus after stopping all load traffic for longer than open_load_stream_timeout_ms, or continues to grow with no load activity.

Recommended next step:

Try to reproduce with continuous load traffic while collecting repeated heap profiles and the matching load logs. If the ResourcePool::add_block memory grows with load-stream open rate and then plateaus after traffic stops, this is likely brpc timer/resource-pool high-water retention; reducing load concurrency, load_stream_per_node, or carefully lowering open_load_stream_timeout_ms can be used as a diagnostic mitigation. If it keeps increasing after load traffic stops and after the timeout window has passed, the suspicious area is the load-stream open/cancel/error lifecycle around LoadStreamStub::open() and brpc Controller::EndRPC timer cleanup.

[xzjslc]

堆信息对应的指标信息

metrics.txt

堆信息对应的内存信息

profile.txt