improve: provide opts to set the truncate size in text mode to reduce memory cost #731

yuweizzz · 2025-02-10T09:59:11Z

should fix #718.

cli/cmd/root.go

user/module/imodule.go

… memory cost

cfc4n · 2025-02-11T14:24:11Z

pkg/event_processor/iworker.go

@@ -141,6 +141,11 @@ func (ew *eventWorker) writeEvent(e event.IEventStruct) {
 // 解析类型，输出
 func (ew *eventWorker) parserEvents() []byte {
 	ew.status = ProcessStateProcessing
+	tsize := int(ew.processor.truncateSize)
+	if tsize > 0 && ew.payload.Len() > tsize {
+		ew.payload.Truncate(tsize)


Is it just judging the size when parsing the first package? A more suitable solution should be when receiving each Event? For example

ecapture/pkg/event_processor/iworker.go

Line 138 in dcfc3cf

ew.payload.Write(e.Payload())

Is it the most reasonable solution to discard it directly when the payload is greater than tsize? Is it more appropriate to continue receiving after printing out?

dosubot bot added the improve label Feb 10, 2025

cfc4n requested changes Feb 10, 2025

View reviewed changes

cli/cmd/root.go Outdated Show resolved Hide resolved

user/module/imodule.go Outdated Show resolved Hide resolved

improve: provide opts to set the truncate size in text mode to reduce…

4f0321a

… memory cost

yuweizzz force-pushed the mem branch from 3bd2e08 to 4f0321a Compare February 11, 2025 02:15

cfc4n requested changes Feb 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

improve: provide opts to set the truncate size in text mode to reduce memory cost #731

improve: provide opts to set the truncate size in text mode to reduce memory cost #731

yuweizzz commented Feb 10, 2025

cfc4n Feb 11, 2025

improve: provide opts to set the truncate size in text mode to reduce memory cost #731

Are you sure you want to change the base?

improve: provide opts to set the truncate size in text mode to reduce memory cost #731

Conversation

yuweizzz commented Feb 10, 2025

cfc4n Feb 11, 2025

Choose a reason for hiding this comment