Fix memory leak bulk indexer

[mhmtszr]

Bulk indexer makes a lot of heap allocation, it affect our applications' performance. I tried to reduce allocations by using "sync.pool".

Bulkindexers that we regularly open and close cause allocation.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[cla-checker-service]

💚 CLA has been signed

[T-J-L]

What's the reason for opening/closing indexers? It can last the lifetime of the application, when doing so with a large buffer there are zero allocations here.

[mhmtszr]

@T-J-L hello, is there any way to use same indexers for different operations? we needed to close after adding to batch per operation.

[T-J-L]

@T-J-L hello, is there any way to use same indexers for different operations? we needed to close after adding to batch per operation.

I create a single indexer at start up, with a low flush time (100ms). Then for each application request create a couple of channels for success/errors, perform BulkIndexer.Add then write back to the channels in the BulkIndexerItem.OnSuccess and BulkIndexerItem.OnFailure callbacks. So effectivly each request is syncronous, with all requests to ES are batched.

You can set the Index and Action per item, so this works for all types of operation.

[mhmtszr]

@T-J-L hello, is there any way to use same indexers for different operations? we needed to close after adding to batch per operation.

I create a single indexer at start up, with a low flush time (100ms). Then for each application request create a couple of channels for success/errors, perform BulkIndexer.Add then write back to the channels in the BulkIndexerItem.OnSuccess and BulkIndexerItem.OnFailure callbacks. So effectivly each request is syncronous, with all requests to ES are batched.

You can set the Index and Action per item, so this works for all types of operation.

Great solution, but how can you be sure your documents will be written to Elasticsearch? We need to be sure that our documents will be written to Elasticsearch thus we are closing the bulk indexer.