EC write path with ISA-L — anyone have numbers?

[greenwich]

Hi all,

We're evaluating ISA-L for EC writes through the S3 gateway on 2.2.1 (rs-3-2-1024k). We couldn't find any published Ozone numbers comparing the native coder with the Java coder. Does anyone have some to share?

What we measured on a small test cluster (2 s3g, 5 DNs, freon s3kg with a mix of 4 MiB PUTs and 32 MiB MPU parts, ~1.1 GB/s, 16g heap):

Java coder	ISA-L (libhadoop 3.4.3 + isa-l 2.32)
s3g CPU seconds per GiB written	7.80
s3g RSS under load	~12–15 GiB

JFR confirms the native coder is used: RSUtil.encodeData (~78% of Java samples with the Java coder) is replaced by NativeRSRawEncoder.encodeImpl.

Two questions:

1. Heap buffers fed to the native encoder. ECKeyOutputStream.ECChunkBuffers.allocateBuffers() takes heap buffers (byteBufferPool.getBuffer(false, cellSize), still the same on master). The native encoder prefers direct buffers, so RawErasureEncoder.encode() converts. On the native path, ByteArrayEncodingState.convertToByteBufferState() then calls ByteBuffer.allocateDirect() for every input and output cell of every stripe, copies in, and copies the parity back out. At ~1 GB/s that's on the order of 2 GB/s of fresh direct allocation plus copies. We think this is both what eats most of the ISA-L gain and what causes the off-heap growth, since direct memory is only reclaimed by GC up to MaxDirectMemorySize, which defaults to the heap size. Is this known? Would allocating direct buffers when encoder.preferDirectBuffer() is true be the right fix? The reconstruction input streams also call getBuffer(false, …).
2. End-to-end numbers. Has anyone measured ISA-L vs Java for EC writes (client or s3g CPU per byte, throughput) or for reconstruction/degraded reads on a real cluster?

[100rootrain]

Your reading of the encode path matches master (c15b66e). One thing to know before trying the fix you suggested: switching allocateBuffers() to getBuffer(true, …) on its own will break writes, because two places in ECKeyOutputStream assume heap buffers:

• handleOutputStreamWrite() passes buffer.array() to writeToOutputStream(). On a direct buffer that throws UnsupportedOperationException, which the catch turns into markStreamAsFailed(e), so every stripe would fail.
• padBufferToLimit() does Arrays.fill(buf.array(), …) for the last partial stripe. ECBlockReconstructedStripeInputStream.zeroFill() already has a hasArray() / put((byte) 0) fallback that could be reused there.

The datanode-facing side seems to be in place already: ECBlockOutputStream.write(ByteBuffer) exists and does ChunkBuffer.wrap(buff), the same thing write(byte[], …) does after ByteBuffer.wrap. So the change could be: decide direct = encoder.preferDirectBuffer() once, pass it to ECChunkBuffers, and add a ByteBuffer path through ECBlockOutputStreamEntry that keeps the currentChunkRspFuture / updateWrittenDataLength bookkeeping the byte[] path does.

Two side effects:

1. Pool cap. BoundedElasticByteBufferPool has a single cap for heap and direct buffers together, and ozone.client.elastic.byte.buffer.pool.max.size defaults to 16GB. With heap buffers, -Xmx already bounds that. With direct buffers, whatever the pool keeps is never freed, so after a burst the s3g could stay at a high RSS. The cap probably needs to come down, or get a separate direct limit, once the pool holds direct buffers.
2. Reads. Decode is harder than encode. In ECBlockReconstructedStripeInputStream the caller's buffers go straight into the decoder input/output arrays (see the class javadoc), and the coders don't allow mixing heap and direct. So making allocateInternalBuffer() direct isn't enough when the caller passes heap buffers. Datanode-side ECReconstructionCoordinator also uses getBuffer(false, …) but owns all of its buffers, so it looks like the easier second target.

On the +10 GiB: before putting all of it on unreclaimed DirectByteBuffers, compare java.nio:type=BufferPool,name=direct (MemoryUsed) with RSS, or run with -XX:NativeMemoryTracking=summary and jcmd <pid> VM.native_memory summary. If direct memory in use is small while RSS stays high, part of it is glibc keeping freed 1 MiB blocks in its arenas, and MALLOC_ARENA_MAX matters as much as the buffer change.

I don't have cluster numbers for your second question, sorry.

(Drafted with help from Claude; code references checked against master c15b66e.)