feat(encoding): improve hex conversion performance

[reallesee]

Replace string concatenation in toHex() with Array.from().map().join() pattern for better performance with large arrays. This approach avoids creating intermediate string objects during concatenation, resulting in better memory usage and execution speed, especially when processing large byte arrays

[webmaster128]

Sounds good. I just wonder if we can avoid creating the Array<number> in the first line which would be quite some overhead if not optimized away.

Would it be an option to do something like data.values().map((byte) => ...)? Do you have tests to measure the performance?

[reallesee]

You're right about the Array.from() overhead - thank you.

I could use the spread syntax instead:

export function toHex(data: Uint8Array): string {
  return [...data].map((byte) => byte.toString(16).padStart(2, "0")).join("");
}

Or for maximum efficiency, reduce without intermediate arrays:

export function toHex(data: Uint8Array): string {
  return Array.prototype.reduce.call(
    data, 
    (str, byte) => str + byte.toString(16).padStart(2, "0"), 
    ""
  );
}

I don't have benchmarks yet, but I can run some tests if you'd like to see which approach performs best with different data sizes. What do you think?

[webmaster128]

Getting some benchmarks in place would be very helpful to ensure we don't accidentally slow things down.

[...data] also creates a Array<number>, doesn't it? The reduce implementation creates all those intermediate strings we had before in the str accumulator, so I guess this is not better than the current implementation.

[dynst]

Would it be an option to do something like data.values().map((byte) => ...)?

With a simple polyfill it would be: (I was mistaken, this example can't work)

if (!Iterator.prototype.map) {
  Iterator.prototype.map = function* (f) {
    for (const value of this) {
      yield f(value);
    }
  };
}

[reallesee]

Implemented both suggestions - using data.values().map() with the polyfill for Iterator.prototype.map
Thanks!

[dynst]

Sorry, I didn't think that one through. The global class Iterator was standardized as the same time Iterator.prototype.map was so of course it doesn't exist to be assigned to.

You just need a helper function for map(data.values(), byte => ...). Something like...

function* map<T, U>(iterable: Iterable<T>, fn: (value: T) => U): IterableIterator<U> {
  for (const value of iterable) {
    yield fn(value);
  }
}

[reallesee]

what tool do you use for formatting?

[dynst]

Tests pass locally, only whitespace changes needed by eslint (and those can be done automatically post-merge)

[reallesee]

done!
thanks for the suggestion

[webmaster128]

Implementation looks nice now. We have the Array<string> with 2-char string elements in order to do the joining but no accumulator string anymore.

However, we are still missing benchmarks that show if this change is useful.

What was to motination for touching this function?

[dynst]

Benchmarks would make sense, we don't know what magic optimizations the v8 javascript engine is doing under the hood that might make .join() pointless, or if it's gotten even better since 2020. (+= was only about half the speed of .join() in Chrome.)

https://josephmate.github.io/java/javascript/stringbuilder/2020/07/27/javascript-does-not-need-stringbuilder.html