Create a simple tool that tells me whether SIMD is working in my environment

[mikemccand]

Accessing SIMD instructions is devilishly complex from the clouds of javaland:

• Panama Vector API: One can use Panama's vector API, but that is quite abstract and might not do the right thing on your particular CPU and java version. And it remained experimental until Java 22 (maybe?), but Lucene devs got impatient and began integrating it while it was still experimental, and the original "Plan B" issue ~2 years ago.

• Hotspot compiler's auto-vectorization: One can write the java sources "just so" and hope Hotspot's autovectorization kicks in (Lucene's postings decode of blocks of 128 integers does this).

• Native extensions: Finally, one can also take matters into their own hands and create a native extension, using JNI or foreign functions support (also in Panama) to call C code from java and explicitly embed the right SIMD instructions in your C code (or rely on gcc to properly compile your native extension) -- this PR takes that approach, but it's draft/experimental.

These optimizations are highly CPU arch dependent -- x86-64 (Intel, AMD) are usually the focus (SSE/AVX) since developers mostly build and run on these CPUs, and Arm (e.g. Amazon's Graviton CPUs, Apples MX CPUs) doesn't often get the attention it should (though it does get some -- thank you @rmuir!). It has a different SIMD instruction set (Neon).

There are problems and frustrations with all of the above paths, and Lucene to varying degrees in various releases and on certain JDKs, tries to do the right thing. But even the right thing is brittle -- is Hotspot still auto-vectorizing correctly after my Java upgrade? Is Panama vector extensions really using SIMD instructions, or emulating them in slower java code? Have new SIMD instructions arrived (e.g. AVX-512) which could do our dot products faster?

Anyways, with all of that complicated backdrop... on to this actual issue:

Let's create a simple tool that one could run on their production environment, that reports whether or not your world (Java version, Lucene version, CPU arch, OS, virtualization layers, ...) is in fact using the right SIMD instructions when running Lucene's KNN search. It's OK if this tool is kinda slow to run -- I see it as something like an integration test. Ideas:

• Make a Python wrapper that runs knnPerfTest, but turns on -XX:+PrintAssembly, and then watches the output: when Hotspot says "I compiled dotProduct and here is the ASM code", grep that output for specific instructions. You'd first have to enumerate which instructions you would expect to see in this case (float or quantized vectors? which CPU arch?)

• Or ... are there performance counters that we could ask the CPU to report on (perf or so?). Then run knnPerfTest, and gather those counters, and see if "enough" SIMD instructions were executed?

• Or ... maybe from a single JVM invocation, there is some way to introspect (java reflection APIs maybe...) and see the ASM for functions and just look "yourself" from within the JVM that is running KnnGraphTester?

• Other ideas....?

We badly need this tooling because the whole situation is so fraught. Ideally this tool would be available from Lucene, and even more ideally Lucene itself could introspect and print a warning that it is not optimized properly for the current Java/CPU/OS. But we can start with a rough first cut here and maybe iterate it to something more compact.

These optimizations are crazy powerful, but they are also insanely brittle, so we need to give Lucene users stronger testing that the opto is still working...

[mikemccand]

Here's a timely Lucene issue showing how trappy these SIMD optimizations are!

[rmuir]

It has a different SIMD instruction set (Neon).

The newer amazon instances actually support SVE which is fantastic. This one gets even less attention, because Apple STILL does not support it, even on the M4 chips.

[rmuir]

The best idea I had for this general issue was https://github.com/apache/lucene/tree/main/dev-tools/aws-jmh

Maybe we need to tweak it / bring it up to speed (e.g. it should support java 24 and main). I know it also needs some SSH tweaks for folks that don't have great latency to us-east-1 as well.

[uschindler]

Let's create a simple tool that one could run on their production environment, that reports whether or not your world (Java version, Lucene version, CPU arch, OS, virtualization layers, ...) is in fact using the right SIMD instructions when running Lucene's KNN search. It's OK if this tool is kinda slow to run -- I see it as something like an integration test. Ideas:

For production users that's already solved. When your start the JVM and it uses vector code, Lucene prints a warning stating that it has disabled vectorization when not working.

You can make a simple main of it, by just doing a fake calculation using VectorUtil.

[uschindler]

Users of Elasticserach/Opensearch/Solr should see the warning (unless the startup/bootstrap disables logging of warnings). For Solr this is not the case; OS/ES supressed warnings in the past, although I think this is a bad idea.

[uschindler]

We don't have a static final boolean constant in VectorUtil, but that should be easy to do. Although it's not relevant for production users, the warning /info message on startup is more important.

On startup it also prints an info message which features are enabled and what your platform's bitSize is.

[uschindler]

E.g. for AVX2 enabled:

Juli 07, 2025 7:17:02 PM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
INFORMATION: Java vector incubator API enabled; uses preferredBitSize=256

if you have for example C2 compiler disabled (that happens when you run tests):

Juli 07, 2025 7:15:22 PM org.apache.lucene.internal.vectorization.VectorizationProvider lookup
WARNUNG: C2 compiler is disabled; Java vector incubator API can't be enabled

[uschindler]

For developers on Lucene that want to implement new vectorized code, of course some tool like described above could be useful. Just not for endusers. Tha problem is solved since early beginning.

[mikemccand]

The best idea I had for this general issue was https://github.com/apache/lucene/tree/main/dev-tools/aws-jmh

I absolutely love this tooling! It's under appreciated and under used!

(It lets you run a JMH microbenchmark on N different AWS instances -- it should be standard whenever we are tweaking SIMD stuff, or any other low level optos, since you can quickly test across different CPU arch/revisions)

[mikemccand]

It has a different SIMD instruction set (Neon).

The newer amazon instances actually support SVE which is fantastic. This one gets even less attention, because Apple STILL does not support it, even on the M4 chips.

Whoa, I didn't know that! I wonder which Graviton revision added SVE ... according to Gemini, it's Graviton 3.

[mikemccand]

For developers on Lucene that want to implement new vectorized code, of course some tool like described above could be useful. Just not for endusers. Tha problem is solved since early beginning.

Thanks @uschindler! I love that Java vector incubator message we get on startup ...

But ... I am somehow still wanting a deeper end-to-end test. This print from Lucene tells us whether Panama vector API is accessible in the current setup, and which preferredBitSize it's using (and maybe other fun details?), but I don't trust Panama itself to actually use SIMD vs fallback to the slow emulation that's even slower than if we wrote our own Java code to do the thing. So i'm hoping for a tester that somehow says whether SIMD is even being emitted in the ASM / executed by the CPU.

I.e. Lucene's message states how effectively Lucene thinks it's using vector API (or not), but I also want to know / multiply into that, whether and how well vector API is in turn actually using SIMD and the right SIMD.

Like it freaks me out that bypassing vector API and going straight to gcc compiled native extension seems to show so much performance gain. Is it because vector API was emulating on Graviton2? Or it chose a poor bit width? Or ... something?

(Sorry, I know only just enough to be dangerous about this SIMD situation).

[rmuir]

@mikemccand the last time i looked at that one, it was unclear how the copying from heap<->native space was working and if the benchie was fair.

separately, yes, you are gonna see huge gains for anything on the 128-bit NEON. My strong advice: stop worrying about the Neon: the macs, stop worrying about the graviton 1 and 2s. The vectorization is hacky and we do our best but it is not good. Instead spend time on SVE which is the future (and the now, if you choose graviton 3)

[rmuir]

These instructions may help you: https://github.com/apache/lucene/blob/main/help/jmh.txt

My best advice is to run our benchmarks, if you really care, run with perfasm and look and see what instructions get used.

[mikemccand]

++ I like the perfasm idea -- I think that's what #423 is using actually! It looks compelling!