[lucene] Support vector index using Lucene

[jerry-024]

Purpose

Introduce vector index support for Paimon using Apache Lucene 9.x to enable:

• Approximate nearest neighbor (ANN) search for vector embeddings

Implementation

• Uses Lucene's HNSW algorithm for efficient ANN search
• Supports float32 and byte vectors
• Supports EUCLIDEAN, COSINE, DOT_PRODUCT, and MAXIMUM_INNER_PRODUCT similarity metrics

Tests

• VectorGlobalIndexTest

API and Format

Documentation

[JingsongLi]

Maybe paimon-lucene is better.

[kaori-seasons]

Current situation: serializeDirectory / deserializeDirectory only write the number of files, the length of each filename, the filename itself, and the file length, then directly write the bytes. There is no magic number, version number, or verification/checksum mechanism.

Impact: If the serialization structure is modified in the future, older versions will not be able to recognize it; it's difficult to pinpoint which part is corrupted if the stream is damaged; security/robustness is poor (prone to silent corruption).

Recommendation: Add a magic number, version number, and CRC32 for each file (or an overall checksum), and use standard DataOutputStream/DataInputStream to write int/long, avoiding manual ByteBuffer wrapping. This will be more robust and easier to maintain backward compatibility (using a version field).

examples:

private static final int MAGIC = 0x50414D4F; // 'PAMO' or any magic
private static final short VERSION = 1;

private void serializeDirectory(Directory directory, OutputStream out) throws IOException {
    DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(out));
    dos.writeInt(MAGIC);
    dos.writeShort(VERSION);

    String[] files = directory.listAll();
    dos.writeInt(files.length);

    for (String fileName : files) {
        byte[] nameBytes = fileName.getBytes(StandardCharsets.UTF_8);
        dos.writeInt(nameBytes.length);
        dos.write(nameBytes);

        long fileLength = directory.fileLength(fileName);
        dos.writeLong(fileLength);

        try (IndexInput input = directory.openInput(fileName, IOContext.DEFAULT)) {
            byte[] buffer = new byte[32 * 1024];
            long remaining = fileLength;
            CRC32 crc = new CRC32();
            while (remaining > 0) {
                int toRead = (int) Math.min(buffer.length, remaining);
                input.readBytes(buffer, 0, toRead);
                dos.write(buffer, 0, toRead);
                crc.update(buffer, 0, toRead);
                remaining -= toRead;
            }
            dos.writeLong(crc.getValue()); // per-file checksum
        }
    }
    dos.flush();
}

And the corresponding deserializeDirectory function (which reads the magic number/version and verifies the CRC).

private IndexMMapDirectory deserializeDirectory(SeekableInputStream in) throws IOException {
    DataInputStream dis = new DataInputStream(new BufferedInputStream(in));
    int magic = dis.readInt();
    if (magic != MAGIC) {
        throw new IOException("Invalid vector index file magic: " + Integer.toHexString(magic));
    }
    short version = dis.readShort();
    if (version != VERSION) {
        throw new IOException("Unsupported vector index version: " + version);
    }

    IndexMMapDirectory indexMMapDirectory = new IndexMMapDirectory();
    try {
        int numFiles = dis.readInt();
        byte[] buffer = new byte[BUFFER_SIZE];

        for (int i = 0; i < numFiles; i++) {
            int nameLength = dis.readInt();
            byte[] nameBytes = new byte[nameLength];
            dis.readFully(nameBytes);
            String fileName = new String(nameBytes, StandardCharsets.UTF_8);
            long fileLength = dis.readLong();

            long expectedCrc = -1L;
            // We need to stream file data to index output, and compute CRC while reading
            try (IndexOutput output = indexMMapDirectory.directory().createOutput(fileName, IOContext.DEFAULT)) {
                long remaining = fileLength;
                CRC32 crc = new CRC32();
                while (remaining > 0) {
                    int toRead = (int) Math.min(buffer.length, remaining);
                    dis.readFully(buffer, 0, toRead);
                    output.writeBytes(buffer, 0, toRead);
                    crc.update(buffer, 0, toRead);
                    remaining -= toRead;
                }
                expectedCrc = dis.readLong();
                if (crc.getValue() != expectedCrc) {
                    throw new IOException("CRC mismatch for file " + fileName);
                }
            }
        }
        return indexMMapDirectory;
    } catch (Exception e) {
        try {
            indexMMapDirectory.close();
        } catch (Exception ee) {
            // ignore or add suppressed
            e.addSuppressed(ee);
        }
        if (e instanceof IOException) {
            throw (IOException) e;
        } else {
            throw new IOException("Failed to deserialize directory", e);
        }
    }
}

[leaves12138]

@kaori-seasons Please do not copy all AI answer to the review comment, it will cost author a lot to understand a simple point of view. You should summarize them into one or two words, like "I think add a MAGIC number is better".

[JingsongLi]

+1 to @leaves12138 This is really a waste of everyone's time.

[JingsongLi]

Why not cover this in utitcase-jdk11?

[leaves12138]

Thanks for @jerry-024
Left some comments below

[leaves12138]

Please do not use java.nio.file.Files in class, import them in head.

[leaves12138]

ditto

[leaves12138]

ditto

[leaves12138]

We don't need to sort

[leaves12138]

ditto

[leaves12138]

lucene-hnw? "vector" is too wide.

[jerry-024]

how about lucene-hnsw? (ps: HNSW (Hierarchical Navigable Small World))

[leaves12138]

Do not throw Exception. Please return GlobalIndexResult.fromRange(<the range pass to you in GlobalIndexIOMeta>)

[JingsongLi]

1. All classes should start with Lucene.
2. Identifier should be lucene-vector-knn?

[JingsongLi]

+1