investigate jflex 1.9.x buffer size/expansion feature

[rmuir]

Description

#14583 only bumps the dependency and regenerates, but doesn't take advantage of the new features. I think we are currently taking care of this with skeleton files in gradle/generation/jflex. Using the builtin functionality might be cleaner.

[rlaehdals]

Hello, I’d like to start contributing to Lucene and this issue seems like a good opportunity. Since this is my first time contributing to open source, could you guide me on the first steps or assign me this task if possible?

[rmuir]

Read this file first: https://github.com/apache/lucene/blob/main/help/regeneration.txt
Play with the jflex regeneration, maybe tweak some of the input files to understand it.

Look at the #14583 and maybe dig deeper by looking at the actual commits/pull requests against jflex instead of just the summary. I think that if we use the new functionality we can configure the input files differently and remove the "skeletons" in https://github.com/apache/lucene/tree/main/gradle/regenerate/jflex

Please note: I am not 100% sure that we can remove these skeletons. But we should do the investigation to see if it is possible/reasonable.

[rlaehdals]

I have been looking into whether the skeleton.txt files can be removed.
Since JFlex already provides a built-in skeleton, it seems that skeleton.default.txt can safely be deleted.

The main issue is with skeleton.disable.buffer.expansion.txt. Because JFlex does not provide a built-in option to disable buffer expansion, the tests related to buffer size are currently failing.

As a temporary workaround, I modified the generated code in JFlexTask using regex replacements to remove the buffer expansion logic. However, I am not certain whether this is the appropriate solution. I would greatly appreciate any feedback or suggestions on a better approach.

configure(project(":lucene:core")) {
  task generateStandardTokenizerInternal(type: JFlexTask) {
    description = "Regenerate StandardTokenizerImpl.java"
    group = "generation"

    jflexFile = file('src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex')

    
    // NOTE: The following modifications in `doLast` are applied
    // after JFlex generates StandardTokenizerImpl.java.
    // These changes adjust buffer handling and error conditions.
    doLast {
      ant.replace(
          file: file('src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java'),
          encoding: "UTF-8",
          token: "private static final int ZZ_BUFFERSIZE =",
          value: "private int ZZ_BUFFERSIZE ="
          )

      def content = file('src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java').text
      content = content.replaceAll(
          /\/\* is the buffer big enough\? \*\/[\s\S]*?(?=\/\* fill the buffer with new input \*\/)/,
          ''
      )

      content = content.replaceAll(
          /int requested = zzBuffer\.length - zzEndRead;/,
          """int requested = zzBuffer.length - zzEndRead - zzFinalHighSurrogate;
    if (requested == 0) {
      return true;
    }"""
      )

      content = content.replaceAll(
          /if \(numRead == 0\) \{\s*if \(requested == 0\) \{[\s\S]*?\}\s*else \{[\s\S]*?\}\s*\}/,
          """if (numRead == 0) {
      throw new java.io.IOException(
          "Reader returned 0 characters. See JFlex examples/zero-reader for a workaround.");
    }"""
      )

      content = content.replaceAll(
          /if \(numRead == requested\) \{[\s\S]*?zzFinalHighSurrogate = 1;[\s\S]*?\}/,
          """if (numRead == requested) { // We requested too few chars to encode a full Unicode character
          --zzEndRead;
          zzFinalHighSurrogate = 1;
          if (numRead == 1) {
            return true;
          }
        }"""
      )
      file('src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java').text = content
    }
  }

  def generateStandardTokenizer = wrapWithPersistentChecksums(generateStandardTokenizerInternal, [
    andThenTasks: [
      "applyGoogleJavaFormat"
    ],
    mustRunBefore: ["compileJava"]
  ])

  regenerate.dependsOn generateStandardTokenizer
}

[rmuir]

Because JFlex does not provide a built-in option to disable buffer expansion, the tests related to buffer size are currently failing.

Did you look at jflex-de/jflex#197 ? It adds the new limit that I'm referring to here.

[rlaehdals]

I noticed that token_limit_size controls the input length, and I tried running the following test:

public void testMaxTokenLengthNonDefault() throws Exception {
  StandardAnalyzer a = new StandardAnalyzer();
  a.setMaxTokenLength(5);
  assertAnalyzesTo(a, "ab cd toolong xy z", new String[] {"ab", "cd", "toolo", "ng", "xy", "z"});
  a.close();
}

org.junit.ComparisonFailure: term 2 
requirement: toolo
actual: xy

When token_limit_size is not explicitly set, the issue I observed is that instead of producing "toolo", the analyzer outputs "xy" because the buffer is expanded.

However, if the token limit size is explicitly set, the behavior changes: instead of expanding the buffer, it throws
java.io.EOFException("Scan buffer limit reached [" + zzBuffer.length + "]");.

Is this the correct and expected behavior, or is it indicating a potential issue?

[rmuir]

don't confuse setMaxTokenLength with the new limit of %token_size_limit here. you would have to integrate this new limit to see what it is all about.

[rlaehdals]

Thank you for your clarification, and I apologize for any confusion. I really appreciate your guidance on this.

I have looked into the %token_size_limit option. If this option is not set, the behavior is as follows for two methods:

/** Returns the maximum size of the scanner buffer, which limits the size of tokens. */
private int zzMaxBufferLen() {
    return Integer.MAX_VALUE;
}


/** Whether the scanner buffer can grow to accommodate a larger token. */
private boolean zzCanGrow() {
    return true;
}

By modifying this value, you can control the maximum token size. Setting the limit to be equal to the buffer size prevents any further buffer expansion.

%token_size_limit ZZ_BUFFERSIZE

Then, the methods would become:

/** Returns the maximum size of the scanner buffer, which limits the size of tokens. */
private int zzMaxBufferLen() {
    return ZZ_BUFFERSIZE;
}

/** Whether the scanner buffer can grow to accommodate a larger token. */
private boolean zzCanGrow() {
    return zzBuffer.length < ZZ_BUFFERSIZE;
}

If the input token is larger than the buffer and the scanner attempts to expand, it cannot exceed the token limit, which may result in the following exceptions:

throw new java.io.EOFException("Scan buffer limit reached ["+zzBuffer.length+"]");

So in summary, while it is possible to completely remove the skeleton, it should be carefully considered since doing so would cause exceptions to be thrown.