feat: add projection support to TapeDecoder for skipping unknown fields in json parsing (1.4x speedup)

[Weijun-H]

Which issue does this PR close?

• Closes #NNN.

Rationale for this change

What changes are included in this PR?

This PR implements projection-aware field skipping in the arrow-json reader:

1. New API: ReaderBuilder::with_projection(bool) enables opt-in field filtering
2. Skip optimization: When enabled, JSON fields not in schema are skipped during tape parsing rather than fully parsed and discarded later
3. Fail-fast for strict_mode: Unknown fields now error immediately during tape parsing instead of waiting until array decoding

Behavior matrix:

strict_mode	projection	Behavior
false	false	Parse all fields, ignore unknown (original)
false	true	Skip unknown fields at tape level
true	*	Error on first unknown field (fail-fast)

Are these changes tested?

Yes, all existing tests pass

Are there any user-facing changes?

Yes, new public API:

• ReaderBuilder::with_projection(bool) - opt-in to skip unknown JSON fields during parsing

This is additive and does not break existing behavior (default is false).

[scovich]

I really like the idea of skipping unwanted fields -- pure overhead to keep them -- but this PR feels overly complex/nested. I wonder if there's a "flatter" way to handle the situation?

[scovich]

Why not just use the hex constants directly, out of curiosity?

[scovich]

Also -- my intuition is that these two flags are only needed because SkipValue does too much. The newly introduced code has a lot of looping and nesting, where the existing enum variants are quite flat. The difference seems to be that the existing variants hand off to a new state whenever they detect a state change?

So e.g. instead of messing with flags, one might declare three new enum variants, SkipValue, SkipString and SkipEscape, where each nests exclusively inside the one before it? e.g. if the projection skipped field foo, then the following JSON fragment:

{ 
  "foo": {
    "bar": "hello\nworld!"
  }
}

would:

• push a SkipValue as soon as : detects that foo is not selected
• push a SkipString as soon as it hits the opening " of the string
• push a SkipEscape as soon as it hits the \ inside the string
• pop once the escape was processed
• pop once the closing " is found
• pop once the next field starts (or whatever is currently the ending condition for SkipValue)

[scovich]

By the same token, one would arguably want to push multiple SkipValue states instead of tracking nesting depth with a new variable? But then enum variants start to proliferate (basically need two of each).

Would it instead make sense to have a single skip offset that is the first stack index being skipped?
And then have pairs of match arms that decide what state gets pushed vs. merely traversed?

[Weijun-H]

Change to enum variant in a255860

[scovich]

Hmm, when I ran the benchmark on my laptop:

• Your original PR had 8% overhead (wide run) and 36% benefit (narrow run)
• The revised PR has overhead 9% and benefit 30%

[Weijun-H]

In order to remove the regression, I added a projection in ReaderBuilder to enable projection-aware parsing.

When enabled, JSON fields not present in the schema are skipped during tape parsing rather than being fully parsed and later ignored. This improves performance for narrow projections over wide JSON data.

[scovich]

Re projection -- Very slick!

Were you able to repro the slowdown going from a421d89 (original version) to a2aa758 (extra enum variants)? It's pretty consistent for me.

[Weijun-H]

Were you able to repro the slowdown going from a421d89 (original version) to a2aa758 (extra enum variants)? It's pretty consistent for me.

This is also consistent on my side

[scovich]

I threw an LLM at this whole situation during a boring meeting, and arrived at a surprisingly different potential approach, if you're game to try it out?

The short version is:

• Keep the existing (highly optimized and efficient) decoding logic, but factor it out to a helper method that is generic over const SKIP: bool that says whether to actually store the parsed output.
• Wrap that helper in decode and decode_skip methods, with a clean transition between the two: enter at the : match (like the PR does today), and decode_skip breaks back out to decode when the stack length drops back down.
• We need a new boolean skipping field to handle cases where input bytes were exhausted while skipping (so the next call to decode can jump straight to decode_skip when starting the next buffer of bytes)
• The state stack tracks everything related to skipping (small memory cost but very efficient).
• No new tape decoder enum variants needed.

In theory, the approach should be simpler (less duplicated source code) while also having friendlier branching (fewer and/or more predictable branches).

Is that something you'd want me to put a bit more time into exploring further?
Or something you'd prefer to dig into yourself?