Profiling support (like --profile) for validate

[channi23]

Hi, I am Hari!

While understanding Blaze and experimenting with the validation performance of the JSON Schema CLI (as it is built upon Blaze), I was using --benchmark to compare different schemas and understand how the validation cost changes as more keywords and structures are introduced.

The benchmark is useful for telling me how much time the complete validation takes, but it does not tell me what part of the validation is responsible for that time. So I had to create many small schemas and instances to isolate individual pieces of functionality and measure the difference between them.

Because of this, I thought we could have the CLI provide an option like:

jsonschema validate schema.json instance.json --profile

The idea would not necessarily be to implement a profiler inside the CLI. Instead, the option could provide a profiling-friendly execution path that works with external profilers such as Instruments on macOS or perf on Linux.

Conceptually, the distinction I’m thinking about is:

• --benchmark → How long does validation take?
• --trace → What validation instructions are being executed?
• --profile → Where is the execution time being spent?

I noticed that Blaze already has contrib/validate.cc, which appears to provide a convenient way to run validation under an external profiler. So I’m wondering whether exposing a similar workflow through the CLI would be useful.

Also, I’m doing considerably more research into profiling than was probably necessary for this issue. So, if anyone is reading this and thinking about implementing it immediately, please don’t. It will make me sad.

[jviotti]

Hey @channi23 ,

This is actually not a bad idea. The jsonschema CLI is for end usage, but I think having something like contrib/profile.cc as a little experimental CLI in this repo so we can easily use it to gather statistics about any run while we develop Blaze itself would be super cool.

[channi23]

Yes @jviotti I agree with your idea, also I tested external profiling on macOS using Instruments / Time Profiler, with contrib/validate.cc as the workload driver.

The workflow is:

cmake -DBLAZE_CONTRIB=ON ...

Then:

./build/contrib/sourcemeta_blaze_contrib_validate \
  path/to/schema.json \
  path/to/instances.jsonl \
  --warmup

For profiling:

xcrun xctrace record \
  --template "Time Profiler" \
  --output blaze-profile.trace \
  --launch -- \
  ./build/contrib/sourcemeta_blaze_contrib_validate \
  path/to/schema.json \
  path/to/instances.jsonl \
  --warmup

With a sufficiently long JSONL workload, Instruments produces a useful call tree down into the Blaze evaluator, instruction dispatch, and sourcemeta::core JSON operations, as shown in the attached screenshot.

I initially expected the regular jsonschema validate CLI to provide something similar, but currently --benchmark and --trace expose different levels of information. The contrib/validate.cc + external profiler workflow already works well for development-time profiling.

This also seems like a useful baseline for deciding what additional Blaze-specific statistics a future contrib/profile.cc could provide.

[channi23]

also @jviotti , I’ve also been thinking about the statistics side of profiling. With the different schemas and instances I’m planning to test, timing alone may not always tell us where the real bottleneck is. A function taking more time doesn’t necessarily mean it needs optimization, since it could simply be called much more frequently, so I think call counts and inclusive/self time would be really useful alongside the timings. I’d like to explore what other statistics we can collect and how we can correlate them back to Blaze instructions/call stacks. I think this would be interesting to discuss in tomorrow’s meeting.(also again i do not need to too much pessimistic to only find out bottlenecks but should be sure if we have one, so much to learn)

[channi23]

but again, only you can tell, do we really need profiling or it would be good to have , as i got this up out of my curiosity and my own problem of writing schema separately to measure time.

[jviotti]

Yeah, Instruments is great at giving you a full blown CPU Profile. I think where a profiling mode specific for Blaze shines is if you complement a CPU Profile with Blaze specific information. Things that no generic CPU Profile could be able to see so easily.

With the different schemas and instances I’m planning to test, timing alone may not always tell us where the real bottleneck is.

Exactly! One area where I think more info would be interesting is that in many cases, we notice that some instructions tend to always happen together. And therefore seeing that, thinking about a fused instruction that accomplishes both things in ONE pass can speed things up a bit. If any, I think most major optimisations we did in Blaze so far were around reducing the number of instructions. Coming up with higher level ones that replace many small ones, etc.

I think this would be interesting to discuss in tomorrow’s meeting

100%. Would love to see you there!

also again i do not need to too much pessimistic to only find out bottlenecks but should be sure if we have one, so much to learn

I'm sure there are still LOTS of places for optimisation. I wouldn't be surprised if you find quite a few of them. And it's all really about the process. If you come up with a good Blaze-specific profiler that can surface good insights and clear the noise away, I'm sure you will find a lot of them.

[jviotti]

Also an angle worth considering is that if you can capture good data on schema evaluation, from i.e. traces and your own Blaze profile, alongside real CPU Profiles from i.e. Instruments plus any other type of data, LLMs can be quite good at ingesting all of that at once and finding patterns that are tricky for humans to see.

[channi23]

Hi @jviotti after our last meeting, I’m thinking of approaching this in two stages. First, I’d like to design the architecture for a Blaze-specific (contrib/profile.cc), covering things like validation timing, instruction/keyword counts, execution statistics, and eventually CPU/resource metrics. I’d like to finish that architecture and discuss the implementation details in our next meeting. Once we have the Blaze profiler implemented and I’ve learned more about the execution model and useful statistics, I’d like to use that experience to explore a validator-agnostic JSON Schema profiling model that could potentially work across Blaze, AJV, and other validators. What do you think about this approach? or any specific step that i need to add or read or that you would like me to do in the first place, before taking this!

[jviotti]

Sounds cool! I don't have any specific tips other than the fact you should dogfood it. I guess there are infinite amount of things you could display in the profiles, but you will actually know what works by trying to identify optimisation opportunities with it. i.e. if the tool you write can give us (or an LLM) better insights on how to improve the speed of the evaluator or at least pin-point a root cause deeper in the stack (like on sourcemeta/core), then that's a job very well done!