Lingering performance issues after dealing with a very complex model #1026
Description
Hi!
First of all, thanks for all the hard work put into this awesome library.
I'm in the process of migrating a service heavily reliant on pydantic from v1 to v2.
For context, this service deals with document content, utilizing complex, deeply/recursively nested models with heavy use of tagged unions. Models represent various "building blocks" like block, text, image, table, etc. Basically, imagine parsing an HTML with pydantic and you get a feel of what these models look like.
I was particularly excited about the performance improvements promised by Pydantic v2. However, after completing the migration, our benchmarks showed unexpected results.
There's a significant performance degradation (compared to v1) and notable variance in benchmarks that serialize and deserialize synthetic data using various models. After looking closer, I noticed that this issue only occurs when running multiple benchmarks sequentially. When executed individually, the performance is much better and more consistent. I also noticed that it depends on the order of benchmarks. Eventually, I came to the conclusion that degradation starts on the first fairly complex benchmark and lingers, affecting subsequent simple benchmarks. I managed to boil down the repro to this (full gist here):
benchmark_simple_model()
benchmark_complex_model() # with wide and deep synthetic data
benchmark_simple_model()
The first benchmark performs well, outpacing v1 with minimal variance. However, the second and third benchmarks exhibit much worse performance compared to v1, with significant variance. What's most interesting is that the third benchmark's performance is more than an order of magnitude worse than the first one, even though they exercise the same simple model with the same small/simple data.
Removing the complex_model benchmark removes this issue - both simple_model benchmarks perform consistently and well.
It seems that processing large and complex data with a highly intricate model leaves a lingering negative performance .
I attempted to profile pydantic-core during these benchmarks. The flamegraph revealed that about 80% of the time in the third benchmark is spent in GILPool::drop.
In comparison, the first benchmark only spends 3% of the time in GILPool::drop.
All in all, it feels like the second benchmark leaves behind obscene amounts of garbage, somehow jamming up future deallocations. I also feel like this issue might be related to PyO3/pyo3#1056, this feeling is reinforced by #997 (comment).
At this point, I'm unsure how to proceed and would appreciate your help. Thanks in advance!