perf: desugar remaining #[async_trait] stubs capturing Expr/LogicalPlan - #24370
perf: desugar remaining #[async_trait] stubs capturing Expr/LogicalPlan#24370alamb wants to merge 1 commit into
#[async_trait] stubs capturing Expr/LogicalPlan#24370Conversation
Applies the rewrite from apache#24325/apache#24326/apache#24329/apache#24330 to the last live methods on the serial build path whose futures capture Expr- or LogicalPlan-reaching types: UnsupportedQueryPlanner::create_physical_plan and ExtensionPlanner::plan_table_scan (datafusion-session), EmptyTable::scan (datafusion-catalog), and TestTableProvider::scan (datafusion core). No public signature changes after macro expansion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24370 +/- ##
==========================================
- Coverage 81.19% 81.18% -0.02%
==========================================
Files 1110 1110
Lines 388616 388677 +61
Branches 388616 388677 +61
==========================================
Hits 315534 315534
- Misses 54500 54555 +55
- Partials 18582 18588 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
I had codex run some tests on a gcp machine and this PR doesn't improve compialtion speed Details below: Compared PR head d868efb against its base 571477c, using 5 interleaved rounds. A full
The first session/catalog measurements are cold-build outliers. Excluding round 1, the PR was approximately 1.2% slower for session, 0.9% slower for catalog, and 0.2% slower for core—within normal timing noise on this machine. Exact benchmark commands# once per checkout
cargo clean
cargo build -p datafusion --quiet
# each round, before the timed commands
cargo clean -p datafusion-session -p datafusion-catalog -p datafusion --quiet
# timed commands
/usr/bin/time -f "RESULT package=datafusion-session seconds=%e" cargo rustc -p datafusion-session --lib --quiet
/usr/bin/time -f "RESULT package=datafusion-catalog seconds=%e" cargo rustc -p datafusion-catalog --lib --quiet
/usr/bin/time -f "RESULT package=datafusion seconds=%e" cargo rustc -p datafusion --lib --quiet |
Which issue does this PR close?
datafusioncore crate) #13814datafusion-catalogcompile time ~6x #24325, perf(session): cutdatafusion-sessioncompile time ~1.7x #24326, perf(core): cutdatafusioncore compile time ~10x #24329 and perf(catalog-listing): cutdatafusion-catalog-listingcompile time ~4.4x #24330Rationale for this change
Applies the same rewrite as the four PRs above to the remaining
#[async_trait]methods on the serial build path whose futures captureExpr/LogicalPlan-reaching types (so theirSend/Syncproofs are re-proved per method in a non-emptyParamEnv):UnsupportedQueryPlanner::create_physical_plan(datafusion-session) — captures&LogicalPlanExtensionPlanner::plan_table_scandefault body (datafusion-session) — captures&TableScan, which holdsVec<Expr>EmptyTable::scan(datafusion-catalog) — captures&[Expr]TestTableProvider::scan(datafusioncore;test_utilcompiles into the lib unconditionally) — captures&[Expr]What changes are included in this PR?
Each method becomes the hand-written desugaring of what
#[async_trait]generates (verified against-Zunpretty=expandedoutput), with no coroutine capturing the arguments:create_physical_plan,plan_table_scan,TestTableProvider::scan) returnready(..)or a capturelessasyncblock directly.EmptyTable::scanhas no.await, so its body moves verbatim to a plain inherent fn wrapped inready(..)(same asTestTableFactory::create_innerin perf(core): cutdatafusioncore compile time ~10x #24329).plan_table_scankeeps itsSelf: Sync + 'async_traitbound so the public trait signature is unchanged.Each converted method carries the explanatory comment from #24362.
Are these changes tested?
Functionally:
cargo fmt --check,cargo clippy -p datafusion-catalog -p datafusion-session --all-targets,cargo clippy -p datafusion --lib,cargo test -p datafusion-catalog -p datafusion-session, andcargo test -p datafusion --liball pass. The compiler checks each rewritten signature against its trait declaration.Compile-time impact is not yet measured (my machine is too noisy); this PR is a draft until the numbers below are collected.
Compile-time measurements to run (on a quiet machine)
Interleaved A/B of each affected crate's unit, alternating
mainand this branch at least 3 times each so machine drift cancels out (the methodology from #24325–#24330):Expected: a drop in
datafusion-session(which had ~0.65s ofevaluate_obligationremaining after #24326 in the standalone build, several times more in the feature-unified build) and a smaller drop indatafusion-catalogand core.EmptyTable/TestTableProviderare unproven — they did not appear in the per-impl profiles of #24325/#24329 — so it is worth confirming their marginal contribution before merging, e.g. by reverting one file at a time.To attribute the change to the trait solver rather than noise:
Are there any user-facing changes?
No. After macro expansion the method signatures are identical to before, including
plan_table_scan'sSelf: Syncbound.