Skip to content

Commit 4110bcd

Browse files
authored
chore: Reimplement ShuffleWriterExec using interleave_record_batch (#1511)
* Reimplement ShuffleWriterExec using interleave_record_batch * Coalesce small batches into larger batches when shuffle writing single partition * buffer writes to shuffle files and spill files * Reimplemented single partition shuffle write * Clean up * Address review comments * Address review comments * Fix timer in write and flush to measure nothing but write_all and flush * Measure more disk writing operations
1 parent d879e1a commit 4110bcd

File tree

5 files changed

+658
-1087
lines changed

5 files changed

+658
-1087
lines changed

native/core/src/execution/planner.rs

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1214,7 +1214,7 @@ impl PhysicalPlanner {
12141214
}?;
12151215

12161216
let shuffle_writer = Arc::new(ShuffleWriterExec::try_new(
1217-
Arc::clone(&child.native_plan),
1217+
Self::wrap_in_copy_exec(Arc::clone(&child.native_plan)),
12181218
partitioning,
12191219
codec,
12201220
writer.output_data_file.clone(),

0 commit comments

Comments
 (0)