Skip to content

Commit 2176330

Browse files
Revert the removal of reservation in HashJoin (#13792)
* fix: restore memory reservation in JoinLeftData for accurate memory accounting in HashJoin This commit reintroduces the `_reservation` field in the `JoinLeftData` structure to ensure proper tracking of memory resources during join operations. The absence of this field could lead to inconsistent memory usage reporting and potential out-of-memory issues as upstream operators increase their memory consumption. * fmt Signed-off-by: Jay Zhan <[email protected]> --------- Signed-off-by: Jay Zhan <[email protected]>
1 parent bd2c975 commit 2176330

File tree

1 file changed

+8
-0
lines changed

1 file changed

+8
-0
lines changed

datafusion/physical-plan/src/joins/hash_join.rs

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,11 @@ struct JoinLeftData {
9090
/// Counter of running probe-threads, potentially
9191
/// able to update `visited_indices_bitmap`
9292
probe_threads_counter: AtomicUsize,
93+
/// We need to keep this field to maintain accurate memory accounting, even though we don't directly use it.
94+
/// Without holding onto this reservation, the recorded memory usage would become inconsistent with actual usage.
95+
/// This could hide potential out-of-memory issues, especially when upstream operators increase their memory consumption.
96+
/// The MemoryReservation ensures proper tracking of memory resources throughout the join operation's lifecycle.
97+
_reservation: MemoryReservation,
9398
}
9499

95100
impl JoinLeftData {
@@ -99,12 +104,14 @@ impl JoinLeftData {
99104
batch: RecordBatch,
100105
visited_indices_bitmap: SharedBitmapBuilder,
101106
probe_threads_counter: AtomicUsize,
107+
reservation: MemoryReservation,
102108
) -> Self {
103109
Self {
104110
hash_map,
105111
batch,
106112
visited_indices_bitmap,
107113
probe_threads_counter,
114+
_reservation: reservation,
108115
}
109116
}
110117

@@ -897,6 +904,7 @@ async fn collect_left_input(
897904
single_batch,
898905
Mutex::new(visited_indices_bitmap),
899906
AtomicUsize::new(probe_threads_count),
907+
reservation,
900908
);
901909

902910
Ok(data)

0 commit comments

Comments
 (0)