compute: factor out PeekResultIterator #32514
Conversation
aljoscha
force-pushed
the
compute-refactor-peek-result-iterator
branch
from
570a68c to
c3a400a
Compare
aljoscha
force-pushed
the
compute-refactor-peek-result-iterator
branch
2 times, most recently
from
e75dd05 to
214ebd8
Compare
src/compute/src/compute_state.rs
Outdated
| PendingPeek::Index(peek) => 'response: { | ||
| let is_ready = peek.is_ready(upper); | ||
|
|
||
| match is_ready { | ||
| Ok(false) => break 'response None, | ||
| Err(err) => break 'response Some(err), | ||
| Ok(true) => (), // Falling through..., | ||
| } | ||
|
|
||
| if let Some(err) = peek.extract_errs(upper) { | ||
| break 'response Some(err); | ||
| } | ||
|
|
||
| Some(peek.read_result(upper, self.compute_state.max_result_size)) |
There was a problem hiding this comment.
I like early returns usually, but in this case the nested-if version ends up more readable, imo:
PendingPeek::Index(peek) => match peek.is_ready(upper) {
Ok(true) => {
let resp = peek.extract_errs(upper).unwrap_or_else(|| {
peek.read_result(upper, self.compute_state.max_result_size)
});
Some(resp)
}
Ok(false) => None,
Err(err) => Some(err),
},There was a problem hiding this comment.
sorry, this was from an outdated branch, I updated and backed out all of the extra refactorings, it's not only about the peek iterator
| // Copyright Materialize, Inc. and contributors. All rights reserved. | ||
| // | ||
| // Use of this software is governed by the Business Source License | ||
| // included in the LICENSE file. |
There was a problem hiding this comment.
Huh, isn't there usually also a paragraph about transitioning to the Apache license here? 😅
There was a problem hiding this comment.
This matches the header of compute_state.rs, but I checked and some of our other source files do indeed have that Apache license blob as well. So, I don't know ... 🤷♂️
There was a problem hiding this comment.
Well, I guess that part is redundant anyway because it's also in the LICENSE file 🤷
| peek_timestamp: mz_repr::Timestamp, | ||
| has_literal_constraints: bool, | ||
| literals: L, | ||
| oks_handle: &mut Tr, |
There was a problem hiding this comment.
Nit: It's a bit confusing to have an oks_handle if we don't also have an errs_handle. We could call it trace/trace_reader/trace_handle instead?
There was a problem hiding this comment.
will change to trace_reader
| literals: L, | ||
| oks_handle: &mut Tr, | ||
| ) -> Self { | ||
| let (cursor, storage): (<Tr as TraceReader>::Cursor, <Tr as TraceReader>::Storage) = |
There was a problem hiding this comment.
These type annotations are not needed, are they?
There was a problem hiding this comment.
I think no, these where from some work in progress where types didn't line up
| tracing::trace!( | ||
| ?self.literals_exhausted, | ||
| key_valid = self.cursor.key_valid(&self.storage), | ||
| val_valid = self.cursor.val_valid(&self.storage), "next"); |
There was a problem hiding this comment.
How useful are these traces? I think they will mostly print a lot of true/false that are hard to interpret? Should they also print the actual keys and values?
There was a problem hiding this comment.
Reading the key/val wouldn't always succeed because it fails when they're not valid. So I should rather remove all these trace logs, yeah?
| if self.cursor.val_valid(&self.storage) { | ||
| return false; | ||
| } |
There was a problem hiding this comment.
This is surprising I think. The doc string says that this method will step the key forward, but it only does so if the current value is not valid, which seems like an important detail!
A thing that would make sense to check here is key_valid. Maybe you meant to do that?
There was a problem hiding this comment.
The intended behavior was actually maybe_step_key, and it would only step when the val is not valid. Which is what the code did. In practice the method is only called when the val is not valid, so I changed that check into an assert and left the docstring as is.
| if self.cursor.val_valid(&self.storage) { | ||
| break; | ||
| } |
There was a problem hiding this comment.
For this break to be invoked we would need to have a key with zero values. Is this possible?
There was a problem hiding this comment.
I think you meant it the other way round, right? That you expect val_valid to always be true so we always break?
I think it can happen that we go around the loop multiple times but I don't remember why. I pushed a commit that panics when that happens, so let's see what ci has to say.
|
Pretty sure this will conflict with #32593, sorry 🙈 |
aljoscha
changed the title
The original motivation for this is so that the code that extracts peek results can be re-used in MaterializeInc/database-issues#9180, where we want to use a different transport for sending back peek responses but still need to read them out of arrangements the same way. The nice side effect is that we separate extracting the result from the logic that accumulates it in a response for sending it back. Which leads to clearer separation.
aljoscha
force-pushed
the
compute-refactor-peek-result-iterator
branch
from
4640550 to
a74366e
Compare
|
@teskje thanks for the review! I hope I addressed all comments, could you please take a look again? 🙇♂️ |
The original motivation for this is so that the code that extracts peek results can be re-used in
https://github.com/MaterializeInc/database-issues/issues/9180, where we want to use a different transport for sending back peek responses but still need to read them out of arrangements the same way.
The nice side effect is that we separate extracting the result from the logic that accumulates it in a response for sending it back. Which leads to clearer separation.
Work towards https://github.com/MaterializeInc/database-issues/issues/9180