Fix the bug when aggregation stops before a group is finished #3177
Conversation
| .build(false); | ||
|
|
||
| // In the testing data, there are 2 groups, each group has 3 rows. | ||
| // recordScanLimit = 5: scans 3 rows, and the 4th scan hits SCAN_LIMIT_REACHED |
There was a problem hiding this comment.
not sure why it only scans 4 times instead of 5, but the offset=1 seems consistent.
pengpeng-lu
changed the title
ScottDugas
changed the title
|
It looks like there's a conflict in the release notes. PRs no longer need to update the release notes file as they will be auto-generated by #3174, so I think the solution is just "take what is on main" |
| // last row in last group, is null if the current group is the first group | ||
| @Nullable | ||
| private RecordCursorResult<QueryResult> lastInLastGroup; | ||
| byte[] continuation; |
There was a problem hiding this comment.
| byte[] continuation; | |
| @Nullable | |
| private byte[] continuation; |
There was a problem hiding this comment.
It looks like this still hasn't marked the continuation field as private. Is there a reason it needs package visibility?
docs/sphinx/source/ReleaseNotes.md
Outdated
| @@ -20,7 +20,7 @@ Users performing online updates are encouraged to update from [4.0.559.4](#40559 | |||
| <h4> New Features </h4> | |||
|
|
|||
| * FRL respects PLAN_CACHE_*_MAX_ENTRIES - [PR #3156](https://github.com/FoundationDB/fdb-record-layer/pull/3156) | |||
| <h4> Bug Fixes </h4> | |||
| <h4> Bug Fixes </h4> | |||
There was a problem hiding this comment.
I'm not sure what happened here, but we probably want to back this out and take what's on main
pengpeng-lu
force-pushed
the
scan_limit_bug
branch
from
142279c to
14c8cfa
Compare
| // last row in last group, is null if the current group is the first group | ||
| @Nullable | ||
| private RecordCursorResult<QueryResult> lastInLastGroup; | ||
| byte[] continuation; |
There was a problem hiding this comment.
It looks like this still hasn't marked the continuation field as private. Is there a reason it needs package visibility?
| } else { | ||
| if (Verify.verifyNotNull(previousResult).getNoNextReason() == NoNextReason.SOURCE_EXHAUSTED) { | ||
| if (previousValidResult == null) { | ||
| return RecordCursorResult.exhausted(); |
There was a problem hiding this comment.
Does this need to check to insert the empty result if isCreateDefaultOnEmpty? It feels like that information is being lost here.
Perhaps it would be simpler if there were fewer moving parts here. If we wait until 4.2, we could drop support entirely for isCreateDefaultOnEmpty (see #3107) which may simplify this somewhat.
| currentContinuation = lastInLastGroup.getContinuation(); | ||
| } | ||
| previousValidResult = lastInLastGroup; | ||
| return RecordCursorResult.withoutNextValue(currentContinuation, Verify.verifyNotNull(previousResult).getNoNextReason()); |
There was a problem hiding this comment.
Should this also set previousResult? I think as written, if you call getNext a second time after getting one of these out-of-bounds values, you'll get an advanced continuation.
That is:
RecordCursor<T> aggCursor = getAggregateCursor();
RecordCursorResult<T> withValueResult = aggCursor.getNext(); // has a value
RecordCursorResult<T> withoutValueResult1 = aggCursor.getNext(); // first result with out-of-bound limit; continuation borrowed from withValueResult
RecordCursorResult<T> withoutValueResult2 = aggCursor.getNext(); // second result with out-of-bound limit; continuation may be greater than withoutValueResult1In general, once a cursor returns a RecordCursorResult without a next value, it should continue to return the same result if getNext() is returned again.
This resolves #3180