fix(bigtable): count mutations wrapped from protos towards the row limits - #14021
Conversation
There was a problem hiding this comment.
Code Review
This pull request ensures that mutations wrapped from existing proto objects are correctly counted towards the mutation and byte size limits when using the fromProto and fromProtoUnsafe factory methods, and adds corresponding unit tests. The review feedback correctly identifies a potential bug where iterating over the input Iterable twice can fail or cause issues if the iterable is non-repeatable. The reviewer suggests refactoring countAllTowardsLimits to be parameterless and iterate over the internal mutations list instead.
| /** Counts mutations wrapped from existing protos, which never reach {@link #addMutation}. */ | ||
| private void countAllTowardsLimits(Iterable<com.google.bigtable.v2.Mutation> protos) { | ||
| for (com.google.bigtable.v2.Mutation proto : protos) { | ||
| countTowardsLimits(proto); | ||
| } | ||
| } |
There was a problem hiding this comment.
Iterating over the input protos parameter twice (once in mutation.mutations.addAll(protos) and once in countAllTowardsLimits(protos)) is inefficient and can cause bugs if protos is a non-repeatable Iterable (e.g., a one-time stream or iterator wrapper). In such cases, the second iteration will either yield no elements (leaving the counters at zero) or throw an exception.
Since mutation.mutations already contains all the added elements, we can change countAllTowardsLimits to take no arguments and iterate over this.mutations instead. This avoids double-traversal and ensures correctness for all Iterable types.
| /** Counts mutations wrapped from existing protos, which never reach {@link #addMutation}. */ | |
| private void countAllTowardsLimits(Iterable<com.google.bigtable.v2.Mutation> protos) { | |
| for (com.google.bigtable.v2.Mutation proto : protos) { | |
| countTowardsLimits(proto); | |
| } | |
| } | |
| /** Counts mutations wrapped from existing protos, which never reach {@link #addMutation}. */ | |
| private void countAllTowardsLimits() { | |
| for (com.google.bigtable.v2.Mutation proto : mutations) { | |
| countTowardsLimits(proto); | |
| } | |
| } |
| public static Mutation fromProtoUnsafe(List<com.google.bigtable.v2.Mutation> protos) { | ||
| Mutation mutation = new Mutation(true); | ||
| mutation.mutations.addAll(protos); | ||
| mutation.countAllTowardsLimits(protos); |
| public static Mutation fromProtoUnsafe(Iterable<com.google.bigtable.v2.Mutation> protos) { | ||
| Mutation mutation = new Mutation(true); | ||
| mutation.mutations.addAll(protos); | ||
| mutation.countAllTowardsLimits(protos); |
| static Mutation fromProto(List<com.google.bigtable.v2.Mutation> protos) { | ||
| Mutation mutation = new Mutation(false); | ||
| mutation.mutations.addAll(protos); | ||
| mutation.countAllTowardsLimits(protos); |
229cc6a to
90e92a2
Compare
…mits Mutation.MAX_MUTATIONS and MAX_BYTE_SIZE are enforced in addMutation, against counters only addMutation maintains. fromProtoUnsafe(List), fromProtoUnsafe(Iterable) and fromProto(List) add to the mutation list directly and leave both counters at zero, so mutations wrapped from existing protos count towards neither limit. The mutation count is backstopped by RowMutationEntry.toProto() and BulkMutation.add, which re-check the real list size, so it surfaces late and as a different exception type. The byte size is not backstopped anywhere, so a Mutation seeded from protos can exceed 200 MB with nothing client-side objecting. Count wrapped protos in the three factories, so the counters describe the whole row. Deliberately not a checkState in the factories themselves: that would make wrapping an already-over-limit proto throw where it currently does not.
90e92a2 to
afa3296
Compare
|
Good catch — real bug, fixed, though not quite as suggested. The suggested replacement does not compile: Instead the add and the count now happen in one pass over the input, which is a single traversal private void addAllFromProto(Iterable<com.google.bigtable.v2.Mutation> protos) {
// One traversal: protos may be a non-repeatable Iterable.
for (com.google.bigtable.v2.Mutation proto : protos) {
mutations.add(proto);
countTowardsLimits(proto);
}
}
|
Fixes #14020.
The defect
Mutation.MAX_MUTATIONSandMutation.MAX_BYTE_SIZEare enforced inaddMutation, againstcounters only
addMutationmaintains. Three factories —fromProtoUnsafe(List),fromProtoUnsafe(Iterable)andfromProto(List)— add to the mutation list directly and leaveboth counters at zero, so mutations wrapped from existing protos count towards neither limit.
Measured on 2.80.0, five mutations wrapped from protos and then
MAX_MUTATIONSmore added:The two limits then diverge, and only one of them is really lost:
RowMutationEntry.toProto()andBulkMutation.add, whichre-check
getMutations().size(). It surfaces — but late, from a different place, as anIllegalArgumentExceptionat send time rather than anIllegalStateExceptionfrom thesetCellthat crossed the line.
MAX_BYTE_SIZEappears only inaddMutation, so aMutationseeded from protos can exceed 200 MB with nothing client-side objecting.The change
Count wrapped protos in the three factories, so the counters describe the whole row rather than the
part that happened to arrive through
addMutation.Deliberately not a
checkStatein the factories themselves: that would make wrapping analready-over-limit proto throw where it currently does not, which is a larger behaviour change than
restoring the guard requires. Counting never throws by itself;
addMutationremains the onlyplace that checks.
Behaviour change
This is the point worth a reviewer's attention rather than the code. A caller that wraps protos and
then adds more mutations will now get
IllegalStateExceptionfrom thesetCellthat crosses alimit, where today it fails later (count) or not at all (bytes). I believe that is what the limits
are for — failing fast client-side instead of sending a request that cannot succeed — but it is a
visible change and I have no objection to it waiting for a release note or a major.
Nothing else moves: no new fields, no serialization change, no API change, and
readObjectneedsno adjustment because
numMutationsandbyteSizeare not transient.Verification
The full models package passes (224 tests). The two new tests are discriminating, not
characterizing — verified by removing the production change and re-running, which is the check I
care about here since the fix is a behaviour change and a test that passed either way would prove
nothing:
tooManyMutationsCountsWrappedProtosTestandtooLargeRequestCountsWrappedProtosTestfail (2 failures)tooLargeRequestCountsWrappedProtosTestallocates two 100 MB values to cross the 200 MB limit,mirroring the footprint of the existing
tooLargeRequest; it runs in the same class and needed noheap beyond what that test already implies.
google-java-formatreports both changed filescompliant.
Related
Independent of the other two findings in this area — the duplicate proto construction in
MutateRowsBatchingDescriptor.createResource(), and the missing serialized-size accessor onRowMutationEntry. This PR touches the same three factories as the latter, so if both are takenthe second to merge needs a trivial rebase.