Repository navigation
Conversation
…G-30722, ENG-30723 ENG-30720: Client2Impl.streamSnapshotAsync completes before its chunk consumer has every chunk. ENG-30722: a streaming snapshot also streams the files of every snapshot whose name begins with its own. ENG-30723: a stream whose table lists match no table never ends.
… ENG-30725 ENG-30724: after a stream, Client2 keeps one request outstanding for 60 s, so close() and drain() take 60 s. ENG-30725: a client disconnecting after its stream ended deletes the participant nodes of the stream in progress, whose state then blocks every later stream.
…e 51) Client2Impl.streamSnapshotAsync of the oracle's own tables, of some or of every table but some: each table's rows streamed exactly, no other table's, the catalog metadata listing them; a second stream under the same name or at the same time refused. It avoids ENG-30720, ENG-30722, ENG-30723 and ENG-30725 with --avoid-known-bugs, and closes clients in the background (ENG-30724). SNAPSHOT shares its lock and @SnapshotDelete helper. The insert generator's UPSERT now names every primary key column, which VoltDB requires.
…(milestone 51)
The README's STREAMSNAP section, its numbers and the six bugs. STREAMSNAP gives up a
stream after 120 s while nodes fail. run-sqlancer.sh also discounts the FATAL lines a
stopped node logs after a forced exit ("Forcing the database node to exit", ENG-30704),
which milestone 51's coverage run hit.
…ast chunks ENG-30720's late chunks can be dropped for good once the stream has completed (a local TLS run lost a 22-row table). While avoiding known bugs, a stream still short of rows after the 10 s wait is streamed again, up to three times in all.
This was referenced Nov 18, 2026
Closed
Author
|
Superseded by #51, which combines every ENG-30396 PR of this repo into one. This PR's branch is kept, and its description still has this milestone's details and lab results. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Milestone 50's coverage runs left VoltDB's streaming snapshots unreached:
org.voltdb.streamingsnapshotshad 9 of its 708 lines covered. In a streaming snapshot, a client asks the cluster to take a snapshot, and each host streams its partitions' chunks straight to the client. snapshot-transfer is built on it.What it changes
VoltDBStreamingSnapshotOracle). Each check fills two tables of its own, one partitioned and one replicated, and reads them back. It then streams a snapshot throughClient2Impl.streamSnapshotAsync, asking for both tables, one of them, every table of the cluster but one, or a table list withskipTables. Checks:@SnapshotDelete; SNAPSHOT shares its lock and its delete helper.repros/ENG-30720.sh,ENG-30722.sh,ENG-30723.sh,ENG-30724.shandENG-30725.sh, with their Java programs inrepros/streaming/.run-coverage.shruns STREAMSNAP in every run, and the summary has a STREAMSNAP line.run-sqlancer.shalso discounts the FATAL lines a gracefully stopped node logs after a forced exit (ENG-30704 by another path, which the coverage run hit).--avoid-known-bugshit this generator bug.On the job side (VoltDB/pro#7150, VoltDB/internal#6474), STREAMSNAP joins the default oracles, and the Jenkinsfile pins this branch's head.
Bugs found
Client2Impl.streamSnapshotAsynccompletes before its chunk consumer has every chunk: the client handles messages on a thread pool, so a host's end-of-tables message can overtake its chunks. snapshot-transfer writes_SUCCESSat that point. Released client 16.0.0 too.s1also streams the files of every snapshot whose name begins withs1(s10, ...), so a table can send rows it no longer holds.Client2keeps one request outstanding for 60 s, soclose()anddrain()take 60 s.Coverage (
run-coverage.sh's 10-minute kill run, against the best of milestones 44 to 50)org.voltdb.streamingsnapshots: 463 of 708 lines, from 9.SingleHostSnapshotTableProcessorSingleHostSnapshotDataProcessorStreamingSnapshotUtilsSingleHostCatalogReaderSingleHostCatalogSenderExternalSnapshotStreamWorkerNTProcSnapshotStreamWorkerInvocationDispatcherIn all, 642 more lines are covered, in 25 classes.
Testing
Local trials on 3 nodes, k=1, 5 minutes each:
--avoid-known-bugs: found ENG-30723 twice and ENG-30720 once.Coverage run: STREAMSNAP passed its 8 streams. The kill run failed only on a stopped node's FATAL line (ENG-30704), now discounted.
mvn verify -DskipTestspasses.Lab: branch job
ENG-30396-sqlancer-nightlyofmatrix-systemtest-apprunner-sqlancer, build Build problem on MacOS with OpenJDK 13 sqlancer/sqlancer#67, a replay against 16.1.0-SNAPSHOT (master kit 400). It ran fork fc92064, all thirty-four oracles, 30 minutes per run, and passed:3:1:300, 3 kills, 2 graceful stops): 50 streams, 1,203 rows in 7,098 chunks; 3 given up to node failures;3:1:u, from 16.0.0 in service, 3 of 3 nodes upgraded): 71 streams, 3,383 rows;1:0:sec): 70 streams, 3,322 rows, 10 concurrent pairs; 103 LOGIN checks.No failed database and no FATAL line. The head adds the ENG-30720 retry, the 120 s limit while nodes fail, the forced-exit exclusion and the README; the local runs above cover those.