Skip to content

ENG-30396: STREAMSNAP checks streaming snapshots (milestone 51) - #50

Closed
jwiebema1 wants to merge 5 commits into
ENG-30396-sqlancer-m50from
ENG-30396-sqlancer-m51
Closed

jwiebema1 wants to merge 5 commits into
ENG-30396-sqlancer-m50from
ENG-30396-sqlancer-m51

Conversation

@jwiebema1

Copy link
Copy Markdown

Why

Milestone 50's coverage runs left VoltDB's streaming snapshots unreached: org.voltdb.streamingsnapshots had 9 of its 708 lines covered. In a streaming snapshot, a client asks the cluster to take a snapshot, and each host streams its partitions' chunks straight to the client. snapshot-transfer is built on it.

What it changes

  • A STREAMSNAP oracle (VoltDBStreamingSnapshotOracle). Each check fills two tables of its own, one partitioned and one replicated, and reads them back. It then streams a snapshot through Client2Impl.streamSnapshotAsync, asking for both tables, one of them, every table of the cluster but one, or a table list with skipTables. Checks:
    • the stream must end;
    • each table asked for must send exactly the table's rows, and no other table may send rows;
    • the catalog metadata must list the tables;
    • some checks also stream again under the same name, which must be refused, or start two streams at once from two clients; each must stream the right rows or be refused as already in progress.
  • Snapshot cleanup: the snapshot each stream writes is deleted with @SnapshotDelete; SNAPSHOT shares its lock and its delete helper.
  • Reproductions: repros/ENG-30720.sh, ENG-30722.sh, ENG-30723.sh, ENG-30724.sh and ENG-30725.sh, with their Java programs in repros/streaming/.
  • run-coverage.sh runs STREAMSNAP in every run, and the summary has a STREAMSNAP line.
  • run-sqlancer.sh also discounts the FATAL lines a gracefully stopped node logs after a forced exit (ENG-30704 by another path, which the coverage run hit).
  • The insert generator's UPSERT now names every primary key column, as VoltDB requires. A run without --avoid-known-bugs hit this generator bug.
  • README: a STREAMSNAP section, the six bugs among the known bugs, and the status now says milestone 51.

On the job side (VoltDB/pro#7150, VoltDB/internal#6474), STREAMSNAP joins the default oracles, and the Jenkinsfile pins this branch's head.

Bugs found

Ticket Priority Bug Workaround
ENG-30720 Blocker Client2Impl.streamSnapshotAsync completes before its chunk consumer has every chunk: the client handles messages on a thread pool, so a host's end-of-tables message can overtake its chunks. snapshot-transfer writes _SUCCESS at that point. Released client 16.0.0 too. Wait up to 10 s for late chunks.
ENG-30722 Blocker A stream named s1 also streams the files of every snapshot whose name begins with s1 (s10, ...), so a table can send rows it no longer holds. Fixed-width names with a final letter, deleted at once.
ENG-30723 Blocker A stream whose table lists match no table never ends: no host sends its end-of-tables message, and that client cannot stream again. Ask only for tables that exist.
ENG-30725 Blocker A client that disconnects after its stream ended deletes the participant nodes of the stream in progress. That stream's state is never removed, and every later stream is refused until its client disconnects. Found in a local run, reproduced on 3 nodes. Two shared clients that stay connected.
ENG-30724 Minor After a stream, Client2 keeps one request outstanding for 60 s, so close() and drain() take 60 s. Close clients in the background.
ENG-30721 Minor The Java client prints its streaming debug lines to stderr on every stream. Released client 16.0.0 too. None.

Coverage (run-coverage.sh's 10-minute kill run, against the best of milestones 44 to 50)

org.voltdb.streamingsnapshots: 463 of 708 lines, from 9.

Class Before Milestone 51
SingleHostSnapshotTableProcessor 0 of 122 92 of 122
SingleHostSnapshotDataProcessor 0 of 88 77 of 88
StreamingSnapshotUtils 4 of 147 73 of 147
SingleHostCatalogReader 0 of 88 50 of 88
SingleHostCatalogSender 0 of 57 48 of 57
ExternalSnapshotStreamWorkerNTProc 4 of 45 35 of 45
SnapshotStreamWorker 0 of 54 26 of 54
InvocationDispatcher 357 of 1028 471 of 1028

In all, 642 more lines are covered, in 25 classes.

Testing

  • Local trials on 3 nodes, k=1, 5 minutes each:

    • STREAMSNAP with SNAPSHOT and NOREC: passed, 57 streams and 3,855 rows. The run before the shared clients found ENG-30725.
    • With security and TLS, beside SNAPSHOT and LOGIN: passed, 29 streams. The run before the retry lost a 22-row table to ENG-30720 (its chunks dropped after completion).
    • Without --avoid-known-bugs: found ENG-30723 twice and ENG-30720 once.
  • Coverage run: STREAMSNAP passed its 8 streams. The kill run failed only on a stopped node's FATAL line (ENG-30704), now discounted.

  • mvn verify -DskipTests passes.

  • Lab: branch job ENG-30396-sqlancer-nightly of matrix-systemtest-apprunner-sqlancer, build Build problem on MacOS with OpenJDK 13 sqlancer/sqlancer#67, a replay against 16.1.0-SNAPSHOT (master kit 400). It ran fork fc92064, all thirty-four oracles, 30 minutes per run, and passed:

    • kill run (3:1:300, 3 kills, 2 graceful stops): 50 streams, 1,203 rows in 7,098 chunks; 3 given up to node failures;
    • upgrade run (3:1:u, from 16.0.0 in service, 3 of 3 nodes upgraded): 71 streams, 3,383 rows;
    • security run (1:0:sec): 70 streams, 3,322 rows, 10 concurrent pairs; 103 LOGIN checks.

    No failed database and no FATAL line. The head adds the ENG-30720 retry, the 120 s limit while nodes fail, the forced-exit exclusion and the README; the local runs above cover those.

…G-30722, ENG-30723

ENG-30720: Client2Impl.streamSnapshotAsync completes before its chunk consumer has
every chunk. ENG-30722: a streaming snapshot also streams the files of every snapshot
whose name begins with its own. ENG-30723: a stream whose table lists match no table
never ends.
… ENG-30725

ENG-30724: after a stream, Client2 keeps one request outstanding for 60 s, so close()
and drain() take 60 s. ENG-30725: a client disconnecting after its stream ended
deletes the participant nodes of the stream in progress, whose state then blocks
every later stream.
…e 51)

Client2Impl.streamSnapshotAsync of the oracle's own tables, of some or of every table
but some: each table's rows streamed exactly, no other table's, the catalog metadata
listing them; a second stream under the same name or at the same time refused. It
avoids ENG-30720, ENG-30722, ENG-30723 and ENG-30725 with --avoid-known-bugs, and
closes clients in the background (ENG-30724). SNAPSHOT shares its lock and
@SnapshotDelete helper. The insert generator's UPSERT now names every primary key
column, which VoltDB requires.
…(milestone 51)

The README's STREAMSNAP section, its numbers and the six bugs. STREAMSNAP gives up a
stream after 120 s while nodes fail. run-sqlancer.sh also discounts the FATAL lines a
stopped node logs after a forced exit ("Forcing the database node to exit", ENG-30704),
which milestone 51's coverage run hit.
…ast chunks

ENG-30720's late chunks can be dropped for good once the stream has completed (a
local TLS run lost a 22-row table). While avoiding known bugs, a stream still short
of rows after the 10 s wait is streamed again, up to three times in all.
@jwiebema1

Copy link
Copy Markdown
Author

Superseded by #51, which combines every ENG-30396 PR of this repo into one. This PR's branch is kept, and its description still has this milestone's details and lab results.

@jwiebema1 jwiebema1 closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant