Repository navigation
Conversation
… (milestone 32) Milestone 30's coverage run found the engine's handler for views over joins (MaterializedViewHandler) never ran. A third of the views now read an inner join: distinct tables, at most one partitioned table unless joined on the partitioning columns, ON conditions that compare two columns, are random predicates, or both, and a random WHERE clause a third of the time (join views do not hit ENG-30583). The VIEWS oracle changes one of the view's tables before it compares. Tables any view reads are tracked through getViewedTables(). While avoiding known bugs, a join view groups by columns with distinct names (ENG-30529, found again through the view's query).
… primary key crashes the server) and ENG-30255 through join views on 16.0.0; workarounds and reproduction Lab build #48 (milestone 32) lost two servers to the new views over joins: - ENG-30664 (new): with a view over a join of T and a partitioned table, a UNIQUE index on T's primary key column, created while T has a row, crashes the server on the next INSERT into T ("Could not find Index with name ..."). Master and 16.0.0, 8 sites per host. While avoiding known bugs, an index on a table a view reads is not unique. - ENG-30255 (fixed after 16.0.0): a join view grouped by a NULL DECIMAL aborts a 16.0.0 server, the DIFF mirror. With a mirror or an in-service upgrade, join views do not group by DECIMAL. - voltdb/repros/ENG-30664.sql (8 sites); README rows.
…4's check does more after the crashing INSERT Lab build #49 reported ENG-30664 FIXED on Linux, where the same statements crash a server (macOS, and Linux in a container): the INSERT can be answered before the server dies in the fragments after it, and the runner looked for 1 s. A crash reproduction is now watched for 10 s, and ENG-30664's check (now with a second index on the column, as in the lab) inserts into both tables and reads the view after the INSERT.
… out-of-range key crashes the server); workaround and reproduction Lab build #49's resize run lost two nodes at once: a view over a join of a partitioned and a replicated table on TINYINT < INTEGER, with an index on the TINYINT column created after the view, and a key below TINYINT's range in the maintenance's index scan (ENG-30511) left the view without a tuple it expected. Not the resize: one node crashes the same way. While avoiding known bugs, columns of different types in a join view's condition are compared with = or <> only. voltdb/repros/ENG-30665.sql; README.
… (ENG-12656) Lab build #49's upgrade run: an INSERT into a join view's source table failed with "Type VARCHAR can't be cast as VARBINARY": the view's maintenance scans the other table's index on its VARBINARY(1) column with a 2-byte value, the existing ENG-12656 (a plain join query fails the same way). While avoiding known bugs, a join view's condition compares VARBINARY columns only with one of the same size, as its partitioning-column joins already did.
…even when sqlcmd got its answers Lab builds #49 and #50 reported ENG-30664 FIXED, yet the lab's own kit (16.1.0-SNAPSHOT-ENG-30396-sqlancer-nightly/2, aarch64, in a container) crashes on the same statements: the server answers the INSERT, logs FATAL and takes long to exit while it dumps core, and the runner only read the logs after a lost connection. For a "-- bug: crash" reproduction, a crash line in the server's output now counts whether or not the connection was lost.
Author
|
Superseded by #51, which combines every ENG-30396 PR of this repo into one. This PR's branch is kept, and its description still has this milestone's details and lab results. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 32 of ENG-30396 (tracked in ENG-30581, epic ENG-30580 "SQL Housekeeping"): materialized views over joins, the second coverage gap milestone 30's measurement found; and what lab builds #48 and #49 found through them. Stacked on #30 (milestone 31), which is stacked on #29 to #1; review those first.
Why
Milestone 30's coverage run found that the engine's code for views over joins,
MaterializedViewHandler, never ran. VoltDB keeps such a view up to date by running the view's query for each changed row, which is different code from the triggers that keep a view of one table up to date.What it changes
Views over joins
gen/VoltDBViewGenerator) now read an inner join of two or three tables:JOIN ... ON(VoltDB rejects outer joins in a view);getViewedTables()) for the other oracles' rules about views.--avoid-known-bugsis true:Bug reproductions
ENG-30664.sqlandENG-30665.sql.Bugs found
--avoid-known-bugsis trueT0.tinyint < T1.int, with an index on the TINYINT column created after the view. When a T1 value is below TINYINT's range, the view's maintenance reads T0 through the index with that key, which matches every row (ENG-30511). The next change kills the server: "MaterializedViewHandler ... went looking for a tuple in the view and expected to find it but didn't". Also on 16.0.0. Found by lab build #49's resize run, where an UPDATE of the replicated table killed two nodes at once and the cluster with them. Blocker for V16.1.0.Coverage
In probe runs on the instrumented engine (master bf8602e3),
MaterializedViewHandler.cppwent from 0% to 89.2% of its lines, andmaterializedviewhandlerinfo.cppto 71.9%.Testing
Lab build ENG-30396: STREAMSNAP checks streaming snapshots (milestone 51) #50 (
matrix-systemtest-apprunner-sqlancer, branch jobENG-30396-sqlancer-nightly) ran this milestone's a86d159, with milestone 31, on 16.1.0-SNAPSHOT beside one node of 16.0.0. Every run was 45 minutes with all twenty-two oracles:Lab builds ENG-30396: XDCR with node failures, --xdcr with --kill-interval (milestone 49) #48 and ENG-30396: the JAVAPROC oracle, Java stored procedures (milestone 50) #49 ran earlier commits of this milestone and found the bugs above.
Local trials (macOS, master): VIEWS only, 5 minutes, 172,000 queries; all oracles, 8 minutes; 3 nodes k=1 with 4 kills and 4 rejoins; 1 node with security and the 16.0.0 mirror, 10 minutes, 9,338 queries; VIEWS, NOREC, TLP_WHERE and PQS after the ENG-30665 workaround, 5 minutes, 17,968 queries. All passed.
The reproductions ENG-30664, ENG-30665, ENG-30611, ENG-30595, ENG-30588 and ENG-30493 are reported still broken by the runner at the head of this branch.
mvn verify(formatter, TestVoltDBExpressionEvaluator) passes.