Repository navigation
Conversation
Adds a VoltDB provider that generates random replicated and partitioned tables, indexes and single-row inserts through the VoltDB JDBC driver (org.voltdb:voltdbclient 16.0.0, pinned in pom.xml). Its only oracle so far, SMOKE, reads every table and checks that the queries succeed. Each SQLancer database is the set of tables with its own name prefix, since a VoltDB cluster has one schema. Statements VoltDB aborts because another thread changed the schema are retried. The connection's close() is idempotent to avoid ENG-30489. src/sqlancer/voltdb/README.md covers running it, reading the results and VoltDB's intentional differences.
Adds a typed expression generator, the VoltDB AST and NoREC oracle wiring. Joins between partitioned tables are on their partitioning columns. --avoid-known-bugs (default true) keeps the generator away from VoltDB bugs filed so far: ENG-30493, ENG-30494, ENG-30495, ENG-30496, ENG-30497, ENG-30499, ENG-30500, ENG-30502, ENG-30503, ENG-30504.
…unds Steers the expression generator away from the VoltDB bugs NoREC found, so that a failure is something new (--avoid-known-bugs, default true): - ENG-30504: NOT is never applied directly to OR - ENG-30505, ENG-30509: string constants and LIKE literal prefixes are at most one character; strings are not compared with <, >, <=, >= or BETWEEN - ENG-12656: VARBINARY constants are one byte, VARBINARY comparisons have a constant on one side and are never ordered - ENG-30497: IS NULL operands are leaves with an outer join, on every path - ENG-30506: function calls are printed in parentheses - ENG-30507: no IN predicates with an outer join - ENG-30508: the planner's internal error for a constant DECIMAL IN join filter is an expected error - ENG-30510: no RIGHT JOIN Partition joins use only the partitioning-column equality, which VoltDB needs to see to plan them. DECIMAL overflow in folded constants is an expected error. Adds a NoREC JUnit test and the milestone 2 README.
Workarounds for the bugs the 30-minute NoREC runs found, all under --avoid-known-bugs: - ENG-30511, ENG-30515: ordered numeric comparisons and BETWEEN compare with constants between -126 and 126 - ENG-30512: BETWEEN bounds are constants in ascending order - ENG-30513: a numeric comparison, BETWEEN or IN uses only FLOAT or only exact operands - ENG-30514: primary key columns are NOT NULL - ENG-30516: IS NULL is written NOT (x IS NOT NULL) - ENG-12656: VARBINARY partitioning columns are joined only when their sizes match Also expects "data exception: division by zero" from folded constants. A 30-minute NoREC run (4 threads, seed 20261102) is clean: 112,694 checks, no failures.
Adds TLP_WHERE: a query without a WHERE clause must return the same rows as WHERE p, WHERE NOT p and WHERE p IS NULL together. The VoltDB oracle combines the partitions with UNION ALL only over replicated tables (VoltDB rejects set operations over partitioned tables) and compares whole rows as multisets. The connection now returns null from getString for a NULL value (ENG-4643), and TLP's IS NULL partition is written NOT (p IS NOT NULL) while avoiding known bugs (ENG-30516).
TLP negates a predicate that is often itself a NOT, and VoltDB's planner fails on NOT (NOT (x OR y)) even when the OR is hidden behind a constant the join planner folds away.
The oracle hands SQLancer a reproducer that re-runs the original query and its partitions, for row mismatches and for unexpected errors. The mismatch report comes from the reproducer and, after reduction, shows the rows of the reduced database.
…gs (ENG-30504) (NOT (K AND (x OR y))) IS NOT NULL fails once the planner folds K away, in TLP's IS NULL partition of a predicate that starts with NOT. (NOT q) IS NULL is q IS NULL, so the operand loses its leading NOTs.
…hains plannable ENG-30521 (wrong rows): exact-domain comparisons get only integer literals within BIGINT's range. ENG-30522: the parser's NumberFormatException for a constant FLOAT overflow is an expected error. Explicit joins of replicated tables are LEFT or FULL when partitioned tables are joined, since an INNER chain loses the partitioning equality when an ON condition folds to FALSE.
voltdb/run-sqlancer.sh starts a local VoltDB cluster (any number of nodes, k-safety, sites per host, its own ports), runs SQLancer for a set time with a fixed seed, stops the cluster and saves one results directory: manifest, SQLancer output and logs, every node's server log, the failing databases' logs (reduced with --use-reducer) and a PASS/FAIL summary. CompositeTestOracle now hands back the failing oracle's reproducer, so --use-reducer works when several oracles run.
…0 is up If another server took the admin port between the free-port check and startup, a shutdown by port would reach it. The run's own processes are always stopped by their directory.
…REGATE Three more oracles. TLP_DISTINCT and TLP_HAVING are modes of the TLP oracle (VoltDBTLPWhereOracle becomes VoltDBTLPOracle); TLP_AGGREGATE combines MIN, MAX, SUM and COUNT over the three WHERE partitions in Java, since VoltDB rejects the shared oracles' UNION ALL over partitioned tables. The generator gains aggregates, grouping columns and HAVING predicates, and works around ENG-30529 (aliases on grouped output), ENG-30530 (no grouping on partitioning columns with an outer join) and ENG-30504 in HAVING (NOT pushed down). A partition that returns no row is the empty aggregate (ENG-19957). getObject() now returns null for a NULL value too (ENG-4643).
…output to 12 digits The 30-minute runs found ENG-30531: GROUP BY or DISTINCT on an indexed column, inlined into a FULL JOIN as a serial aggregation, returns the NULL group twice. TLP DISTINCT and HAVING now make no FULL JOINs while --avoid-known-bugs is on. A FLOAT SUM in a HAVING query's output can differ in its last digit between queries, so FLOAT values compare to 12 digits.
…CIMAL precision errors The second 30-minute runs found ENG-30533: MIN of a column without GROUP BY, through a multi-column index whose next column is NULL, skips the row. TLP AGGREGATE now writes MIN and MAX of a bare column as (c + 0) or (c || ''), which the shortcut does not apply to. Trials found ENG-30529 again where the grouped columns are not selected, so grouping columns now have distinct names. With FULL JOINs gone from TLP DISTINCT and HAVING, a LEFT JOIN of a replicated table after partitioned joins could become "not plannable"; those generators keep replicated tables in the FROM list there. "has more than 38 digits of precision" is DECIMAL overflow in folded constants, an expected error.
…gates The third 30-minute runs found ENG-30534: aggregates of x'' and of '' in a GROUP BY join whose ON condition holds x'' fail with a cast error.
Pivoted Query Synthesis: pick a row of each table, evaluate random predicates on it in Java, rewrite them to be TRUE for it, and check that the query returns the row, with the evaluated values of random expressions in its SELECT list. The evaluator models VoltDB's EE semantics as probed (BIGINT integer arithmetic, DECIMAL truncation at scale 12, IEEE FLOAT, three-valued logic, LIKE without escapes) and skips what it cannot predict: errors, NULL sentinels, DECIMAL/FLOAT mixing. Known bugs avoided while --avoid-known-bugs is on: - ENG-30545: temporal comparisons draw operands from DATE or TIMESTAMP only - ENG-30546: a concatenation's right operand is a non-empty constant - ENG-30547: folded integer sums outside INTEGER's range are unpredictable - ENG-30549: folded string comparisons decided by trailing spaces are unpredictable
README: the PQS oracle, its evaluator, and the bugs found while building it (ENG-30545 to ENG-30550). run-sqlancer.sh lists every oracle in its help. TestVoltDB gains a PQS test.
A subnormal FLOAT constant in a statement that is not parameterized makes its plan fail to load, and re-running it after DDL crashes the node. The constant generator never writes one, and PQS never sends an expression of constants that folds to one.
The literal '\N' reaches VoltDB's EE as NULL. The constant generator never writes it, and PQS never sends an expression of constants equal to it.
…tly lab job An enterprise kit, which the lab runs, needs a license for voltdb init.
The first nightly lab run failed on three TLP HAVING databases. Two were ENG-30560: a DATE constant makes the statement's DECIMAL literals FLOAT, so temporal comparisons use TIMESTAMP constants only. The third was the oracle's: MIN over 0.0 and -0.0 may return either, and the row text differed. ENG-30561 was found while reducing ENG-30560.
The second lab run's 1-node half failed with "malformed bit string": a HAVING clause held '\\' and a later LIKE pattern with /* in it, which VoltDB stripped as a comment. INSERT stores such literals mangled, so the constant generator avoids them unconditionally, as for ENG-30552.
… JOINed The third lab run's 1-node half failed twice with "This query is not plannable": a LEFT JOIN of a replicated table whose ON condition folds to FALSE, with a WHERE clause on that table, loses the partitioning columns' equality, as an INNER JOIN does. A FULL JOIN keeps it.
The fourth lab run's 1-node half found ENG-30497 again, through a later FULL JOIN's ON condition: (t0.b OR t2.b) IS NOT NULL, with t2 null-extended by an earlier join. ON conditions now keep boolean operands to columns and constants, as WHERE clauses with outer joins do.
The fifth lab run's 1-node half hit it again, through a FULL JOIN after a LEFT JOIN of the partitioned tables, so making replicated joins FULL (aa5dd2e, reverted here) was not enough. VoltDB plans a partitioned join only when it sees the partitioning columns' equality, and several outer-join shapes with an ON condition that folds to FALSE hide it. The README lists it with the other expected errors.
Runs each query with its constants as literals and again with some of them as PreparedStatement parameters, and requires the same rows. Its trials found ENG-30571 (the JDBC driver counts a ? in a string literal) and ENG-30572 (HAVING aggregates that differ only in their parameters are merged), widened ENG-30560 and ENG-30534, and hit ENG-30551; each is avoided while --avoid-known-bugs is on.
…a database The oracles now check data that rows were updated, replaced and deleted to reach. UPDATE sets one to three columns, never the partitioning one; DELETE almost always has a WHERE clause; INSERT is sometimes UPSERT into a table with a primary key, which the schema now reads from JDBC metadata. The first trials found ENG-30494 and ENG-30561 again, through UPDATE SET; both are avoided while --avoid-known-bugs is on.
This was referenced Nov 18, 2026
Closed
DEFRAG fragments a wide table of its own (one partition, 40 in-line columns, indexes on a number, an out-of-line string and a unique key), calls @defrag on it or on every table, full or 1 to 3 blocks, and then reads it in full and through each index against the rows it should hold, again after changes through the indexes. VoltDBConnection no longer retries executeBatch: after a batch fails the driver has cleared it, and the retry ran an empty batch and reported success, so DEFRAG's model counted rows that were never inserted (seen as rows missing after @defrag).
With every oracle running, other databases' DDL rejects DEFRAG's insert batches
("planned against a catalog version"), and the coverage run's DEFRAG checks all gave
up. A rejected batch may have inserted some rows: DEFRAG now reads its model back
from the table and goes on filling, in batches of 50, up to 20 times.
…ode on a keystore it cannot load)
…tone 54) With TLS, TLSRELOAD calls @ReloadSslContext with nothing changed (refused), rotates the certificate (new key pair, added to the truststore, keystore replaced) and reloads, then requires every node's TLS handshake to present the new certificate and the existing and new connections to work. Without --avoid-known-bugs it also reloads a keystore that is no keystore, which must answer STATUS 2 (ENG-30745: the node crashes).
run-sqlancer.sh's --ssl now combines with --xdcr: both clusters use the run's certificate, DR runs over TLS (<ssl dr="true">), and the seed picks the consumer connection's ssl= (a properties file or the server's truststore) and ssl-host-check. The certificate names this machine's addresses, which a DR producer advertises; TLSRELOAD's new certificates keep them (cluster/tls/san). TLSRELOAD reloads and checks both clusters, which read the same stores. DEFRAG starts its model afresh for each database, as a reused name may write through the other cluster. run-coverage.sh gets an xdcrssl run. ENG-30747 (Minor): a failed DR TLS handshake is logged without its cause; repros/ENG-30747.sh.
…G-30708 workarounds - run-sqlancer.sh: --xdcr now combines with --resize-interval; the first cluster grows and shrinks while the second keeps its nodes. run-coverage.sh gets an xresize run. - ENG-30758 (found by the milestone 56 coverage baseline): a shrink left the MP repair blocker held for 128 s, refusing all DDL. @UpdateClasses refusals are now retried like DDL's; a statement still refused after --retry-seconds is skipped by TTL and JAVAPROC and counted in the summary, which also reports MP repair blockers held over 60 s. - ENG-30708 (seen again in the first full XDCR resize trial, on SWAP's tables): while avoiding known bugs, @SwapTables waits during each grow (--join-hold).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
ENG-30396: a SQLancer provider for VoltDB, so that randomized testing finds VoltDB's logic bugs, crashes and hangs, nightly, against master's kit. This one PR replaces the 50 stacked milestone PRs (#1 to #50) and adds milestones 52 to 55, so the provider can be reviewed and merged as a whole.
The job side is VoltDB/internal (the Jenkins pipeline) and VoltDB/pro (the job script); see the companion PRs linked below.
What it adds
All under
src/sqlancer/voltdb/andvoltdb/; nothing outside the VoltDB provider changes. 199 files, about 25,000 lines.@SetYamlConfiguration);@UpdateClasses);@Defrag);@ReloadSslContext);voltdb/run-sqlancer.sh. It starts a cluster of any shape on one machine, runs SQLancer against it and saves the results. While SQLancer runs, it can disturb the cluster:--kill-interval,--graceful);--resize-interval);--crash-interval);--xdcr), also with kills;--upgrade-from);--security,--ssl), and XDCR over TLS, DR connections included (--xdcr --ssl);--export);--mirror-voltdb).voltdb/run-coverage.sh: Java (JaCoCo) and C++ (llvm-cov) coverage of the runs, compared with VoltDB's own sqlcoverage suite.voltdb/repros/andvoltdb/run-repros.sh: a minimal reproduction (102 in all: SQL files, shell scripts and their Java programs) for the bugs found, each of which exits 0 while its bug is present.run-repros.shruns them all and reports which are still broken.--avoid-known-bugs: on by default, it keeps the nightly runs from failing on bugs already filed;falselets them find those bugs again.src/sqlancer/voltdb/README.mdcovers every option, oracle and run mode, how to read results, the coverage numbers, and every known bug with its workaround.Bugs found
110 VoltDB bugs filed under epic ENG-30580, each linked to ENG-30396, most with a reproduction in
voltdb/repros/:95 are still open. They include:
The README's "Known bugs" tables list each one.
How to run
The README's "Run everything with one script" section covers every option. The nightly lab job (VoltDB/internal) runs seven cluster shapes against master's kit.
Testing
ENG-30396-sqlancer-nightlyofmatrix-systemtest-apprunner-sqlancer, from build ENG-30396: VoltDB provider, milestone 1 (schemas and data) #1 to build Test CockroachDB both with NoREC and TLP sqlancer/sqlancer#72.mvn verify -DskipTestspasses (formatter, checkstyle, PMD).Replaces
#1 to #50 (milestones 1 to 51), which are closed in favour of this PR. Their descriptions keep each milestone's details and lab results.
Companion PRs