Skip to content

ENG-30396: SQLancer for VoltDB (milestones 1 to 55) - #51

Open
jwiebema1 wants to merge 188 commits into
mainfrom
ENG-30396-sqlancer
Open

jwiebema1 wants to merge 188 commits into
mainfrom
ENG-30396-sqlancer

Conversation

@jwiebema1

@jwiebema1 jwiebema1 commented Nov 18, 2026 •

Copy link
Copy Markdown

Why

ENG-30396: a SQLancer provider for VoltDB, so that randomized testing finds VoltDB's logic bugs, crashes and hangs, nightly, against master's kit. This one PR replaces the 50 stacked milestone PRs (#1 to #50) and adds milestones 52 to 55, so the provider can be reviewed and merged as a whole.

The job side is VoltDB/internal (the Jenkins pipeline) and VoltDB/pro (the job script); see the companion PRs linked below.

What it adds

All under src/sqlancer/voltdb/ and voltdb/; nothing outside the VoltDB provider changes. 199 files, about 25,000 lines.

  • The provider. It generates random schemas (partitioned and replicated tables, indexes, materialized views, views over joins, stored procedures). It fills them with inserts, upserts, updates and deletes, and runs SQLancer's oracles through the VoltDB JDBC driver, connected to every node.
  • 38 oracles:
    • logic-bug oracles: NoREC, TLP (WHERE, DISTINCT, HAVING, AGGREGATE), PQS with an expression evaluator, PARAMS, LIMIT, SUBQUERY, WINDOW, SETOPS, DERIVED, VIEWS, PROCEDURE, SCHEMA, GEO, PLANS;
    • DIFF, against a mirror cluster on the last release;
    • DURABLE, across crashes;
    • XDCR convergence and XCONFLICT;
    • PERMS and LOGIN, with security on;
    • STREAMS and IMPORT (export and import);
    • TTL;
    • LOAD and SWAP;
    • CONFIG (@SetYamlConfiguration);
    • SNAPSHOT and STREAMSNAP (saved and streaming snapshots);
    • TASK (scheduled tasks);
    • STATS;
    • JAVAPROC (Java and compound stored procedures, @UpdateClasses);
    • METRICS (the metrics system);
    • DEFRAG (table compaction, @Defrag);
    • TLSRELOAD (certificate rotation, @ReloadSslContext);
    • SMOKE.
  • voltdb/run-sqlancer.sh. It starts a cluster of any shape on one machine, runs SQLancer against it and saves the results. While SQLancer runs, it can disturb the cluster:
    • node kills and graceful stops (--kill-interval, --graceful);
    • elastic resizes (--resize-interval);
    • whole-cluster crashes and recovery (--crash-interval);
    • an XDCR peer (--xdcr), also with kills;
    • an in-service upgrade from the last release (--upgrade-from);
    • security and TLS (--security, --ssl), and XDCR over TLS, DR connections included (--xdcr --ssl);
    • export targets (file, HTTP in several modes, JDBC, loopback) and the socket importer (--export);
    • a mirror on another kit (--mirror-voltdb).
  • voltdb/run-coverage.sh: Java (JaCoCo) and C++ (llvm-cov) coverage of the runs, compared with VoltDB's own sqlcoverage suite.
  • voltdb/repros/ and voltdb/run-repros.sh: a minimal reproduction (102 in all: SQL files, shell scripts and their Java programs) for the bugs found, each of which exits 0 while its bug is present. run-repros.sh runs them all and reports which are still broken.
  • --avoid-known-bugs: on by default, it keeps the nightly runs from failing on bugs already filed; false lets them find those bugs again.
  • README: src/sqlancer/voltdb/README.md covers every option, oracle and run mode, how to read results, the coverage numbers, and every known bug with its workaround.

Bugs found

110 VoltDB bugs filed under epic ENG-30580, each linked to ENG-30396, most with a reproduction in voltdb/repros/:

Priority Bugs
Blocker 68
Critical 2
Major 27
Minor 13

95 are still open. They include:

  • crashes during elastic joins and restores;
  • XDCR divergence;
  • lost and duplicated export rows;
  • streaming snapshots that complete early or hang;
  • many wrong query results.

The README's "Known bugs" tables list each one.

How to run

voltdb/run-sqlancer.sh --voltdb <kit> --nodes 3 --kfactor 1 --duration 600 --kill-interval 120

The README's "Run everything with one script" section covers every option. The nightly lab job (VoltDB/internal) runs seven cluster shapes against master's kit.

Testing

Replaces

#1 to #50 (milestones 1 to 51), which are closed in favour of this PR. Their descriptions keep each milestone's details and lab results.

Companion PRs

Adds a VoltDB provider that generates random replicated and partitioned
tables, indexes and single-row inserts through the VoltDB JDBC driver
(org.voltdb:voltdbclient 16.0.0, pinned in pom.xml). Its only oracle so
far, SMOKE, reads every table and checks that the queries succeed.

Each SQLancer database is the set of tables with its own name prefix,
since a VoltDB cluster has one schema. Statements VoltDB aborts because
another thread changed the schema are retried. The connection's close()
is idempotent to avoid ENG-30489. src/sqlancer/voltdb/README.md covers
running it, reading the results and VoltDB's intentional differences.
Adds a typed expression generator, the VoltDB AST and NoREC oracle wiring.
Joins between partitioned tables are on their partitioning columns.
--avoid-known-bugs (default true) keeps the generator away from VoltDB
bugs filed so far: ENG-30493, ENG-30494, ENG-30495, ENG-30496,
ENG-30497, ENG-30499, ENG-30500, ENG-30502, ENG-30503, ENG-30504.
…unds

Steers the expression generator away from the VoltDB bugs NoREC found, so
that a failure is something new (--avoid-known-bugs, default true):

- ENG-30504: NOT is never applied directly to OR
- ENG-30505, ENG-30509: string constants and LIKE literal prefixes are at
  most one character; strings are not compared with <, >, <=, >= or BETWEEN
- ENG-12656: VARBINARY constants are one byte, VARBINARY comparisons have a
  constant on one side and are never ordered
- ENG-30497: IS NULL operands are leaves with an outer join, on every path
- ENG-30506: function calls are printed in parentheses
- ENG-30507: no IN predicates with an outer join
- ENG-30508: the planner's internal error for a constant DECIMAL IN join
  filter is an expected error
- ENG-30510: no RIGHT JOIN

Partition joins use only the partitioning-column equality, which VoltDB
needs to see to plan them. DECIMAL overflow in folded constants is an
expected error. Adds a NoREC JUnit test and the milestone 2 README.
Workarounds for the bugs the 30-minute NoREC runs found, all under
--avoid-known-bugs:

- ENG-30511, ENG-30515: ordered numeric comparisons and BETWEEN compare
  with constants between -126 and 126
- ENG-30512: BETWEEN bounds are constants in ascending order
- ENG-30513: a numeric comparison, BETWEEN or IN uses only FLOAT or only
  exact operands
- ENG-30514: primary key columns are NOT NULL
- ENG-30516: IS NULL is written NOT (x IS NOT NULL)
- ENG-12656: VARBINARY partitioning columns are joined only when their
  sizes match

Also expects "data exception: division by zero" from folded constants.
A 30-minute NoREC run (4 threads, seed 20261102) is clean: 112,694
checks, no failures.
Adds TLP_WHERE: a query without a WHERE clause must return the same
rows as WHERE p, WHERE NOT p and WHERE p IS NULL together. The VoltDB
oracle combines the partitions with UNION ALL only over replicated
tables (VoltDB rejects set operations over partitioned tables) and
compares whole rows as multisets.

The connection now returns null from getString for a NULL value
(ENG-4643), and TLP's IS NULL partition is written NOT (p IS NOT NULL)
while avoiding known bugs (ENG-30516).
TLP negates a predicate that is often itself a NOT, and VoltDB's planner
fails on NOT (NOT (x OR y)) even when the OR is hidden behind a constant
the join planner folds away.
The oracle hands SQLancer a reproducer that re-runs the original query and
its partitions, for row mismatches and for unexpected errors. The
mismatch report comes from the reproducer and, after reduction, shows the
rows of the reduced database.
…gs (ENG-30504)

(NOT (K AND (x OR y))) IS NOT NULL fails once the planner folds K away, in
TLP's IS NULL partition of a predicate that starts with NOT. (NOT q) IS NULL
is q IS NULL, so the operand loses its leading NOTs.
…hains plannable

ENG-30521 (wrong rows): exact-domain comparisons get only integer literals
within BIGINT's range. ENG-30522: the parser's NumberFormatException for a
constant FLOAT overflow is an expected error. Explicit joins of replicated
tables are LEFT or FULL when partitioned tables are joined, since an INNER
chain loses the partitioning equality when an ON condition folds to FALSE.
voltdb/run-sqlancer.sh starts a local VoltDB cluster (any number of nodes,
k-safety, sites per host, its own ports), runs SQLancer for a set time with
a fixed seed, stops the cluster and saves one results directory: manifest,
SQLancer output and logs, every node's server log, the failing databases'
logs (reduced with --use-reducer) and a PASS/FAIL summary.

CompositeTestOracle now hands back the failing oracle's reproducer, so
--use-reducer works when several oracles run.
…0 is up

If another server took the admin port between the free-port check and
startup, a shutdown by port would reach it. The run's own processes are
always stopped by their directory.
…REGATE

Three more oracles. TLP_DISTINCT and TLP_HAVING are modes of the TLP oracle
(VoltDBTLPWhereOracle becomes VoltDBTLPOracle); TLP_AGGREGATE combines MIN,
MAX, SUM and COUNT over the three WHERE partitions in Java, since VoltDB
rejects the shared oracles' UNION ALL over partitioned tables.

The generator gains aggregates, grouping columns and HAVING predicates, and
works around ENG-30529 (aliases on grouped output), ENG-30530 (no grouping on
partitioning columns with an outer join) and ENG-30504 in HAVING (NOT pushed
down). A partition that returns no row is the empty aggregate (ENG-19957).
getObject() now returns null for a NULL value too (ENG-4643).
…output to 12 digits

The 30-minute runs found ENG-30531: GROUP BY or DISTINCT on an indexed
column, inlined into a FULL JOIN as a serial aggregation, returns the NULL
group twice. TLP DISTINCT and HAVING now make no FULL JOINs while
--avoid-known-bugs is on. A FLOAT SUM in a HAVING query's output can differ
in its last digit between queries, so FLOAT values compare to 12 digits.
…CIMAL precision errors

The second 30-minute runs found ENG-30533: MIN of a column without GROUP BY,
through a multi-column index whose next column is NULL, skips the row. TLP
AGGREGATE now writes MIN and MAX of a bare column as (c + 0) or (c || ''),
which the shortcut does not apply to.

Trials found ENG-30529 again where the grouped columns are not selected, so
grouping columns now have distinct names. With FULL JOINs gone from TLP
DISTINCT and HAVING, a LEFT JOIN of a replicated table after partitioned
joins could become "not plannable"; those generators keep replicated tables
in the FROM list there. "has more than 38 digits of precision" is DECIMAL
overflow in folded constants, an expected error.
…gates

The third 30-minute runs found ENG-30534: aggregates of x'' and of '' in a
GROUP BY join whose ON condition holds x'' fail with a cast error.
Pivoted Query Synthesis: pick a row of each table, evaluate random predicates
on it in Java, rewrite them to be TRUE for it, and check that the query returns
the row, with the evaluated values of random expressions in its SELECT list.

The evaluator models VoltDB's EE semantics as probed (BIGINT integer
arithmetic, DECIMAL truncation at scale 12, IEEE FLOAT, three-valued logic,
LIKE without escapes) and skips what it cannot predict: errors, NULL
sentinels, DECIMAL/FLOAT mixing.

Known bugs avoided while --avoid-known-bugs is on:
- ENG-30545: temporal comparisons draw operands from DATE or TIMESTAMP only
- ENG-30546: a concatenation's right operand is a non-empty constant
- ENG-30547: folded integer sums outside INTEGER's range are unpredictable
- ENG-30549: folded string comparisons decided by trailing spaces are
  unpredictable
README: the PQS oracle, its evaluator, and the bugs found while building it
(ENG-30545 to ENG-30550). run-sqlancer.sh lists every oracle in its help.
TestVoltDB gains a PQS test.
A subnormal FLOAT constant in a statement that is not parameterized makes its
plan fail to load, and re-running it after DDL crashes the node. The constant
generator never writes one, and PQS never sends an expression of constants
that folds to one.
The literal '\N' reaches VoltDB's EE as NULL. The constant generator never
writes it, and PQS never sends an expression of constants equal to it.
…tly lab job

An enterprise kit, which the lab runs, needs a license for voltdb init.
The first nightly lab run failed on three TLP HAVING databases. Two were
ENG-30560: a DATE constant makes the statement's DECIMAL literals FLOAT,
so temporal comparisons use TIMESTAMP constants only. The third was the
oracle's: MIN over 0.0 and -0.0 may return either, and the row text
differed. ENG-30561 was found while reducing ENG-30560.
The second lab run's 1-node half failed with "malformed bit string": a
HAVING clause held '\\' and a later LIKE pattern with /* in it, which
VoltDB stripped as a comment. INSERT stores such literals mangled, so
the constant generator avoids them unconditionally, as for ENG-30552.
… JOINed

The third lab run's 1-node half failed twice with "This query is not
plannable": a LEFT JOIN of a replicated table whose ON condition folds
to FALSE, with a WHERE clause on that table, loses the partitioning
columns' equality, as an INNER JOIN does. A FULL JOIN keeps it.
The fourth lab run's 1-node half found ENG-30497 again, through a later
FULL JOIN's ON condition: (t0.b OR t2.b) IS NOT NULL, with t2
null-extended by an earlier join. ON conditions now keep boolean
operands to columns and constants, as WHERE clauses with outer joins do.
The fifth lab run's 1-node half hit it again, through a FULL JOIN after
a LEFT JOIN of the partitioned tables, so making replicated joins FULL
(aa5dd2e, reverted here) was not enough. VoltDB plans a partitioned
join only when it sees the partitioning columns' equality, and several
outer-join shapes with an ON condition that folds to FALSE hide it.
The README lists it with the other expected errors.
Runs each query with its constants as literals and again with some of
them as PreparedStatement parameters, and requires the same rows. Its
trials found ENG-30571 (the JDBC driver counts a ? in a string literal)
and ENG-30572 (HAVING aggregates that differ only in their parameters
are merged), widened ENG-30560 and ENG-30534, and hit ENG-30551; each
is avoided while --avoid-known-bugs is on.
…a database

The oracles now check data that rows were updated, replaced and deleted
to reach. UPDATE sets one to three columns, never the partitioning one;
DELETE almost always has a WHERE clause; INSERT is sometimes UPSERT into
a table with a primary key, which the schema now reads from JDBC
metadata. The first trials found ENG-30494 and ENG-30561 again, through
UPDATE SET; both are avoided while --avoid-known-bugs is on.
This was referenced Nov 18, 2026
DEFRAG fragments a wide table of its own (one partition, 40 in-line columns, indexes
on a number, an out-of-line string and a unique key), calls @defrag on it or on
every table, full or 1 to 3 blocks, and then reads it in full and through each
index against the rows it should hold, again after changes through the indexes.

VoltDBConnection no longer retries executeBatch: after a batch fails the driver has
cleared it, and the retry ran an empty batch and reported success, so DEFRAG's model
counted rows that were never inserted (seen as rows missing after @defrag).
@jwiebema1 jwiebema1 changed the title ENG-30396: SQLancer VoltDB provider, its oracles and run scripts (milestones 1 to 52) ENG-30396: SQLancer VoltDB provider, its oracles and run scripts (milestones 1 to 53) Nov 18, 2026
With every oracle running, other databases' DDL rejects DEFRAG's insert batches
("planned against a catalog version"), and the coverage run's DEFRAG checks all gave
up. A rejected batch may have inserted some rows: DEFRAG now reads its model back
from the table and goes on filling, in batches of 50, up to 20 times.
…tone 54)

With TLS, TLSRELOAD calls @ReloadSslContext with nothing changed (refused), rotates
the certificate (new key pair, added to the truststore, keystore replaced) and
reloads, then requires every node's TLS handshake to present the new certificate and
the existing and new connections to work. Without --avoid-known-bugs it also reloads
a keystore that is no keystore, which must answer STATUS 2 (ENG-30745: the node
crashes).
@jwiebema1 jwiebema1 changed the title ENG-30396: SQLancer VoltDB provider, its oracles and run scripts (milestones 1 to 53) ENG-30396: SQLancer for VoltDB (milestones 1 to 54) Nov 19, 2026
run-sqlancer.sh's --ssl now combines with --xdcr: both clusters use the
run's certificate, DR runs over TLS (<ssl dr="true">), and the seed picks
the consumer connection's ssl= (a properties file or the server's
truststore) and ssl-host-check. The certificate names this machine's
addresses, which a DR producer advertises; TLSRELOAD's new certificates
keep them (cluster/tls/san).

TLSRELOAD reloads and checks both clusters, which read the same stores.
DEFRAG starts its model afresh for each database, as a reused name may
write through the other cluster. run-coverage.sh gets an xdcrssl run.

ENG-30747 (Minor): a failed DR TLS handshake is logged without its cause;
repros/ENG-30747.sh.
@jwiebema1 jwiebema1 changed the title ENG-30396: SQLancer for VoltDB (milestones 1 to 54) ENG-30396: SQLancer for VoltDB (milestones 1 to 55) Nov 19, 2026
…G-30708 workarounds

- run-sqlancer.sh: --xdcr now combines with --resize-interval; the first
  cluster grows and shrinks while the second keeps its nodes.
  run-coverage.sh gets an xresize run.
- ENG-30758 (found by the milestone 56 coverage baseline): a shrink left the
  MP repair blocker held for 128 s, refusing all DDL. @UpdateClasses
  refusals are now retried like DDL's; a statement still refused after
  --retry-seconds is skipped by TTL and JAVAPROC and counted in the summary,
  which also reports MP repair blockers held over 60 s.
- ENG-30708 (seen again in the first full XDCR resize trial, on SWAP's
  tables): while avoiding known bugs, @SwapTables waits during each grow
  (--join-hold).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant