Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 38 additions & 3 deletions src/sqlancer/voltdb/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ A SQLancer provider for [VoltDB](https://www.voltactivedata.com/), developed for
VoltDB schemas, both replicated and partitioned tables, fills them with single-row inserts, upserts, updates and deletes, and runs SQLancer's oracles
against them through the VoltDB JDBC driver.

**Status: milestone 38.** The provider creates schemas and data and runs these logic-bug oracles: NoREC (the default,
**Status: milestone 39.** The provider creates schemas and data and runs these logic-bug oracles: NoREC (the default,
milestone 2), TLP WHERE (milestone 3), TLP DISTINCT, TLP HAVING and TLP AGGREGATE (milestone 5), PQS (milestone 6), and
PARAMS, literals against JDBC parameters (milestone 8). Since milestone 9 each database's rows are also updated,
upserted and deleted before the oracles run. Since milestone 10 each database also has materialized views, which the
Expand Down Expand Up @@ -50,7 +50,8 @@ Since milestone 36 the CONFIG oracle changes the configuration while the databas
[The CONFIG oracle](#the-config-oracle). Since milestone 37 the SNAPSHOT oracle saves and restores a table by
snapshot; see [The SNAPSHOT oracle](#the-snapshot-oracle). Since milestone 38 the XCONFLICT oracle makes XDCR
conflicts on purpose and checks that every divergence they leave is reported; see
[The XCONFLICT oracle](#the-xconflict-oracle).
[The XCONFLICT oracle](#the-xconflict-oracle). Since milestone 39 the TASK oracle runs scheduled tasks and checks that
they run as scheduled and stop when disabled; see [The TASK oracle](#the-task-oracle).
`SMOKE`, from
milestone 1, only reads every table and checks that the queries succeed. `voltdb/run-sqlancer.sh` (milestone 4) starts a
cluster, runs SQLancer against it and saves the results. A nightly lab job (milestone 7) runs it against master's kit;
Expand Down Expand Up @@ -460,7 +461,7 @@ the kit's runtime JDK and makes seven runs, one after the other, every node on t

| Run (triagedb name) | Cluster | Oracles | Time |
| --- | --- | --- | --- |
| `sqlancer-3n-k1-kill300` | 3 nodes, k=1, 8 sites per host; a node is killed and rejoined every 5 minutes | all twenty-eight (NoREC, TLP WHERE, DISTINCT, HAVING, AGGREGATE, PQS, PARAMS, VIEWS, LIMIT, SUBQUERY, WINDOW, SETOPS, DERIVED, PROCEDURE, SCHEMA, GEO, DIFF, DURABLE, XDCR, PLANS, PERMS, STREAMS, TTL, LOAD, SWAP, CONFIG, SNAPSHOT, XCONFLICT; DURABLE checks across crashes in the crash run, XDCR only in the XDCR run) | 45 minutes |
| `sqlancer-3n-k1-kill300` | 3 nodes, k=1, 8 sites per host; a node is killed and rejoined every 5 minutes | all twenty-nine (NoREC, TLP WHERE, DISTINCT, HAVING, AGGREGATE, PQS, PARAMS, VIEWS, LIMIT, SUBQUERY, WINDOW, SETOPS, DERIVED, PROCEDURE, SCHEMA, GEO, DIFF, DURABLE, XDCR, PLANS, PERMS, STREAMS, TTL, LOAD, SWAP, CONFIG, SNAPSHOT, XCONFLICT, TASK; DURABLE checks across crashes in the crash run, XDCR only in the XDCR run) | 45 minutes |
| `sqlancer-1n-k0-security` | 1 node, 8 sites per host, security on (see [The PERMS oracle](#the-perms-oracle)); `sqlancer-1n-k0` until milestone 28 | the same | 45 minutes |
| `sqlancer-3n-k2-export-s2` | 3 nodes, k=2, 2 sites per host: every partition on every node, and 2 partitions in all; the streams export (see [The STREAMS oracle](#the-streams-oracle)); `sqlancer-3n-k2-s2` until milestone 29 | the same | 45 minutes |
| `sqlancer-3n-k1-resize120` | 3 nodes, k=1, 8 sites per host; every 2 minutes, 2 nodes are added, or removed again (see [Elastic resizes](#elastic-resizes)) | the same | 45 minutes |
Expand Down Expand Up @@ -1125,6 +1126,34 @@ unexercised. In an XDCR run (`--xdcr`), each XCONFLICT check:
converged and how many differing keys were reported as diverged. In a 5-minute trial on two 3-node clusters (k=1,
4 sites per host), beside XDCR and NoREC: 719 checks, 716 converged.

## The TASK oracle

Milestone 39. Milestone 30's coverage run found scheduled tasks (`CREATE TASK`) little exercised: the scheduler
(`TaskManager`), its statistics (`TaskStatsSource`) and the CRON schedule. Each TASK check creates a task of its own,
`XKT_DB0_<n>`, whose procedure inserts a row tagged with the task's name into `XK_DB0` (partitioned on `k`):

- on the database (`XKP_DB0`, a multi-partition insert), or on every partition (`RUN ON PARTITIONS`, through the
partitioned `XKQ_DB0`, so that each partition's scheduler calls it);
- on a schedule of `EVERY` or `DELAY` 200, 500 or 1000 milliseconds, or `CRON * * * * * *` (every second);
- `ON ERROR LOG` or `IGNORE`.

It lets the task run for 3 s and disables it (`ALTER TASK ... DISABLE`), then checks that:

1. it ran: at least one row, or two on every partition, and no more than the schedule allows in the time it really ran,
from `CREATE TASK` until the `ALTER` returned (`ALTER TASK` is DDL, which waits while a node is killed);
2. after the `ALTER`, at most one more row per scheduler (one in flight), and then none for a few seconds;
3. `@Statistics TASK` shows it `DISABLED`.

A third of the checks instead run a task whose procedure fails from its second run on (a duplicate primary key in the
replicated `XKS_DB0`) with `ON ERROR STOP`: it must stop at the first failure, with at most one row, and show `ERROR`
in `@Statistics TASK`. Every task is dropped at the end of its check. A failure lists the times of the task's rows.

While `--avoid-known-bugs` is true and nodes fail, resize, crash or upgrade, a task may run twice as often as its
schedule allows, and two more rows per scheduler may follow `ALTER TASK ... DISABLE` (ENG-30680 below).

In two 5-minute trials on 3 nodes (k=1, 4 sites per host) with a node killed and rejoined every 45 s: 173 checks, 61 of
them of a task stopped by an error.

## The PERMS oracle

With `--security`, the deployment enables security with three users, `sqlancer` (administrator, whom SQLancer and
Expand Down Expand Up @@ -1674,6 +1703,12 @@ The bugs below were found in milestone 29's lab run of the milestones before it.
| --- | --- | --- |
| ENG-30628 | A LEFT JOIN of a partitioned table P whose WHERE clause compares a scalar subquery with P's primary key (its partitioning column) fails to plan: "SQL error while compiling query: java.lang.IllegalStateException" (`SubPlanAssembler.buildIndexAccessPlanForTable`). The inner join, and the WHERE clause on P alone, plan. Also on 16.0.0. Found by TLP AGGREGATE in lab build #41 (the upgrade run, after the upgrade). | No scalar subqueries in a query with an outer join. |

The bug below was found in milestone 39.

| Ticket | Issue | Workaround |
| --- | --- | --- |
| ENG-30680 | A task on the database (`DELAY 20 MILLISECONDS`, a multi-partition procedure, `ON ERROR LOG`) runs in two scheduling chains, about twice as often as its schedule allows, after the node of the multi-partition initiator's leader is killed while one of its calls is in flight; it lasts until the task is disabled or restarted. `@Statistics TASK` counts only one chain, and its `PROCEDURE_*` times go to about ±2.4e18. 16.0.0 too. Found by TASK in a 3-node trial with kills; `repros/ENG-30680.sh` kills the MP leader's node once and finds half the gaps between the task's rows under 20 ms. Extra writes, Blocker for V16.1.0. | While `--avoid-known-bugs` is true and nodes fail, resize, crash or upgrade, TASK allows twice the runs |

The bug below was found in milestone 38.

| Ticket | Issue | Workaround |
Expand Down
11 changes: 11 additions & 0 deletions src/sqlancer/voltdb/VoltDBOracleFactory.java
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import sqlancer.voltdb.oracle.VoltDBSubqueryOracle;
import sqlancer.voltdb.oracle.VoltDBTLPAggregateOracle;
import sqlancer.voltdb.oracle.VoltDBTLPOracle;
import sqlancer.voltdb.oracle.VoltDBTaskOracle;
import sqlancer.voltdb.oracle.VoltDBTtlOracle;
import sqlancer.voltdb.oracle.VoltDBViewOracle;
import sqlancer.voltdb.oracle.VoltDBWindowOracle;
Expand Down Expand Up @@ -219,6 +220,16 @@ public TestOracle<VoltDBGlobalState> create(VoltDBGlobalState globalState) {
return new VoltDBXdcrConflictOracle(globalState);
}
},
/**
* Scheduled tasks (milestone 39): a task of each schedule (EVERY, DELAY, CRON; on the database or on every
* partition) must run as often as scheduled and stop when disabled, and ON ERROR STOP must stop it.
*/
TASK {
@Override
public TestOracle<VoltDBGlobalState> create(VoltDBGlobalState globalState) {
return new VoltDBTaskOracle(globalState);
}
},

/**
* Permissions, with --security: the users reader (SQLREAD) and writer (SQL) may run what their roles allow and
Expand Down
256 changes: 256 additions & 0 deletions src/sqlancer/voltdb/oracle/VoltDBTaskOracle.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,256 @@
package sqlancer.voltdb.oracle;

import java.sql.CallableStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicLong;

import sqlancer.IgnoreMeException;
import sqlancer.Randomly;
import sqlancer.Reproducer;
import sqlancer.common.oracle.TestOracle;
import sqlancer.voltdb.VoltDBConnection;
import sqlancer.voltdb.VoltDBOptions;
import sqlancer.voltdb.VoltDBProvider.VoltDBGlobalState;

/**
* Scheduled tasks (milestone 39), which milestone 30's coverage run found little exercised ({@code TaskManager},
* {@code TaskStatsSource}, the CRON schedule). Each check creates a task of its own that runs a procedure inserting a
* row tagged with the task's name into {@code XK_<db>} (partitioned on k): on the database, or on every partition (RUN
* ON PARTITIONS: each partition's scheduler calls a partitioned procedure); lets it run, disables it, and checks:
* <ul>
* <li>it ran: at least one row (two for a task on every partition), and no more than its schedule allows;</li>
* <li>once {@code ALTER TASK ... DISABLE} has returned, at most one more row per scheduler (one in flight), and none
* after that;</li>
* <li>{@code @Statistics TASK} shows it DISABLED.</li>
* </ul>
* A third of the checks instead run a task whose procedure fails (a duplicate primary key) with {@code ON ERROR STOP}:
* it must stop after the first failure, at most one row in all, and show ERROR in {@code @Statistics TASK}. The task is
* dropped at the end.
* <p>
* Timing on a loaded machine varies, so the bounds are loose: a task that does not run at all, runs on after being
* disabled or stopped, or runs far more often than scheduled, fails the check.
*/
public class VoltDBTaskOracle implements TestOracle<VoltDBGlobalState> {

private static final AtomicLong IDS = new AtomicLong();
private static final AtomicLong CHECKS = new AtomicLong();
private static final AtomicLong STOPS = new AtomicLong();
private static final Set<String> CREATED = ConcurrentHashMap.newKeySet();

static {
Runtime.getRuntime().addShutdownHook(new Thread(
() -> System.out.println("TASK checks: " + CHECKS + "; tasks stopped by an error checked: " + STOPS)));
}

private final VoltDBGlobalState state;
private String lastQuery;

public VoltDBTaskOracle(VoltDBGlobalState state) {
this.state = state;
}

@Override
public void check() throws Exception {
String db = state.getDatabaseName().toUpperCase();
String table = "XK_" + db;
create(table, db);
String task = "XKT_" + db + "_" + IDS.incrementAndGet();
if (Randomly.fromOptions(true, false, false)) {
stopOnError(db, task);
return;
}
boolean partitions = Randomly.getBoolean();
long millis;
String schedule;
switch ((int) Randomly.getNotCachedInteger(0, 3)) {
case 0:
millis = Randomly.fromOptions(200L, 500L, 1000L);
schedule = "EVERY " + millis + " MILLISECONDS";
break;
case 1:
millis = Randomly.fromOptions(200L, 500L, 1000L);
schedule = "DELAY " + millis + " MILLISECONDS";
break;
default:
millis = 1000;
schedule = "CRON * * * * * *";
break;
}
// on every partition, the partitioned procedure: each partition's scheduler calls it (with the same k, so
// the rows all go to that k's partition)
String call = partitions ? "XKQ_" + db + " WITH (" + Randomly.getNotCachedInteger(0, 1000) + ", '" + task + "')"
: "XKP_" + db + " WITH ('" + task + "')";
// ON ERROR comes before RUN ON
ddl("CREATE TASK " + task + " ON SCHEDULE " + schedule + " PROCEDURE " + call + " ON ERROR "
+ Randomly.fromOptions("LOG", "IGNORE") + (partitions ? " RUN ON PARTITIONS" : ""));
long created = System.currentTimeMillis();
try {
Thread.sleep(3000);
ddl("ALTER TASK " + task + " DISABLE");
// the time it really ran: ALTER TASK is DDL, which waits while a node is killed (--ddl-hold)
long runMillis = System.currentTimeMillis() - created;
long ran = count(table, task);
int schedulers = partitions ? partitions() : 1;
// ENG-30680: after a leader's node fails while the task's procedure is in flight, the task runs in two
// chains; while avoiding known bugs, where nodes fail, it may run twice as often (ALTER TASK DISABLE still
// stops both)
VoltDBOptions o = state.getDbmsSpecificOptions();
int chains = o.avoidKnownBugs && (o.nodeFailures || o.resizes || o.crashes || o.upgrades) ? 2 : 1;
long most = chains * schedulers * (runMillis / millis + 3);
if (ran < Math.min(schedulers, 2) || ran > most) {
throw new AssertionError("task " + task + " (" + schedule + (partitions ? ", RUN ON PARTITIONS" : "")
+ ") inserted " + ran + " rows in " + runMillis + " ms (at " + times(table, task)
+ "); expected " + Math.min(schedulers, 2) + " to " + most);
}
Thread.sleep(Math.max(1500, 2 * millis));
long after = count(table, task);
Thread.sleep(Math.max(1500, 2 * millis));
long later = count(table, task);
if (after > ran + chains * schedulers || later != after) {
throw new AssertionError("task " + task + " (" + schedule + ") went on after ALTER TASK DISABLE: " + ran
+ " rows when disabled, " + after + " and then " + later + " later");
}
String status = status(task);
if (status != null && !"DISABLED".equals(status)) {
throw new AssertionError(
"@Statistics TASK shows task " + task + " as " + status + " after ALTER TASK " + "DISABLE");
}
CHECKS.incrementAndGet();
} finally {
dropQuietly(task);
}
}

// a task whose procedure fails every time it runs (it inserts the same primary key again), ON ERROR STOP
private void stopOnError(String db, String task) throws Exception {
ddl("CREATE TASK " + task + " ON SCHEDULE EVERY 200 MILLISECONDS PROCEDURE XKU_" + db + " WITH ("
+ IDS.incrementAndGet() + ", '" + task + "') ON ERROR STOP");
try {
Thread.sleep(3000);
long rows = count("XKS_" + db, task);
Thread.sleep(1500);
long later = count("XKS_" + db, task);
String status = status(task);
if (rows > 1 || later != rows || status != null && !"ERROR".equals(status)) {
throw new AssertionError("task " + task + " ON ERROR STOP, whose procedure fails from its second run "
+ "on, should stop: " + rows + " rows after 3 s, " + later + " after 4.5 s, @Statistics TASK "
+ "state " + status);
}
STOPS.incrementAndGet();
} finally {
dropQuietly(task);
}
}

// the table and its procedures, once per database name in the run
private void create(String table, String db) {
if (CREATED.contains(table)) {
return;
}
ddl("CREATE TABLE " + table + " (k BIGINT NOT NULL, t VARCHAR(40) NOT NULL, ts TIMESTAMP); PARTITION TABLE "
+ table + " ON COLUMN k");
// a row on the database (a multi-partition procedure), k from the time
ddl("CREATE PROCEDURE XKP_" + db + " AS INSERT INTO " + table + " VALUES (SINCE_EPOCH(MICROS, NOW), ?, NOW)");
// a row through k's partition
ddl("CREATE PROCEDURE XKQ_" + db + " PARTITION ON TABLE " + table + " COLUMN k AS INSERT INTO " + table
+ " VALUES (?, ?, NOW)");
// a replicated table with a primary key, and a procedure whose second run with the same k fails on it
ddl("CREATE TABLE XKS_" + db + " (k BIGINT NOT NULL, t VARCHAR(40) NOT NULL, PRIMARY KEY (k))");
ddl("CREATE PROCEDURE XKU_" + db + " AS INSERT INTO XKS_" + db + " VALUES (?, ?)");
CREATED.add(table);
}

private int partitions() throws SQLException {
lastQuery = "@GetPartitionKeys('INTEGER')";
int n = 0;
try (CallableStatement call = connection().prepareCall("{call @GetPartitionKeys(?)}")) {
call.setString(1, "INTEGER");
try (ResultSet rs = call.executeQuery()) {
while (rs.next()) {
n++;
}
}
}
return n;
}

// the task's state in @Statistics TASK (one row per scheduler; the first), or null if it is not shown
private String status(String task) throws SQLException {
lastQuery = "@Statistics('TASK', 0)";
try (CallableStatement call = connection().prepareCall("{call @Statistics(?, ?)}")) {
call.setString(1, "TASK");
call.setInt(2, 0);
try (ResultSet rs = call.executeQuery()) {
while (rs.next()) {
if (task.equalsIgnoreCase(rs.getString("TASK_NAME"))) {
return rs.getString("STATE");
}
}
}
}
return null;
}

// the times of the task's rows, for a report
private String times(String table, String task) throws SQLException {
String sql = "SELECT ts FROM " + table + " WHERE t = '" + task + "' ORDER BY ts";
StringBuilder sb = new StringBuilder();
try (Statement st = connection().createStatement(); ResultSet rs = st.executeQuery(sql)) {
while (rs.next()) {
String t = String.valueOf(rs.getTimestamp(1));
sb.append(sb.length() == 0 ? "" : ", ").append(t.length() > 11 ? t.substring(11) : t);
}
}
return sb.toString();
}

private long count(String table, String task) throws SQLException {
String sql = "SELECT COUNT(*) FROM " + table + " WHERE t = '" + task + "'";
lastQuery = sql;
try (Statement st = connection().createStatement(); ResultSet rs = st.executeQuery(sql)) {
rs.next();
return rs.getLong(1);
} catch (SQLException e) {
throw new IgnoreMeException();
}
}

private VoltDBConnection connection() {
return (VoltDBConnection) state.getConnection();
}

// DDL that may already have been done for this database name earlier in the run
private void ddl(String sql) {
lastQuery = sql;
try (Statement st = state.getConnection().createStatement()) {
st.execute(sql);
} catch (SQLException e) {
if (!String.valueOf(e.getMessage()).contains("already exists")) {
throw new AssertionError(sql, e);
}
}
}

private void dropQuietly(String task) {
try (Statement st = state.getConnection().createStatement()) {
st.execute("DROP TASK " + task + " IF EXISTS");
} catch (SQLException | IgnoreMeException e) {
// the next check uses another name
}
}

@Override
public Reproducer<VoltDBGlobalState> getLastReproducer() {
return null;
}

@Override
public String getLastQueryString() {
return lastQuery;
}

}
Loading