Skip to content

ENG-30396: the TASK oracle, scheduled tasks, and ENG-30680 (milestone 39) - #38

Closed
jwiebema1 wants to merge 1 commit into
ENG-30396-sqlancer-m38from
ENG-30396-sqlancer-m39
Closed

jwiebema1 wants to merge 1 commit into
ENG-30396-sqlancer-m38from
ENG-30396-sqlancer-m39

Conversation

@jwiebema1

Copy link
Copy Markdown

Milestone 39 of ENG-30396 (tracked in ENG-30581, epic ENG-30580 "SQL Housekeeping"). Stacked on #37; review that and the PRs below it first.

Why

Milestone 30's coverage run found scheduled tasks (CREATE TASK) exercised little: the scheduler (TaskManager) and its schedules ran only at startup.

What it changes

  • The TASK oracle. Each check creates a task of its own and lets it run.
    • The task is EVERY, DELAY or CRON; on the database or RUN ON PARTITIONS; ON ERROR LOG or IGNORE.
  • What a check verifies. After ALTER TASK DISABLE:
    • the task ran within its schedule's bounds;
    • it stopped running;
    • @Statistics TASK shows it as DISABLED.
  • Failing tasks. A third of the checks run a failing procedure ON ERROR STOP and expect the task to show ERROR.
  • Runners. The oracle is registered as TASK and counted in run-sqlancer.sh's summary and run-coverage.sh's oracle list.

Bugs found

Ticket Issue Workaround while --avoid-known-bugs is true
ENG-30680 After the MP leader's node fails while a task's procedure is in flight, a task on the database runs its procedure twice per schedule, for good. @Statistics TASK misses the second chain, and its PROCEDURE_* times go to ±2.4e18. Reproduction: voltdb/repros/ENG-30680.sh (kill host 1 with a 20 ms delay). Blocker. In runs where nodes fail, the bounds allow twice the runs.

Coverage

Milestone 41's coverage run, which included TASK: TaskManager 281 of 412 lines.

Testing

  • Local trials: 3 nodes, k=1, with kills.
    • With a 500 ms delay and random kills, ENG-30680 did not reproduce.
    • Killing the MP leader with a 20 ms delay reproduced it every time.
  • mvn verify passes.
  • Lab: builds Add random seed to log sqlancer/sqlancer#56 and Log the random seed value sqlancer/sqlancer#57 of matrix-systemtest-apprunner-sqlancer (branch job ENG-30396-sqlancer-nightly, replays with these settings). They ran 16.1.0-SNAPSHOT (master kit 400) beside one node of 16.0.0, 45 minutes per run, with all thirty-one oracles.
    • Build Add random seed to log sqlancer/sqlancer#56 ran fork 585fbb8, milestones 34 to 43 before the last fix.
      • Reproductions: 84 still broken. ENG-30611 reports fixed, and the ticket is closed as fixed in V16.1.0.
      • Kill (3 nodes, k=1, a kill every 300 s): 8 kills, 8 rejoins.
      • Security (1 node, over TLS): PERMS checked 153 allowed and 174 refused statements; TASK made 286 checks.
      • Export with a kill every 300 s (3 nodes, k=2): all 4,617 rows exported, 1,240 through HTTP and 1,120 through JDBC (one twice).
      • Resize every 120 s: 10 grows, 9 shrinks.
      • XDCR: 340 XCONFLICT checks and 115 databases compared after convergence. 24 differing keys had no divergence logged; that is ENG-30679, counted rather than failed.
      • Crash every 600 s: 4 crashes, 4 recoveries.
      • Upgrade from 16.0.0 failed. A node being stopped for its upgrade refused DDL. The next fork commit handles this; see milestone 43.
    • Build Log the random seed value sqlancer/sqlancer#57 ran the upgrade run alone on fork e0cadfe, the head: 3 of 3 nodes upgraded, 414 TASK checks, 612 STATS calls and 15,923 primary key indexes checked.
    • Earlier builds: Bump junit-jupiter-engine from 5.4.0 to 5.6.2 sqlancer/sqlancer#53 to Bump maven-surefire-plugin from 2.19.1 to 2.22.2 sqlancer/sqlancer#55 ran earlier heads of this stack. Their failures led to ENG-30682, ENG-30683, the ENG-30679 changes, the run-repros.sh port wait and TASK's timing fix.

Each check creates a task of its own (EVERY, DELAY or CRON; on the database or
RUN ON PARTITIONS; ON ERROR LOG or IGNORE), lets it run, disables it and checks
that it ran within its schedule's bounds, stopped after ALTER TASK DISABLE and
shows DISABLED in @statistics TASK; a third of the checks run a failing
procedure ON ERROR STOP and expect ERROR. Registered as TASK, counted in
run-sqlancer.sh's summary and run-coverage.sh's oracle list.

ENG-30680 (a database task runs in two chains after the MP leader's node fails
with a call in flight): repros/ENG-30680.sh, and while avoiding known bugs in
runs where nodes fail, the bounds allow twice the runs.
@jwiebema1

Copy link
Copy Markdown
Author

Superseded by #51, which combines every ENG-30396 PR of this repo into one. This PR's branch is kept, and its description still has this milestone's details and lab results.

@jwiebema1 jwiebema1 closed this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant