feat: migrate Global Async Queries onto the Global Task Framework

[villebro]

SUMMARY

Before this epic, Superset had three async execution paths:

• Global Async Queries (GAQ) for dashboard/chart data, built around a bespoke Celery task, Redis-stream events, AsyncQueryManager, qc-<hash> descriptors, and result_url result reassembly.
• SQL Lab async query execution, which remains on its existing path and is intentionally deferred to a later migration.
• Global Task Framework (GTF), the newer shared @task / .schedule() abstraction with a tasks table, deduplication, DAG dependencies, progress, timeouts, cancellation, a Task List UI, and a REST API.

This epic migrates GAQ chart-data execution onto GTF and removes GAQ's dedicated plumbing. GLOBAL_ASYNC_QUERIES remains the operator switch for whether chart-data requests may run asynchronously, but the execution, visibility, deduplication, and completion tracking now use GTF task records and APIs.

The core architectural change is the async task boundary. GAQ treated the query context as the async task, even though the query context is mostly a container for one or more executable queries. This branch changes the task unit to the QueryObject itself. That is a major simplification because Superset already computes cache keys, cache identity, datasource/RLS/impersonation scope, and result caching at the QueryObject level. Each QueryObject now becomes one SHARED GTF task keyed by query_cache_key; when the tasks complete, the client re-issues the original chart-data request and reads from the warmed per-query cache.

Polling is the primary readiness and correctness path in this PR: chart completion is driven by GET /api/v1/task/status_changes and works without a websocket server. The websocket work is included because it unlocks realtime Task List/list-view updates and faster chart wakeups, but that push layer is still WIP and should be treated as secondary to polling while it stabilizes.

Along the way, the branch consolidates distributed coordination behind one service/config and turns superset-websocket into a generic server-to-browser push transport. It also changes the websocket server build so it can run as a compact standalone bundle: the generated server artifact is about 1.2 MB, instead of requiring a runtime dependency tree to be shipped with it. That makes it practical to include in the official Superset image instead of requiring operators to build and ship a separate websocket image. Task List is the first realtime list-view surface; the shared list hook can support other entity lists once those entities publish change nudges.

This is a long-lived feature branch. Each step was reviewed and merged into gaq-to-gtf; this umbrella PR merges the completed epic back to master.

Key Features

• Async chart data on GTF. One SHARED GTF task per QueryObject, keyed by query_cache_key, deduplicated across compatible users. Contribution queries use a real GTF DAG edge to wait for the totals query.
• QueryObject-native result reassembly. The old qc-<hash> query-context descriptor is gone. GTF tasks warm the existing per-query cache, and the client re-issues the same POST once all task IDs settle.
• Request-level async opt-in. async_mode controls whether a chart-data request may run async. Absent async_mode keeps the API synchronous and preserves programmatic client behavior.
• One coordination service. Distributed locks, key/value state, Pub/Sub, and Redis Streams sit behind CoordinationService and the existing DISTRIBUTED_COORDINATION_CONFIG.
• Reliable await/notify where correctness depends on delivery. Task waiters, dependency joins, and lock handoff use Redis Streams when configured, with metastore polling fallback.
• Generic realtime push transport. superset-websocket routes by channel name and forwards {channel, payload} envelopes. It no longer knows about GAQ-specific event shapes.
• Realtime Task List. useListViewResource can subscribe to entity-change nudges and patch visible rows in place. Tasks publish nudges on status/progress changes, so Task List updates without manual refresh.
• Async result-cache floor. GLOBAL_ASYNC_QUERIES_MIN_CACHE_TTL floors result-cache TTL only on the async chart-data write path so the follow-up read does not miss an evicted result.
• Standalone websocket build and official image. The websocket server is bundled with esbuild into a compact standalone artifact (~1.2 MB locally), so it can ship in the official Superset image and run via an alternate entrypoint without carrying a separate node_modules tree.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

With GAQ enabled, opening the deck.gl demo dashboard spawns five separate GTF tasks, one of which is deduplicated because some charts share identical queries. These are visible during and after execution on the Tasks list view:

Additional screens and videos to be added.

TECHNICAL DETAILS

Architecture Overview

The key design decision is that the atomic async unit is the QueryObject, not the query context.

QueryContextProcessor.get_df_payload_result(query_obj) already executes and caches exactly one QueryObject. Its query_cache_key includes the pieces that matter for safe reuse: datasource identity, extra_cache_keys, RLS, impersonation, result type/format, and query payload. The old qc-<hash> cache entry did not hold results; it described a query context so the client could later reconstruct the response. That made GAQ coordinate at one level while caching and execution already happened at another.

This branch aligns those layers:

POST /api/v1/chart/data (async_mode=true)
  -> probe the existing per-query cache
  -> if every QueryObject is cached: return 200
  -> otherwise: schedule one SHARED GTF task per missing QueryObject
       - task_type = superset.query_object_v1
       - task_key = query_cache_key
       - contribution query depends_on totals query where applicable
  -> return 202 { task_ids, cursor }

Client
  -> poll GET /api/v1/task/status_changes with the server-issued cursor
  -> websocket task-status messages can wake the waiter sooner
  -> when all awaited tasks succeed: re-issue the same POST synchronously
  -> response is served from the warmed per-query cache
  -> terminal non-success status rejects the chart request

This removes the GAQ-specific AsyncQueryManager, /api/v1/async_event/, load_chart_data_into_cache, GAQ JWT/transport configuration, GLOBAL_ASYNC_QUERIES_CACHE_BACKEND, and the qc-<hash> / /api/v1/chart/data/<cache_key> replay path.

Coordination Model

The branch introduces superset/coordination/ as the common abstraction for distributed coordination:

Need	Mechanism	Why
Locks	CoordinationService lock/KV helpers	One backend/config for GTF, chart async, and reusable coordination
Task waiters / DAG joins / lock handoff	Redis Streams when configured, metastore polling fallback	Correctness depends on delivery, so signals need replayable await/notify semantics
Realtime UI nudges	Redis Pub/Sub	Best-effort UI freshness; missed events only leave a row stale until the next fetch

Lock release is ownership checked and atomic on Redis-backed coordination, so an expired holder cannot delete a lock acquired by a later holder.

Delivery Guarantees: Redis Streams vs Pub/Sub

This branch deliberately uses two Redis delivery patterns because the signals have different correctness requirements.

Signal type	Mechanism	Delivery expectation	Examples in this PR
Correctness-critical coordination	Redis Streams, with metastore polling fallback	A waiter must not miss a signal just because it started slightly later; delivery needs a replayable handoff window and bounded retention.	Task completion waiters, DAG dependency joins, task-lock release handoff
Best-effort realtime UI	Redis Pub/Sub	Low-latency fanout is useful, but missed messages are acceptable because the REST API and polling remain the source of truth.	Context-free Task List refresh nudges and websocket wakeups

This split is important for the GAQ-to-GTF cutover. A missed task-completion or lock-release signal can hang a chart request, cause a duplicate submit race, or leave a dependent task waiting unnecessarily, so those signals use Streams. A missed realtime nudge only means a visible row stays stale until the next fetch/poll, so Pub/Sub is the right lower-overhead fit. These Pub/Sub nudges are intentionally free of query context, result payloads, or authorization-bearing state; they wake the browser so it can fetch the authoritative state through the normal protected API surface.

The same rule should apply to unrelated future features that use CoordinationService. Use Streams when a consumer is blocked waiting for a specific event and correctness depends on observing it. Use Pub/Sub when the event is only an optimization for UI freshness or latency and the feature has an authoritative fallback path.

Realtime Transport

superset-websocket is now feature-agnostic. It subscribes to named backend channels and forwards a generic envelope to connected browsers.

The server build also changes from a package that required its dependency tree at runtime into a single esbuild-generated bundle. The resulting dist/index.cjs artifact is about 1.2 MB locally, which keeps the runtime artifact compact, lets it run standalone, and makes official-image inclusion feasible without asking operators to maintain a custom websocket image.

There are two realtime channel tiers:

• Public per-entity-type nudges such as entity-changes:task. Payloads carry opaque entity identifiers only. The browser fetches any sensitive data through the normal authorized REST API.
• Per-principal channels such as realtime:<channel_id>. These are still wakeups, not an authoritative state channel; the authenticated user or embedded guest token holder receives the nudge, then reconciles through polling/REST.

The websocket path accelerates the UI, but chart completion correctness still comes from the status_changes poll and GTF task state.

Operator-Facing Changes

Area	Change
Main switch	GLOBAL_ASYNC_QUERIES remains the chart-data async switch and auto-enables GLOBAL_TASK_FRAMEWORK
Existing config with expanded role	DISTRIBUTED_COORDINATION_CONFIG already existed; this branch makes it the single coordination backend for GAQ-on-GTF, GTF await/notify, locks, and realtime Pub/Sub. GLOBAL_ASYNC_QUERIES_POLLING_DELAY already existed and remains the base chart-data poll cadence.
Added config	DISTRIBUTED_COORDINATION_SIGNAL_TTL, GLOBAL_ASYNC_QUERIES_DEFAULT, GLOBAL_ASYNC_QUERIES_MIN_CACHE_TTL, GLOBAL_ASYNC_QUERIES_POLLING_MAX_DELAY, GLOBAL_ASYNC_QUERIES_POLLING_STALE_TIMEOUT, and feature-agnostic WEBSOCKET_* settings
Removed/replaced config/API	GLOBAL_ASYNC_QUERIES_CACHE_BACKEND, GAQ JWT/transport settings (GLOBAL_ASYNC_QUERIES_JWT_*, GLOBAL_ASYNC_QUERIES_TRANSPORT, GLOBAL_ASYNC_QUERIES_WEBSOCKET_URL), AsyncQueryManager, /api/v1/async_event/, qc-<hash> replay
New API	GET /api/v1/task/status_changes, task depends_on, chart-data async_mode request flag
New UI	Realtime Task List updates; Task List Details column for dependencies, dedupe count, payload/error details; dashboard async-mode dropdown

UPDATING.md contains the operator migration notes.

Progress Tracker

Every step was merged into gaq-to-gtf; the branch is complete.

Step	Scope	Status	PR
1	Coordination Service: locks, Pub/Sub, streams, KV, await consolidation	Merged	#43316
2	GTF task dependencies/DAG via task_dependencies and Task List dependency display	Merged	#43408
3	Coordination cleanup: Pub/Sub to Redis Streams for event-driven wait/notify	Merged	#43409
4	Canonical QueryObject serialization	Merged	#43410
5	GTF chart-data cutover: per-QueryObject tasks, status_changes, cache re-request, GAQ plumbing removal, embedded guest visibility	Merged	#43424
6	async_mode opt-in, GTF auto-enable, GLOBAL_ASYNC_QUERIES_DEFAULT, per-dashboard override	Merged	#43429
7	Realtime backend: entity-change nudges and per-principal channel-token service	Merged	#43431
8	Realtime wiring: task-status publish, generic websocket server, frontend WS client, guest face pile	Merged	#43434
9	Coordination KV callable-key generator	Merged	#43435
10	Standalone websocket build, official image inclusion, and alternate entrypoint	Merged	#43437
11	Realtime list views in useListViewResource; task nudges on transitions/progress	Merged	#43436
12	Async waiter race fix and guest_key widening	Merged	#43461
13	Ownership-checked distributed lock release	Merged	#43462
14	GLOBAL_ASYNC_QUERIES_MIN_CACHE_TTL async result-cache floor	Merged	#43463
15	Async-cache guardrails, atomic lock release, dependency-cancel publish fix, async_events retirement	Merged	#43473
16	Async poll lifecycle/backoff/stale timeout and Task List polish	Merged	#43486
17	Wait for task lock instead of failing concurrent submits; surface dedupe_count	Merged	#43492

What each step shipped

• Step 1 - Coordination Service (refactor(coordination): centralize distributed coordination in a single service #43316). Adds superset/coordination/ to consolidate distributed locks, Pub/Sub, Redis Streams, key/value operations, and await/notify helpers over DISTRIBUTED_COORDINATION_CONFIG.
• Step 2 - GTF task dependencies (feat(gtf): add task dependencies (DAG) with chain-icon Task List display #43408). Adds task_dependencies, Task.dependencies, TaskOptions.depends_on, block-and-wait scheduling with all_success semantics, REST API exposure, and Task List dependency indicators.
• Step 3 - Streams-based await/notify (refactor(coordination): reliable Redis Streams await/notify (replace at-most-once pub/sub) #43409). Moves correctness-sensitive wait/notify from at-most-once Pub/Sub to Redis Streams with DISTRIBUTED_COORDINATION_SIGNAL_TTL, falling back to metastore polling when no backend is configured.
• Step 4 - Canonical single-query serialization (feat(common): canonical single-query serialization for async chart data #43410). Adds serialize_query / load_serialized_query so a raw query dict can round-trip through QueryContextFactory and preserve the same query_cache_key.
• Step 5 - GTF chart-data cutover (feat(gaq): chart-data cutover — per-QueryObject GTF tasks + task-status polling #43424). Fans async /chart/data into one SHARED GTF task per QueryObject; returns {task_ids}; polls status_changes; re-issues from warm cache; removes GAQ manager/event/cache replay plumbing and supports embedded guest task visibility.
• Step 6 - Async opt-in and GTF auto-enable (feat(gaq): async chart data opt-in per request (async_mode) + auto-enable GTF #43429). Requires request async_mode for async chart data, keeps absent async_mode synchronous, auto-enables GTF when GAQ is enabled, and adds dashboard/deployment policy controls.
• Step 7 - Realtime backend (feat(websocket): realtime transport backend — entity-change nudges + channel tokens #43431). Adds opaque per-type entity-change nudges and feature-agnostic per-principal channel tokens, including websocket secret validation and logout clearing.
• Step 8 - Transport wiring (feat(websocket): generic task push transport — per-principal chart-data + guest face pile #43434). Publishes task status to subscriber channels, rewrites superset-websocket to generic Pub/Sub routing, adds the frontend realtime client, and anonymizes embedded guests in the Task List face pile.
• Step 9 - Coordinator KV callable-key (feat(coordination): accept a callable key generator in the KV API #43435). Allows KV APIs to accept callable key generators for parity with cache-key helper patterns.
• Step 10 - Standalone websocket build and official image (feat(docker): ship the websocket server in the official image #43437). Bundles the websocket server into a single dist/index.cjs file that can run standalone without a shipped node_modules tree, making it compact enough to include in the official image behind a websocket entrypoint/profile.
• Step 11 - Realtime list views (feat(listview): realtime list updates baked into useListViewResource #43436). Adds realtime support to useListViewResource; visible rows are debounce-batched, fetched through authorized REST endpoints, and merged in place. Task transitions/progress emit entity-change nudges.
• Step 12 - Async waiter race and guest_key widen (fix(gtf): close async-chart-data waiter race + widen guest_key column #43461). Returns a server-issued poll cursor captured before scheduling so fast terminal tasks cannot be missed, and widens task_subscribers.guest_key.
• Step 13 - Ownership-checked lock release (fix(distributed-lock): ownership-checked release (compare-and-delete) #43462). Adds per-acquisition lock tokens so expired holders cannot release locks owned by later holders; threads the token through dashboard Excel export.
• Step 14 - Async result-cache floor (feat(gaq): minimum result-cache TTL for async chart-data queries #43463). Floors async chart-data result-cache TTL only for async task writes, preserving synchronous behavior, cache-forever values, cache keys, and native-filter freshness.
• Step 15 - Async-cache guardrails (fix(gtf): async chart-data robustness + accurate DB errors; retire async_events #43473). Falls back to sync under NullCache DATA backends, makes the post-completion re-request a single synchronous request, atomically releases Redis locks, fixes dependency-cancel publishing, and retires async_events.
• Step 16 - Poll lifecycle and Task List polish (fix(gtf): stop async poll when idle + back off; second-granularity durations #43486). Stops polling when no tasks are awaited, backs off while tasks are quiet, adds stale-timeout rejection, returns advancing poll cursors, and improves duration/name/dependency display.
• Step 17 - Task-lock wait and dedupe count (fix(gtf): wait for the task lock instead of failing concurrent submits #43492). Waits for the task submit lock instead of failing concurrent submits for the same query_cache_key; increments and displays dedupe_count when work is reused.

Security and backward compatibility

Security. The role/capability matrix is unchanged. Query reuse is bounded by query_cache_key, which includes the cache-affecting datasource, RLS, impersonation, and query inputs. Async execution uses override_user; embedded guest subscribers use token-derived guest keys. Realtime public entity nudges carry opaque IDs only, and sensitive fields are fetched through authorized REST APIs and existing task filters.

Backward compatibility. GLOBAL_ASYNC_QUERIES remains the operator-facing switch. API clients that do not send async_mode continue to receive synchronous 200 responses. The breaking removals are limited to GAQ internals and GAQ-specific configuration/API paths and are documented in UPDATING.md.

Remaining follow-ups

• SQL Lab migration to GTF. SQL Lab remains the third async execution framework after this epic's scope decision. Migrating it to GTF is still desirable, but less urgent than the dashboard/chart GAQ migration and should be handled separately.
• Engine-level query cancellation. This epic does not add chart query cancellation. Superset's query-cancellation surface is fragmented across SQL Lab, query DAO/API code, executor paths, and engine-specific hooks: some engines support implicit cancellation, some need an ID from submit/cursor handling, and others expose cancellation through connection/session metadata. That surface should be cleaned up before GTF chart tasks can cancel the underlying database query reliably.
• Other-entity realtime nudges. Task List emits realtime nudges. Other list views can use the shared hook once their DAO/command commit points publish entity-change events.
• Extensions coordination surface. Expose coordination through a superset_core.coordination abstraction for extensions, similar to superset_core.tasks.

TESTING INSTRUCTIONS

Each step PR carries focused unit/integration coverage and was green when merged into gaq-to-gtf.

End-to-end manual validation for the full epic:

1. Enable GLOBAL_ASYNC_QUERIES and configure DISTRIBUTED_COORDINATION_CONFIG; GLOBAL_TASK_FRAMEWORK is auto-enabled.
2. Start the official superset-websocket entrypoint with WEBSOCKET_ENABLED and the corresponding websocket URL/JWT config.
3. Load a dashboard with mixed multi-query charts and contribution charts.
4. Confirm chart-data requests return 202 with GTF task IDs on cache miss, then resolve after the client polls GET /api/v1/task/status_changes and re-issues the chart-data POST.
5. Confirm the Task List shows superset.query_object_v1 tasks and updates live without manual refresh.
6. Confirm terminal task failures propagate back to the waiting chart. Engine-level database query cancellation is a follow-up and is not expected to work from this PR.
7. Stop the websocket server and confirm chart completion still works through interval polling.
8. Reload the same dashboard and confirm warm-cache chart-data requests short-circuit to synchronous 200 responses.

Useful targeted commands:

pytest tests/unit_tests/common/test_query_serialization.py -v
pytest tests/unit_tests/common/test_async_min_cache_ttl.py -v
pytest tests/unit_tests/tasks/ -v
pytest tests/integration_tests/tasks/ -v
npm run test -- src/middleware/realtime.test.ts src/hooks/apiResources/useListViewResource.test.tsx

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags: GLOBAL_ASYNC_QUERIES (auto-enables GLOBAL_TASK_FRAMEWORK); optional WEBSOCKET_ENABLED for realtime transport
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[netlify]

✅ Deploy Preview for superset-docs-preview ready!

Name	Link
🔨 Latest commit	c7fcebf
🔍 Latest deploy log	https://app.netlify.com/projects/superset-docs-preview/deploys/6a8917697accc00008653eb0
😎 Deploy Preview	https://deploy-preview-43407--superset-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

❌ Patch coverage is 84.66135% with 154 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.88%. Comparing base (5812c0e) to head (1f9bbdf).
⚠️ Report is 47 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/tasks/async_queries.py	65.15%	22 Missing and 1 partial ⚠️
superset/daos/tasks.py	64.40%	17 Missing and 4 partials ⚠️
superset/coordination/utils.py	0.00%	12 Missing ⚠️
superset/tasks/api.py	47.05%	9 Missing ⚠️
...tend/src/components/Chart/DrillBy/DrillByModal.tsx	33.33%	8 Missing ⚠️
superset-frontend/src/middleware/asyncEvent.ts	90.90%	7 Missing ⚠️
superset/coordination/base.py	94.69%	3 Missing and 4 partials ⚠️
...set-ui-core/src/chart/components/StatefulChart.tsx	44.44%	5 Missing ⚠️
superset/commands/tasks/cancel.py	64.28%	3 Missing and 2 partials ⚠️
superset/commands/tasks/submit.py	85.29%	4 Missing and 1 partial ⚠️
... and 20 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #43407      +/-   ##
==========================================
+ Coverage   78.82%   78.88%   +0.05%     
==========================================
  Files        2876     2884       +8     
  Lines      164459   165220     +761     
  Branches    37956    38152     +196     
==========================================
+ Hits       129634   130331     +697     
- Misses      32378    32426      +48     
- Partials     2447     2463      +16     

Flag	Coverage Δ
hive	38.06% <31.41%> (-0.01%)	⬇️
mysql	57.68% <49.08%> (-0.09%)	⬇️
postgres	57.71% <49.08%> (-0.10%)	⬇️
presto	39.99% <31.97%> (-0.02%)	⬇️
python	83.55% <84.57%> (+0.02%)	⬆️
sqlite	57.40% <49.08%> (-0.09%)	⬇️
unit	73.71% <81.48%> (+0.16%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.