Potential deadlocks in TPC-H benchmark suite

[hendrikmakait]

Looking at CI failures over the last two months, I noticed several instances of tests timing out. The timeouts appear to be caused by deadlocks: Grafana always shows a single task in processing on the scheduler, but no corresponding task on the workers. These are the affectd instances:

• https://cloud.coiled.io/clusters/740479/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q08 and Q21
• https://cloud.coiled.io/clusters/733683/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q07
• https://cloud.coiled.io/clusters/717346/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q04 and Q21
• https://cloud.coiled.io/clusters/707276/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q19
• https://cloud.coiled.io/clusters/703235/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q13
• https://cloud.coiled.io/clusters/699880/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q12
• https://cloud.coiled.io/clusters/689565/account/dask-benchmarks/information?workspace=dask-benchmarks
  • Q17

Based on Grafana activity, I suspect that this might be related to P2P but that's just a guess at this point.

[hendrikmakait]

• https://cloud.coiled.io/clusters/751395/account/dask-benchmarks/information
  • Q07 and Q13

[hendrikmakait]

The Grafana dashboard misled me since it filters out executing tasks. When this fails, there is indeed a task executing on the worker. On my reproducing cluster, it's this one:

Key: ('readparquetpyarrowfs-fused-operation-a1013f0a679c620b3944185a7bcb851c', 109)

Callstack:

Key: ('readparquetpyarrowfs-fused-operation-a1013f0a679c620b3944185a7bcb851c', 109)
File "/opt/coiled/env/lib/python3.12/threading.py", line 1032, in _bootstrap self._bootstrap_inner()

File "/opt/coiled/env/lib/python3.12/threading.py", line 1075, in _bootstrap_inner self.run()

File "/opt/coiled/env/lib/python3.12/threading.py", line 1012, in run self._target(*self._args, **self._kwargs)

File "/opt/coiled/env/lib/python3.12/site-packages/distributed/threadpoolexecutor.py", line 58, in _worker task.run()

File "/opt/coiled/env/lib/python3.12/site-packages/distributed/_concurrent_futures_thread.py", line 65, in run result = self.fn(*self.args, **self.kwargs)

File "/opt/coiled/env/lib/python3.12/site-packages/distributed/utils.py", line 1508, in <lambda> executor, lambda: context.run(func, *args, **kwargs)

File "/opt/coiled/env/lib/python3.12/site-packages/distributed/worker.py", line 2946, in _run_task msg = _run_task_simple(task, data, time_delay)

File "/opt/coiled/env/lib/python3.12/site-packages/distributed/worker.py", line 2982, in _run_task_simple result = task(data)

File "/opt/coiled/env/lib/python3.12/site-packages/dask/_task_spec.py", line 651, in __call__ return self.func(*new_argspec)

File "/opt/coiled/env/lib/python3.12/site-packages/dask_expr/_expr.py", line 3799, in _execute_internal_graph res = execute_graph(internal_tasks, cache=cache, keys=[outkey])

File "/opt/coiled/env/lib/python3.12/site-packages/dask/_task_spec.py", line 786, in execute_graph cache[key] = node(cache)

File "/opt/coiled/env/lib/python3.12/site-packages/dask/_task_spec.py", line 650, in __call__ return self.func(*new_argspec, **kwargs)

File "/opt/coiled/env/lib/python3.12/site-packages/dask_expr/io/io.py", line 185, in _load_multiple_files table = pa.concat_tables(tables, promote_options="permissive")

File "/opt/coiled/env/lib/python3.12/site-packages/dask_expr/io/io.py", line 177, in <genexpr> ReadParquetPyarrowFS._fragment_to_table(

File "/opt/coiled/env/lib/python3.12/site-packages/dask_expr/io/parquet.py", line 1195, in _fragment_to_table return fragment.to_table(

[hendrikmakait]

Possibly related: apache/arrow#40019