5.10.0

Changelog

Highlighted Features Added

• Add a --caching option which explicitly states whether to use caching with a workflow. Uses a default value depending on whether or not we are using the file job store if not specified. (#4218)
• New prototype WDL runner python -m toil.wdl.wdltoil using MiniWDL (#3468)
• MiniWDL-based WDL implementation can now run the vg Giraffe WDL workflow ( #4353)
• Toil now tests against our own tiny set of WDL conformance tests (#4351)
• Toil can run the HPRC assembly WDL workflows (#4435)
• Toil can now use Mesos roles (#4455)

Breaking Changes

• Replace "preemptable" with "preemptible", add example of using --defaultPreemptible flag to Preemptibility documentation (#1951)

CWL

• CWL: run all ExpressionTools on the Leader node, instead of submitting separate jobs (#4157)

Kubernetes

• Kubernetes batch system: Delete jobs individually when batch delete fails (#3403)
• Documentation for running a Toil leader for a Kubernetes workflow outside Kubernetes now covers examples and common problems for running CWL workflows (document toil-cwl-runner + "Running the Leader Outside Kubernetes" #3422)
• Kubernetes batch system: support --maxCores, --maxDisk, and --maxMemory (#2864)
• Add tutorial for Kubernetes launch cluster (#3743)

Dependencies

• Require htcondor 10 exactly (#4315)
• Toil jobs now have a local parameter which determines if they should run on the leader. (#4388)

Misc

• The offline tests can now be run in parallel (#3493)
• Code updated to be more idiomatic for Python3.7 (#4295)
• Support for a --network for toil launch-cluster for Google cloud (#4196)
• Support for a --use_private_ip for toil launch-cluster to dial nodes by private IP instead of public IP (#4196)
• GPU scheduling should now be supported on Slurm (#4308)
• Toil now supports a --batchLogsDir option and TOIL_BATCH_LOGS_DIR environment variable, to provide a directory other than the work dir where Toil will instruct HPC batch systems to save their captured job logs.
• htcondor batch system should now work again, and will retry connections
• Updated the --coalesceStatusCalls help documentation to reflect the current state of #4431 (#4437)
• Toil no longer trusts XDG_RUNTIME_DIR under Slurm (fixes some of the issues behind #4395 when Slurm is configured not to follow the XDG spec) (#4435)
• Toil now puts it lock files for Singularity cache directories for WDL in those directories (#4435)
• Toil's WDL interpreter can now use local-to-the-leader jobs for evaluating WDL code that doesn't need appreciable resources (#4388)
• Toil now tolerates more possible exceptions related to the panasas network file system (#4440)
• Type hinting to functions in resource.py (#938)
• Added return type to inVirtualEnv() in __init__.py (#938)
• Added None checks to some function bodies (#938)

Bug Fixes

• Stop crashing when predefined batch job exit reasons are used and need to go into the message bus log file (#4321)
• Added import subprocess to restore the behavior of #588. (#4429)
• Toil will no longer use the stored message bus path from an old execution of a workflow when deciding where to save the message bus log when restarting a workflow (#4438)
• Fix --custom-net mutual exclusivity bug. (#4458)

Thank you to our contributors: @stxue1 , @DailyDreaming , @mr-c , @adamnovak , @jfennick , @misterbrandonwalker , @w-gao , @stephanaime , @glennhickey , @Hexotical , @manabuishii @gmloose , @boukn , and @thiagogenez !