Implement a better config file system for CWL/WDL options

[stxue1]

This should resolve #4632

Changelog Entry

To be copied to the draft changelog by merger:

• Updated docs for config file
• Config file will include CWL and WDL arguments
  • Moved nonpositional CWL/WDL argument definitions to common.py
  • Config file will have all options be commented out when generated

Reviewer Checklist

• Make sure it is coming from issues/XXXX-fix-the-thing in the Toil repo, or from an external repo.
  • If it is coming from an external repo, make sure to pull it in for CI with:

    contrib/admin/test-pr otheruser theirbranchname issues/XXXX-fix-the-thing
    

  • If there is no associated issue, create one.
• Read through the code changes. Make sure that it doesn't have:
  • Addition of trailing whitespace.
  • New variable or member names in camelCase that want to be in snake_case.
  • New functions without type hints.
  • New functions or classes without informative docstrings.
  • Changes to semantics not reflected in the relevant docstrings.
  • New or changed command line options for Toil workflows that are not reflected in docs/running/{cliOptions,cwl,wdl}.rst
  • New features without tests.
• Comment on the lines of code where problems exist with a review comment. You can shift-click the line numbers in the diff to select multiple lines.
• Finish the review with an overall description of your opinion.

Merger Checklist

• Make sure the PR passes tests.
• Make sure the PR has been reviewed since its last modification. If not, review it.
• Merge with the Github "Squash and merge" feature.
  • If there are multiple authors' commits, add Co-authored-by to give credit to all contributing authors.
• Copy its recommended changelog entry to the Draft Changelog.
• Append the issue number in parentheses to the changelog entry.

[adamnovak]

This looks pretty good. But some of the docs are worded oddly, and I think we need to not have the CWL and WDL front-ends talk about each other.

I also see some optional opportunities for condensing repeated logic.

[adamnovak]

There's a lot of duplicated logic between these, and between all the CWL options. It would be nice if we had a way to repeat ourselves less somehow. Like maybe a wrapper function that knows how to namespace the options and suppress the help when appropriate? That might be too much extra work though.

[adamnovak]

I wish the different front-ends didn't all have to have their options in one giant file here. We could consider reorganizing things so we have like src/toil/options/cwl.py, src/toil/options/common.py, etc. so we could keep the different runners' options split up even when we can't import their main files.

[adamnovak]

Can we abstract this out somehow? Maybe use the function we wrote make the default parser instead of ArgParser()'s constructor, and then move the check for options for front-ends not in use down into the options-parsing code?

We shouldn't have to know in the CWL front-end that the WDL front-end even exists.

[adamnovak]

We can probably just say wdl=True; we don't want to know anything about the CWL front-end in the WDL front-end either.

[adamnovak]

This should be the same check code as for excluding WDL options from the CWL runner, and it should be somewhere in the option parsing library code we have. Maybe we need something we import that just does this whole block, and we tell it if we support CWL options and if we support WDL options?

[mr-c]

toil-cwl-runner --help no longer lists all options

$ toil-cwl-runner --help
usage: toil-cwl-runner [-h]

options:
  -h, --help  show this help message and exit

Before this PR

usage: toil-cwl-runner [-h] [--logCritical] [--logError] [--logWarning]
                       [--logDebug] [--logInfo] [--logOff]
                       [--logLevel {Critical,Error,Warning,Debug,Info,critical,error,warning,debug,info,CRITICAL,ERROR,WARNING,DEBUG,INFO}]
                       [--logFile LOGFILE] [--rotatingLogging]
                       [--jobstore JOBSTORE] [--workDir WORKDIR]
                       [--coordinationDir COORDINATION_DIR] [--noStdOutErr]
                       [--stats] [--clean {always,onError,never,onSuccess}]
                       [--cleanWorkDir {always,onError,never,onSuccess}]
                       [--clusterStats [CLUSTERSTATS]] [--restart]
                       [--batchSystem {aws_batch,single_machine,grid_engine,lsf,mesos,slurm,torque,htcondor,kubernetes}]
                       [--disableHotDeployment]
                       [--disableAutoDeployment DISABLEAUTODEPLOYMENT]
                       [--maxJobs MAX_JOBS] [--maxLocalJobs MAX_LOCAL_JOBS]
                       [--manualMemArgs] [--runLocalJobsOnWorkers]
                       [--coalesceStatusCalls]
                       [--statePollingWait STATEPOLLINGWAIT]
                       [--batchLogsDir BATCH_LOGS_DIR]
                       [--awsBatchRegion AWS_BATCH_REGION]
                       [--awsBatchQueue AWS_BATCH_QUEUE]
                       [--awsBatchJobRoleArn AWS_BATCH_JOB_ROLE_ARN]
                       [--scale SCALE] [--dont_allocate_mem | --allocate_mem]
                       [--kubernetesHostPath KUBERNETES_HOST_PATH]
                       [--kubernetesOwner KUBERNETES_OWNER]
                       [--kubernetesServiceAccount KUBERNETES_SERVICE_ACCOUNT]
                       [--kubernetesPodTimeout KUBERNETES_POD_TIMEOUT]
                       [--symlinkImports SYMLINKIMPORTS]
                       [--moveOutputs MOVEOUTPUTS] [--caching CACHING]
                       [--provisioner {aws,gce,None}] [--nodeTypes NODETYPES]
                       [--maxNodes MAXNODES] [--minNodes MINNODES]
                       [--targetTime TARGETTIME] [--betaInertia BETAINERTIA]
                       [--scaleInterval SCALEINTERVAL]
                       [--preemptibleCompensation PREEMPTIBLECOMPENSATION]
                       [--nodeStorage NODESTORAGE]
                       [--nodeStorageOverrides NODESTORAGEOVERRIDES]
                       [--metrics METRICS]
                       [--assumeZeroOverhead ASSUME_ZERO_OVERHEAD]
                       [--defaultMemory DEFAULTMEMORY] [--defaultCores FLOAT]
                       [--defaultDisk INT]
                       [--defaultAccelerators ACCELERATOR[,ACCELERATOR...]]
                       [--defaultPreemptible [BOOL]] [--maxCores INT]
                       [--maxMemory INT] [--maxDisk INT]
                       [--retryCount RETRYCOUNT]
                       [--enableUnlimitedPreemptibleRetries ENABLEUNLIMITEDPREEMPTIBLERETRIES]
                       [--doubleMem DOUBLEMEM]
                       [--maxJobDuration MAXJOBDURATION]
                       [--rescueJobsFrequency RESCUEJOBSFREQUENCY]
                       [--maxLogFileSize MAXLOGFILESIZE]
                       [--writeLogs [WRITELOGS]]
                       [--writeLogsGzip [WRITELOGSGZIP]]
                       [--writeLogsFromAllJobs WRITELOGSFROMALLJOBS]
                       [--writeMessages WRITE_MESSAGES]
                       [--realTimeLogging REALTIMELOGGING]
                       [--disableChaining DISABLECHAINING]
                       [--disableJobStoreChecksumVerification DISABLEJOBSTORECHECKSUMVERIFICATION]
                       [--sseKey SSEKEY] [--setEnv NAME=VALUE or NAME]
                       [--servicePollingInterval SERVICEPOLLINGINTERVAL]
                       [--forceDockerAppliance FORCEDOCKERAPPLIANCE]
                       [--statusWait STATUSWAIT]
                       [--disableProgress DISABLEPROGRESS] [--config CONFIG]
                       [--debugWorker] [--disableWorkerOutputCapture]
                       [--badWorker BADWORKER]
                       [--badWorkerFailInterval BADWORKERFAILINTERVAL]
                       [--not-strict] [--enable-dev] [--enable-ext] [--quiet]
                       [--basedir BASEDIR] [--outdir OUTDIR] [--version]
                       [--log-dir LOG_DIR]
                       [--user-space-docker-cmd USER_SPACE_DOCKER_CMD | --singularity | --podman | --no-container | --leave-container]
                       [--custom-net CUSTOM_NET] [--cidfile-dir CIDFILE_DIR]
                       [--cidfile-prefix CIDFILE_PREFIX]
                       [--preserve-environment VAR1 VAR2 [VAR1 VAR2 ...]]
                       [--preserve-entire-environment]
                       [--destBucket DESTBUCKET]
                       [--beta-dependency-resolvers-configuration BETA_DEPENDENCY_RESOLVERS_CONFIGURATION]
                       [--beta-dependencies-directory BETA_DEPENDENCIES_DIRECTORY]
                       [--beta-use-biocontainers] [--beta-conda-dependencies]
                       [--tmpdir-prefix TMPDIR_PREFIX]
                       [--tmp-outdir-prefix TMP_OUTDIR_PREFIX]
                       [--force-docker-pull] [--no-match-user]
                       [--no-read-only] [--strict-memory-limit]
                       [--strict-cpu-limit] [--relax-path-checks]
                       [--default-container DEFAULT_CONTAINER]
                       [--compute-checksum | --no-compute-checksum]
                       [--eval-timeout EVAL_TIMEOUT] [--overrides OVERRIDES]
                       [--mpi-config-file MPI_CONFIG_FILE]
                       [--bypass-file-store] [--disable-streaming]
                       [--provenance PROVENANCE] [--enable-user-provenance]
                       [--disable-user-provenance] [--enable-host-provenance]
                       [--disable-host-provenance] [--orcid ORCID]
                       [--full-name CWL_FULL_NAME]
                       cwltool [cwljob ...]
positional arguments:

cwltool               CWL file to run.

cwljob                Input file or CWL options. If CWL workflow takes an

input, the name of the input can be used as an option.

For example: "toil-cwl-runner workflow.cwl --file1

file". If an input has the same name as a Toil option,

pass '--' before it.
options:

-h, --help            show this help message and exit

--not-strict

--enable-dev          Enable loading and running development versions of CWL

--enable-ext          Enable loading and running 'cwltool:' extensions to

the CWL standards.

--quiet

--basedir BASEDIR

--outdir OUTDIR

--version             show program's version number and exit

--log-dir LOG_DIR     Log your tools stdout/stderr to this location outside

of container

--user-space-docker-cmd USER_SPACE_DOCKER_CMD

(Linux/OS X only) Specify a user space docker command

(like udocker or dx-docker) that will be used to call

'pull' and 'run'

--singularity         Use Singularity runtime for running containers.

Requires Singularity v2.6.1+ and Linux with kernel

version v3.18+ or with overlayfs support backported.

--podman              Use Podman runtime for running containers.

--no-container        Do not execute jobs in a Docker container, even when

DockerRequirement is specified under hints.

--leave-container     Do not delete Docker container used by jobs after they

exit

--preserve-environment VAR1 VAR2 [VAR1 VAR2 ...]

Preserve specified environment variables when running

CommandLineTools

--preserve-entire-environment

Preserve all environment variable when running

CommandLineTools.

--destBucket DESTBUCKET

Specify a cloud bucket endpoint for output files.

--beta-dependency-resolvers-configuration BETA_DEPENDENCY_RESOLVERS_CONFIGURATION

--beta-dependencies-directory BETA_DEPENDENCIES_DIRECTORY

--beta-use-biocontainers

--beta-conda-dependencies

--tmpdir-prefix TMPDIR_PREFIX

Path prefix for temporary directories

--tmp-outdir-prefix TMP_OUTDIR_PREFIX

Path prefix for intermediate output directories

--force-docker-pull   Pull latest docker image even if it is locally present

--no-match-user       Disable passing the current uid to docker run --user

--no-read-only        Do not set root directory in the container as read-

only

--strict-memory-limit

When running with software containers and the Docker

engine, pass either the calculated memory allocation

from ResourceRequirements or the default of 1 gigabyte

to Docker's --memory option.

--strict-cpu-limit    When running with software containers and the Docker

engine, pass either the calculated cpu allocation from

ResourceRequirements or the default of 1 core to

Docker's --cpu option. Requires docker version >=

v1.13.

--relax-path-checks   Relax requirements on path names to permit spaces and

hash characters.

--default-container DEFAULT_CONTAINER

Specify a default docker container that will be used

if the workflow fails to specify one.

--compute-checksum    Compute checksum of contents while collecting outputs

--no-compute-checksum

Do not compute checksum of contents while collecting

outputs

--eval-timeout EVAL_TIMEOUT

Time to wait for a Javascript expression to evaluate

before giving an error, default 20s.

--overrides OVERRIDES

Read process requirement overrides from file.

--mpi-config-file MPI_CONFIG_FILE

Platform specific configuration for MPI (parallel

launcher, its flag etc). See the cwltool README

section 'Running MPI-based tools' for details of the

format: https://github.com/common-workflow-

language/cwltool#running-mpi-based-tools-that-need-to-

be-launched

--bypass-file-store   Do not use Toil's file store and assume all paths are

accessible in place from all nodes.

--disable-streaming   Disable file streaming for files that have

'streamable' flag True
Logging Options:

--logCritical         Turn on loglevel Critical. Default: INFO.

--logError            Turn on loglevel Error. Default: INFO.

--logWarning          Turn on loglevel Warning. Default: INFO.

--logDebug            Turn on loglevel Debug. Default: INFO.

--logInfo             Turn on loglevel Info. Default: INFO.

--logOff              Same as --logCRITICAL.

--logLevel {Critical,Error,Warning,Debug,Info,critical,error,warning,debug,info,CRITICAL,ERROR,WARNING,DEBUG,INFO}

Set the log level. Default: INFO. Options:

['Critical', 'Error', 'Warning', 'Debug', 'Info',

'critical', 'error', 'warning', 'debug', 'info',

'CRITICAL', 'ERROR', 'WARNING', 'DEBUG', 'INFO'].

--logFile LOGFILE     File to log in.

--rotatingLogging     Turn on rotating logging, which prevents log files

from getting too big.
Toil core options.:

Options to specify the location of the Toil workflow and turn on stats

collation about the performance of jobs.
--jobstore JOBSTORE, --jobStore JOBSTORE

The location of the job store for the workflow. A job

store holds persistent information about the jobs,

stats, and files in a workflow. If the workflow is run

with a distributed batch system, the job store must be

accessible by all worker nodes. Depending on the

desired job store implementation, the location should

be formatted according to one of the following

schemes: file: where  points to a

directory on the file systen aws::

where  is the name of an AWS region like us-

west-2 and  will be prepended to the names of

any top-level AWS resources in use by job store, e.g.

S3 buckets. google:<project_id>: TODO: explain

For backwards compatibility, you may also specify

./foo (equivalent to file:./foo or just file:foo) or

/bar (equivalent to file:/bar).

--workDir WORKDIR     Absolute path to directory where temporary files

generated during the Toil run should be placed.

Standard output and error from batch system jobs

(unless --noStdOutErr is set) will be placed in this

directory. A cache directory may be placed in this

directory. Temp files and folders will be placed in a

directory toil- within workDir. The

workflowID is generated by Toil and will be reported

in the workflow logs. Default is determined by the

variables (TMPDIR, TEMP, TMP) via mkdtemp. This

directory needs to exist on all machines running jobs;

if capturing standard output and error from batch

system jobs is desired, it will generally need to be

on a shared file system. When sharing a cache between

containers on a host, this directory must be shared

between the containers. [env var: TOIL_WORKDIR]

--coordinationDir COORDINATION_DIR

Absolute path to directory where Toil will keep state

and lock files.When sharing a cache between containers

on a host, this directory must be shared between the

containers. [env var: TOIL_COORDINATION_DIR]

--noStdOutErr         Do not capture standard output and error from batch

system jobs.

--stats               Records statistics about the toil workflow to be used

by 'toil stats'.

--clean {always,onError,never,onSuccess}

Determines the deletion of the jobStore upon

completion of the program. Choices: ['always',

'onError', 'never', 'onSuccess']. The --stats option

requires information from the jobStore upon completion

so the jobStore will never be deleted with that flag.

If you wish to be able to restart the run, choose

'never' or 'onSuccess'. Default is 'never' if stats is

enabled, and 'onSuccess' otherwise.

--cleanWorkDir {always,onError,never,onSuccess}

Determines deletion of temporary worker directory upon

completion of a job. Choices: ['always', 'onError',

'never', 'onSuccess']. Default = always. WARNING: This

option should be changed for debugging only. Running a

full pipeline with this option could fill your disk

with excessive intermediate data.

--clusterStats [CLUSTERSTATS]

If enabled, writes out JSON resource usage statistics

to a file. The default location for this file is the

current working directory, but an absolute path can

also be passed to specify where this file should be

written. This options only applies when using scalable

batch systems.
Toil options for restarting an existing workflow.:

Allows the restart of an existing workflow
--restart             If --restart is specified then will attempt to restart

existing workflow at the location pointed to by the

--jobStore option. Will raise an exception if the

workflow does not exist
Toil options for specifying the batch system.:

Allows the specification of the batch system.
--batchSystem {aws_batch,single_machine,grid_engine,lsf,mesos,slurm,torque,htcondor,kubernetes}

The type of batch system to run the job(s) with,

currently can be one of aws_batch, single_machine,

grid_engine, lsf, mesos, slurm, torque, htcondor,

kubernetes. default=single_machine

--disableHotDeployment

Hot-deployment was renamed to auto-deployment. Option

now redirects to --disableAutoDeployment. Left in for

backwards compatibility.

--disableAutoDeployment DISABLEAUTODEPLOYMENT

Should auto-deployment of the user script be

deactivated? If True, the user script/package should

be present at the same location on all workers.

Default = False.

--maxJobs MAX_JOBS    Specifies the maximum number of jobs to submit to the

backing scheduler at once. Not supported on Mesos or

AWS Batch. Use 0 for unlimited. Defaults to unlimited.

--maxLocalJobs MAX_LOCAL_JOBS

Specifies the maximum number of housekeeping jobs to

run sumultaneously on the local system. Use 0 for

unlimited. Defaults to the number of local cores (8).

--manualMemArgs       Do not add the default arguments: 'hv=MEMORY' &

'h_vmem=MEMORY' to the qsub call, and instead rely on

TOIL_GRIDGENGINE_ARGS to supply alternative arguments.

Requires that TOIL_GRIDGENGINE_ARGS be set.

--runLocalJobsOnWorkers, --runCwlInternalJobsOnWorkers

Whether to run jobs marked as local (e.g. CWLScatter)

on the worker nodes instead of the leader node. If

false (default), then all such jobs are run on the

leader node. Setting this to true can speed up CWL

pipelines for very large workflows with many sub-

workflows and/or scatters, provided that the worker

pool is large enough.

--coalesceStatusCalls

Ask for job statuses from the batch system in a batch.

Deprecated; now always enabled where supported.

--statePollingWait STATEPOLLINGWAIT

Time, in seconds, to wait before doing a scheduler

query for job state. Return cached results if within

the waiting period. Only works for grid engine batch

systems such as gridengine, htcondor, torque, slurm,

and lsf.

--batchLogsDir BATCH_LOGS_DIR

Directory to tell the backing batch system to log

into. Should be available on both the leader and the

workers, if the backing batch system writes logs to

the worker machines' filesystems, as many HPC

schedulers do. If unset, the Toil work directory will

be used. Only works for grid engine batch systems such

as gridengine, htcondor, torque, slurm, and lsf. [env

var: TOIL_BATCH_LOGS_DIR]

--awsBatchRegion AWS_BATCH_REGION

The AWS region containing the AWS Batch queue to

submit to. [env var: TOIL_AWS_REGION]

--awsBatchQueue AWS_BATCH_QUEUE

The name or ARN of the AWS Batch queue to submit to.

[env var: TOIL_AWS_BATCH_QUEUE]

--awsBatchJobRoleArn AWS_BATCH_JOB_ROLE_ARN

The ARN of an IAM role to run AWS Batch jobs as, so

they can e.g. access a job store. Must be assumable by

ecs-tasks.amazonaws.com. [env var:

TOIL_AWS_BATCH_JOB_ROLE_ARN]

--scale SCALE         A scaling factor to change the value of all submitted

tasks's submitted cores. Used in the single_machine

batch system. Useful for running workflows on smaller

machines than they were designed for, by setting a

value less than 1. (default: 1)

--dont_allocate_mem   A flag that can block allocating memory with '--mem'

for job submissions on SLURM since some system servers

may reject any job request that explicitly specifies

the memory allocation. The default is to always

allocate memory.

--allocate_mem        A flag that can block allocating memory with '--mem'

for job submissions on SLURM since some system servers

may reject any job request that explicitly specifies

the memory allocation. The default is to always

allocate memory.

--kubernetesHostPath KUBERNETES_HOST_PATH

Path on Kubernetes hosts to use as shared inter-pod

temp directory. (default: None) [env var:

TOIL_KUBERNETES_HOST_PATH]

--kubernetesOwner KUBERNETES_OWNER

Username to mark Kubernetes jobs with. If the provided

value is None, the value will be generated at runtime.

(Generated default: michael) [env var:

TOIL_KUBERNETES_OWNER]

--kubernetesServiceAccount KUBERNETES_SERVICE_ACCOUNT

Service account to run jobs as. (default: None) [env

var: TOIL_KUBERNETES_SERVICE_ACCOUNT]

--kubernetesPodTimeout KUBERNETES_POD_TIMEOUT

Seconds to wait for a scheduled Kubernetes pod to

start running. (default: 120) [env var:

TOIL_KUBERNETES_POD_TIMEOUT]
Toil options for configuring storage.:

Allows configuring Toil's data storage.
--symlinkImports SYMLINKIMPORTS

When using a filesystem based job store, CWL input

files are by default symlinked in. Setting this option

to True instead copies the files into the job store,

which may protect them from being modified externally.

When set to False, as long as caching is enabled, Toil

will protect the file automatically by changing the

permissions to read-only.default=True

--moveOutputs MOVEOUTPUTS

When using a filesystem based job store, output files

are by default moved to the output directory, and a

symlink to the moved exported file is created at the

initial location. Setting this option to True instead

copies the files into the output directory. Applies to

filesystem-based job stores only.default=False

--caching CACHING     Enable or disable caching for your workflow,

specifying this overrides default from job store
Toil options for autoscaling the cluster of worker nodes.:

Allows the specification of the minimum and maximum number of nodes in an

autoscaled cluster, as well as parameters to control the level of

provisioning.
--provisioner {aws,gce,None}, -p {aws,gce,None}

The provisioner for cluster auto-scaling. This is the

main Toil '--provisioner' option, and defaults to None

for running on single machine and non-auto-scaling

batch systems. The currently supported choices are

['aws', 'gce', None]. The default is None.

--nodeTypes NODETYPES

Specifies a list of comma-separated node types, each

of which is composed of slash-separated instance

types, and an optional spot bid set off by a colon,

making the node type preemptible. Instance types may

appear in multiple node types, and the same node type

may appear as both preemptible and non-preemptible.

Valid argument specifying two node types:

c5.4xlarge/c5a.4xlarge:0.42,t2.large Node types:

c5.4xlarge/c5a.4xlarge:0.42 and t2.large Instance

types: c5.4xlarge, c5a.4xlarge, and t2.large

Semantics: Bid $0.42/hour for either c5.4xlarge or

c5a.4xlarge instances, treated interchangeably, while

they are available at that price, and buy t2.large

instances at full price. default=[]

--maxNodes MAXNODES   Maximum number of nodes of each type in the cluster,

if using autoscaling, provided as a comma-separated

list. The first value is used as a default if the list

length is less than the number of nodeTypes.

default=[10]

--minNodes MINNODES   Mininum number of nodes of each type in the cluster,

if using auto-scaling. This should be provided as a

comma-separated list of the same length as the list of

node types. default=[0]

--targetTime TARGETTIME

Sets how rapidly you aim to complete jobs in seconds.

Shorter times mean more aggressive parallelization.

The autoscaler attempts to scale up/down so that it

expects all queued jobs will complete within

targetTime seconds. default=1800

--betaInertia BETAINERTIA

A smoothing parameter to prevent unnecessary

oscillations in the number of provisioned nodes. This

controls an exponentially weighted moving average of

the estimated number of nodes. A value of 0.0 disables

any smoothing, and a value of 0.9 will smooth so much

that few changes will ever be made. Must be between

0.0 and 0.9. default=0.1

--scaleInterval SCALEINTERVAL

The interval (seconds) between assessing if the scale

of the cluster needs to change. default=60

--preemptibleCompensation PREEMPTIBLECOMPENSATION, --preemptableCompensation PREEMPTIBLECOMPENSATION

The preference of the autoscaler to replace

preemptible nodes with non-preemptible nodes, when

preemptible nodes cannot be started for some reason.

This value must be between 0.0 and 1.0, inclusive. A

value of 0.0 disables such compensation, a value of

0.5 compensates two missing preemptible nodes with a

non-preemptible one. A value of 1.0 replaces every

missing pre-emptable node with a non-preemptible one.

default=0.0

--nodeStorage NODESTORAGE

Specify the size of the root volume of worker nodes

when they are launched in gigabytes. You may want to

set this if your jobs require a lot of disk space.

(default=50).

--nodeStorageOverrides NODESTORAGEOVERRIDES

Comma-separated list of nodeType:nodeStorage that are

used to override the default value from --nodeStorage

for the specified nodeType(s). This is useful for

heterogeneous jobs where some tasks require much more

disk than others.

--metrics METRICS     Enable the prometheus/grafana dashboard for monitoring

CPU/RAM usage, queue size, and issued jobs.

--assumeZeroOverhead ASSUME_ZERO_OVERHEAD

Ignore scheduler and OS overhead and assume jobs can

use every last byte of memory and disk on a node when

autoscaling.
Toil options for limiting the number of service jobs and detecting service deadlocks:

Allows the specification of the maximum number of service jobs in a

cluster. By keeping this limited we can avoid nodes occupied with services

causing deadlocks.
Toil options for cores/memory requirements.:

The options to specify default cores/memory requirements (if not specified

by the jobs themselves), and to limit the total amount of memory/cores

requested from the batch system.
--defaultMemory DEFAULTMEMORY

The default amount of memory to request for a job.

Only applicable to jobs that do not specify an

explicit value for this requirement. Standard suffixes

like K, Ki, M, Mi, G or Gi are supported. Default is

2.0 Gi.

--defaultCores FLOAT  The default amount of cpu to request for a job. Only

applicable to jobs that do not specify an explicit

value for this requirement. Fractions of a core (for

example 0.1) are supported on some batch systems

[mesos, single_machine]. Default is 1.

--defaultDisk INT     The default amount of disk to request for a job. Only

applicable to jobs that do not specify an explicit

value for this requirement. Standard suffixes like K,

Ki, M, Mi, G or Gi are supported. Default is 2.0 Gi.

--defaultAccelerators ACCELERATOR[,ACCELERATOR...]

The default amount of accelerators to request for a

job. Only applicable to jobs that do not specify an

explicit value for this requirement. Each accelerator

specification can have a type (gpu [default], nvidia,

amd, cuda, rocm, opencl, or a specific model like

nvidia-tesla-k80), and a count [default: 1]. If both a

type and a count are used, they must be separated by a

colon. If multiple types of accelerators are used, the

specifications are separated by commas. Default is [].

--defaultPreemptible [BOOL], --defaultPreemptable [BOOL]

Make all jobs able to run on preemptible (spot) nodes

by default.

--maxCores INT        The max amount of cpu to request for a job. Only

applicable to jobs that do not specify an explicit

value for this requirement. Fractions of a core (for

example 0.1) are supported on some batch systems

[mesos, single_machine]. Default is

9223372036854775807.

--maxMemory INT       The max amount of memory to request for a job. Only

applicable to jobs that do not specify an explicit

value for this requirement. Standard suffixes like K,

Ki, M, Mi, G or Gi are supported. Default is 8.0 Ei.

--maxDisk INT         The max amount of disk to request for a job. Only

applicable to jobs that do not specify an explicit

value for this requirement. Standard suffixes like K,

Ki, M, Mi, G or Gi are supported. Default is 8.0 Ei.
Toil options for rescuing/killing/restarting jobs.:

The options for jobs that either run too long/fail or get lost (some batch

systems have issues!).
--retryCount RETRYCOUNT

Number of times to retry a failing job before giving

up and labeling job failed. default=1

--enableUnlimitedPreemptibleRetries ENABLEUNLIMITEDPREEMPTIBLERETRIES, --enableUnlimitedPreemptableRetries ENABLEUNLIMITEDPREEMPTIBLERETRIES

If set, preemptible failures (or any failure due to an

instance getting unexpectedly terminated) will not

count towards job failures and --retryCount.

--doubleMem DOUBLEMEM

If set, batch jobs which die to reaching memory limit

on batch schedulers will have their memory doubled and

they will be retried. The remaining retry count will

be reduced by 1. Currently supported by LSF.

--maxJobDuration MAXJOBDURATION

Maximum runtime of a job (in seconds) before we kill

it (this is a lower bound, and the actual time before

killing the job may be longer).

default=9223372036854775807

--rescueJobsFrequency RESCUEJOBSFREQUENCY

Period of time to wait (in seconds) between checking

for missing/overlong jobs, that is jobs which get lost

by the batch system. Expert parameter. default=60
Toil log management options.:

Options for how Toil should manage its logs.
--maxLogFileSize MAXLOGFILESIZE

The maximum size of a job log file to keep (in bytes),

log files larger than this will be truncated to the

last X bytes. Setting this option to zero will prevent

any truncation. Setting this option to a negative

value will truncate from the beginning. Default=62.5

Ki

--writeLogs [WRITELOGS]

Write worker logs received by the leader into their

own files at the specified path. Any non-empty

standard output and error from failed batch system

jobs will also be written into files at this path. The

current working directory will be used if a path is

not specified explicitly. Note: By default only the

logs of failed jobs are returned to leader. Set log

level to 'debug' or enable '--writeLogsFromAllJobs' to

get logs back from successful jobs, and adjust

'maxLogFileSize' to control the truncation limit for

worker logs.

--writeLogsGzip [WRITELOGSGZIP]

Identical to --writeLogs except the logs files are

gzipped on the leader.

--writeLogsFromAllJobs WRITELOGSFROMALLJOBS

Whether to write logs from all jobs (including the

successful ones) without necessarily setting the log

level to 'debug'. Ensure that either --writeLogs or

--writeLogsGzip is set if enabling this option.

--writeMessages WRITE_MESSAGES

File to send messages from the leader's message bus

to.

--realTimeLogging REALTIMELOGGING

Enable real-time logging from workers to leader
Toil miscellaneous options.:

Everything else.
--disableChaining DISABLECHAINING

Disables chaining of jobs (chaining uses one job's

resource allocation for its successor job if

possible).

--disableJobStoreChecksumVerification DISABLEJOBSTORECHECKSUMVERIFICATION

Disables checksum verification for files transferred

to/from the job store. Checksum verification is a

safety check to ensure the data is not corrupted

during transfer. Currently only supported for non-

streaming AWS files.

--sseKey SSEKEY       Path to file containing 32 character key to be used

for server-side encryption on awsJobStore or

googleJobStore. SSE will not be used if this flag is

not passed.

--setEnv NAME=VALUE or NAME, -e NAME=VALUE or NAME

Set an environment variable early on in the worker. If

VALUE is null, it will be looked up in the current

environment. Independently of this option, the worker

will try to emulate the leader's environment before

running a job, except for some variables known to vary

across systems. Using this option, a variable can be

injected into the worker process itself before it is

started.

--servicePollingInterval SERVICEPOLLINGINTERVAL

Interval of time service jobs wait between polling for

the existence of the keep-alive flag. Default: 60.0

--forceDockerAppliance FORCEDOCKERAPPLIANCE

Disables sanity checking the existence of the docker

image specified by TOIL_APPLIANCE_SELF, which Toil

uses to provision mesos for autoscaling.

--statusWait STATUSWAIT

Seconds to wait between reports of running jobs.

--disableProgress DISABLEPROGRESS

Disables the progress bar shown when standard error is

a terminal.

--config CONFIG       Get options from a config file.
Toil debug options.:

Debug options for finding problems or helping with testing.
--debugWorker         Experimental no forking mode for local debugging.

Specifically, workers are not forked and stderr/stdout

are not redirected to the log.

--disableWorkerOutputCapture

Let worker output go to worker's standard out/error

instead of per-job logs.

--badWorker BADWORKER

For testing purposes randomly kill --badWorker

proportion of jobs using SIGKILL. default=0.0

--badWorkerFailInterval BADWORKERFAILINTERVAL

When killing the job pick uniformly within the

interval from 0.0 to --badWorkerFailInterval seconds

after the worker starts. default=0.01
--custom-net CUSTOM_NET

Specify docker network name to pass to docker run

command
Options for recording the Docker container identifier into a file.:

--cidfile-dir CIDFILE_DIR

Store the Docker container ID into a file in the

specified directory.

--cidfile-prefix CIDFILE_PREFIX

Specify a prefix to the container ID filename. Final

file name will be followed by a timestamp. The default

is no prefix.
Options for recording provenance information of the execution:

--provenance PROVENANCE

Save provenance to specified folder as a Research

Object that captures and aggregates workflow execution

and data products.

--enable-user-provenance

Record user account info as part of provenance.

--disable-user-provenance

Do not record user account info in provenance.

--enable-host-provenance

Record host info as part of provenance.

--disable-host-provenance

Do not record host info in provenance.

--orcid ORCID         Record user ORCID identifier as part of provenance,

e.g. https://orcid.org/0000-0002-1825-0097 or

0000-0002-1825-0097. Alternatively the environment

variable ORCID may be set.

--full-name CWL_FULL_NAME

Record full name of user as part of provenance, e.g.

Josiah Carberry. You may need to use shell quotes to

preserve spaces. Alternatively the environment

variable CWL_FULL_NAME may be set.
Args that start with '--' can also be set in a config file (specified via

--config). The config file uses YAML syntax and must represent a YAML

'mapping' (for details, see http://learn.getgrav.org/advanced/yaml). In

general, command-line values override environment variables which override

config file values which override defaults.

[mr-c]

From a diff of the old and new options

1. --version no longer has a description (was "show program's version number and exit")
2. --custom-net is now listed as being part of extra_dockergroup (that should probably not be printed)

From a diff of toil 5.12

3. --runCwlInternalJobsOnWorkers is missing from the summary section
4. [--linkImports | --noLinkImports] is missing
5. [--moveExports | --noMoveExports] is missing
6. --disableCaching is missing

The metavars don't seem to be very helpful except to say that a particular command line option requires a value. Example: --disableAutoDeployment DISABLEAUTODEPLOYMENT ; wouldn't it be better to say --disableAutoDeployment BOOLEAN or similar? Likewise --config PATH instead of --config CONFIG, etc..

I appreciate that the environment variables are listed directly, thank you!