remote-job-local-storage-prefix gets parsed in local context

[IceFreez3r]

In our cluster environment the job specific tmp directory is accessible through both $MXQ_JOB_TMPDIR and $TMPDIR. The first one doesn't exist in the local context, the second one points to /scratch/local locally.
With

remote-job-local-storage-prefix: $TMPDIR

files are copied to /scratch/local

Retrieving /scratch/local/fs/resources/rnaseq/ENCSR700EBI.fastq from storage.

while with

remote-job-local-storage-prefix: $MXQ_JOB_TMPDIR

files land in the correct location

Retrieving /dev/shm/mxqd/mnt/job/51813391/fs/resources/rnaseq/ENCSR700EBI.fastq from storage.

[sebschmi]

This bug specifically prevents this snippet from the documentation from working:

If the shared scratch is e.g. specific for each job (e.g. controlled by a $JOBID), one can define a job-specific local storage prefix like this

default-storage-provider: fs
local-storage-prefix: /local/work/$USER
remote-job-local-storage-prefix: /local/work/$USER/$JOBID
shared-fs-usage:
  - persistence
  - software-deployment
  - sources
  - source-cache

[johanneskoester]

From the help text:

In case they [the env vars] shall be expanded only within the remote job, mask them with a leading backslash, i.e. $SLURM_JOB_ID.

But I agree, the example in the docs here should explain that as well. This is done here now: #27

[johanneskoester]

Actually the masking seems to never have properly worked. Hence, I have now changed the behavior into the more robust approach to always only expand the vars in this particular argument within the remote job: snakemake/snakemake#3222.