toil-cwl-runner creates broken symlinks when creating a directory within a singularity container

[jackerschott]

Create the following minimal example

.
├── job.yaml
├── src
│   └── bla
│       └── blubb.py
├── test.py
└── tool.cwl

with

tool.cwl
---
cwlVersion: v1.2
class: CommandLineTool
baseCommand: env
arguments: [PYTHONPATH=$(inputs.src.path), python3]

hints:
    DockerRequirement:
        dockerPull: "url/to/some/container/with/python/environment"

inputs:
    src:
        type: Directory
    script:
        type: File
        inputBinding:
            position: 0

outputs: []

job.yaml
---
src:
    class: Directory
    path: src
script:
    class: File
    path: test.py

test.py
---
import os
blubb_path = os.path.join(os.environ["PYTHONPATH"], "bla/blubb.py")
blubb_path_real = os.readlink(blubb_path)
os.system(f'ls -l {blubb_path} 2>&1')
os.system(f'ls -l {blubb_path_real} 2>&1')

# trigger python err to see output of above
bla

Now, running toil-cwl-runner --singularity tool.cwl job.yaml will output something like

lrwxrwxrwx 1 <user> <group> <time> /var/lib/cwl/stgb45cfd3c-91f0-4075-860d-496e70ae3e2a/src/bla/blubb.py -> /tmp/tmpbyzwn__8/files/no-job/file-93eadd867dc84ce6852512a1d58acafa/blubb.py
ls: cannot access '/tmp/tmpbyzwn__8/files/no-job/file-93eadd867dc84ce6852512a1d58acafa/blubb.py': No such file or directory

i.e. toil is creating a symlink to blubb.py here that is broken. My guess would be that it points to the environment outside of the container. This is in particular a problem when you want to e.g. import bla.blubb in test.py, i.e. import from a module in the executing script.

My toil version is 5.12.0-6d5a5b83b649cd8adf34a5cfe89e7690c95189d3. Note that adding --noLinkImports to the command line doesn't change the result.

┆Issue is synchronized with this Jira Story
┆Issue Number: TOIL-1461

[mr-c]

FYI, when running your example using the CWL reference runner, cwltool I had to modify os.readlink to os.path.realpath as with cwltool it isn't a symlink.

With that change, and using #4707 , your test case works, the only error is the expected NameError: name 'bla' is not defined

[adamnovak]

@jackerschott Have you tested to see if this is fixed on Toil 6 or later? It sounds like a cwltool version bump might have fixed it.