run without containers

[HannoSpreeuw]

make run on a cluster without docker gives
Docker is not available for this tool, try --no-container to disable Docker
So I tried
make run --no-container
but that gives
make: unrecognized option '--no-container'
So I tried
make --no-container
but that also gives
make: unrecognized option '--no-container'

[mr-c]

Hello @HannoSpreeuw

The makefile is provided as a convenience. This error message is from either cwltool or toil-cwl-runner so you could try the suggested option appended to the real command run

For example, try make run -n to see the commands that would be run.

In this instance I suggest make run-udocker as an alternative to Docker or Singularity. Otherwise you would need to install each of the software requirements of the workflow yourself.

[HannoSpreeuw]

Thanks.
I am afraid udocker is not available on this cluster:
make run-udocker
.....
.....
Error: create container: getting layers or json
Error: in download: HTTP/1.1 400 Bad Request
Error: create container: getting layers or json
Error: image or container not available
[job ndppp_prep_cal] Job error:
Error collecting output for parameter 'msout':
steps/ndppp_prep_cal.cwl:164:7: Did not find output file with glob pattern: '[u'calibrated.MS']'
[job ndppp_prep_cal] completed permanentFail
Workflow cannot make any more progress.
Final process status is permanentFail
make: *** [run-udocker] Error 1

[HannoSpreeuw]

But my colleague found this package for installation of udocker.

[HannoSpreeuw]

But is running prefactor-cwl without containers not possible at all?
Oh sorry, I now understand what you mean with

Otherwise you would need to install each of the software requirements of the workflow yourself

Running without containers would involve a lot of extra work.

[mr-c]

Currently the Makefile installs the main udocker branch: https://github.com/EOSC-LOFAR/prefactor-cwl/blob/master/Makefile#L24

Can you try out the latest devel udocker version?

rm .virtualenv/bin/udocker
curl https://raw.githubusercontent.com/indigo-dc/udocker/devel/udocker.py > .virtualenv/bin/udocker
chmod u+rx .virtualenv/bin/udocker
.virtualenv/bin/udocker install

[mr-c]

Also, what is the version of curl that is installed?

[HannoSpreeuw]

[hspreeuw@fs5 prefactor-cwl]$ curl --version
curl 7.49.0 (x86_64-pc-linux-gnu) libcurl/7.49.0 OpenSSL/1.0.2p zlib/1.2.11
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP UnixSockets

[HannoSpreeuw]

make run-udocker
[workflow ] start
[workflow ] starting step ndppp_prep_cal
[step ndppp_prep_cal] start
[job ndppp_prep_cal] Output of job will be cached in /home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f
[job ndppp_prep_cal] /home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f$ /home/hspreeuw/prefactor-cwl/.virtualenv/bin/udocker
--quiet
run
--volume=/home/hspreeuw/prefactor-cwl/cache/187eef3d15e77102e21a6feb47bb8d9f:/var/spool/cwl
--volume=/tmp/tmplmvobV:/tmp
--volume=/home/hspreeuw/prefactor-cwl/data/L570745_SB000_uv_first10.MS:/var/lib/cwl/stg44fa7578-bb33-475e-935a-45fc6824e3e3/L570745_SB000_uv_first10.MS
--workdir=/var/spool/cwl
--rm
--env=TMPDIR=/tmp
--env=HOME=/var/spool/cwl
kernsuite/prefactor
NDPPP
msout=calibrated.MS
average.freqresolution=48.82kHz
avg.freqstep=2
average.timeresolution=4
avg.timestep=2
avg.type=average
baseline=[CS013HBA*]
'filter.baseline=CS*, RS*&&'
filter.remove=True
filter.type=filter
'flag.baseline=[ CS013HBA* ]'
flag.type=filter
flagamp.amplmin=1e-30
flagamp.type=preflagger
msin=/var/lib/cwl/stg44fa7578-bb33-475e-935a-45fc6824e3e3/L570745_SB000_uv_first10.MS
msin.datacolumn=DATA
msout.overwrite=True
msout.writefullresflag=False
steps=[flag,filter,avg,flagamp]
Error: create container: getting layers or json
NDPPP: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory
[job ndppp_prep_cal] Job error:
Error collecting output for parameter 'msout':
steps/ndppp_prep_cal.cwl:164:7: Did not find output file with glob pattern: '[u'calibrated.MS']'
[job ndppp_prep_cal] completed permanentFail
Workflow cannot make any more progress.
Final process status is permanentFail
make: *** [run-udocker] Error 1

[HannoSpreeuw]

Needs lapack. Should be no problem.

[HannoSpreeuw]

module load lapack
make run-udocker
same error

[HannoSpreeuw]

Very weird. We have /usr/lib64/liblapack.so.3
export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH
Still the same error.

[gijzelaerr]

Hi @HannoSpreeuw all these things have been more of a tech demo than a plug-and-play and ready-to-use solution. I'm not sure why udocker doesn't work, it looks like there might be a bug with it and it doesn't initialise the shared libraries properly inside the containers. Can't you use Singularity?

[mr-c]

Is this the default dataset, or did you edit anything?

Error: create container: getting layers or json
NDPPP: error while loading shared libraries: liblapack.so.3: cannot open shared object file: No such file or directory

Means that liblapack inside the container was not found (for whatever reason) so any changes outside the container (like module load ... ) won't fix this issue.

[HannoSpreeuw]

That makes sense....duh.
No I didn't edit anything.
Just
make small
make run-udocker

[HannoSpreeuw]

Somehow this error disappeared when I got more disk space on the cluster.
But that would not have any effect on lapack issues, one would think.
However, there is an error message before the lapack error:
Error: create container: getting layers or json

Now
make run-udocker gives #23