Transfer to target volume failed

[katsikora]

Complete log: .snakemake/log/2025-01-31T104717.066239.snakemake.log
unlocking
removing lock
removing lock
removed all locks
stderr:
<paramiko.ChannelFile from <paramiko.Channel 0 (closed) -> <paramiko.Transport at 0x3037dac0 (cipher aes128-ctr, 128 bits) (active; 0 open channel(s))>>>
Copy QC reports
... to bioinfo: /data/manke/group/pipegrp/qc/20250121_1622_2D_PAW05182_8625922c
... to sambahost: /home/pipegrp/samba/Sequence_Quality_2025/ONT_2025/20250121_1622_2D_PAW05182_8625922c
copy /longus/data/sequencing_data3/20250121_1622_2D_PAW05182_8625922c/transfer/Project_3482_Abini-Agbomson_Cisse/QC/
Warning. json does not exit: /data/cisse/sequencing_data/OxfordNanopore/20250121_1622_2D_PAW05182_8625922c/FASTQC_Project_3482_Abini-Agbomson_Cisse/multiqc/multiqc_data/multiqc_data.json
Traceback (most recent call last):
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/bin/ont", line 8, in <module>
    sys.exit(ont())
             ^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/npr/ont.py", line 141, in ont
    main(config)
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/npr/ont.py", line 221, in main
    config["SM"] = monitor_storage(config)
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/site-packages/npr/snakehelper.py", line 22, in monitor_storage
    du_out = sp.check_output(
             ^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/localenv/pipegrp/anaconda/miniconda3/envs/ont/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['du', '-sh', '/data/cisse/sequencing_data/OxfordNanopore/20250121_1622_2D_PAW05182_8625922c']' returned non-zero exit status 1.

happened twice

[adRn-s]

Maybe due to NFS Latencies while writing to disk?

btw, the error message comes from here https://github.com/maxplanck-ie/nanoporeReads_dataTransfer/blob/main/src/npr/snakehelper.py#L65

[katsikora]

But is that not just a warning?

[adRn-s]

Yes, it's just a print statement that the file wasn't found. Perhaps we could do something like this *1, and see if that helps? The actual error comes a bit later, the du command failed... I think both may be due to NFS latencies, we could use the first as a proxy to ugly-patch on the second since we can't control that part, 🤷🏽

*1

import time
from datetime import datetime, timedelta

timeout = datetime.now() + timedelta(minutes=5)
while not os.path.exists(json):
    if datetime.now() > timeout:
        print(f"[red]Warning. json does not exist: {json}[/red]")
        return QC
    time.sleep(1)

This solution checks for the file every second, and times out after 5 minutes. Maintains the original behaviour but it gives a small time window if things were running too fast (or the underlying networked storage was too slow.)

[katsikora]

the du command failed as no run folder was created under the target data sequencing volume