[Usage]: running workflow on workstation with local flag #138
Comments
|
I ran |
|
Hi Forrest, Thanks for the extensive report, I think I have an idea what might be going on, looks like the important error message is this:
Which means you probably have the Lines 83 to 84 in cb099b7 Hope this helps and please let me know if you have further questions. Best, p.s. if possible, its recommended to run metaGEM on a high performance computer cluster rather than local workstation p.p.s. this is expected behavior btw, since the qfilter rule fails then Snakemake deletes the output folder as a precaution.
|
franciscozorrilla
added
question
|
Hi Francisco, Thanks for getting back to me so quickly, this resolved my problems with fastp and crossMapSeries. However, I a similar problem with the binRefine step, see error below. I am using a server that will be able to handle my data (not that many samples), but it does not have slurm setup; hence the I silenced the lines that activate the conda env, as suggest in a previous issue. #104 (comment). Silenced or not, this step gives an error. I did configure GTDB-tk and CheckM.
=================================================================================================================================
|
|
Hi Forrest, Based on your error messages it looks like concoct and maxbin fail before even getting to the binRefine stage.
Could you try deleting everything inside the tmp folder, and then re-running the jobs? e.g. Sorry you are having trouble with this. For some background: the workflow was tested and designed to run on slurm-based HPC infrastructures, with some users even expanding to qsub-based infrastructures. However, since I do not have access to workstations, I have not developed support for this kind of infrastructure. The local command/flag is mostly meant for troubleshooting, debugging, testing, etc. I am happy, however, to help you to the best of my ability to get metaGEM running on your workstation 💎 |
franciscozorrilla
changed the title
|
Hi Francisco, I really appreciate your help! I deleted all files and started from dataset. I ran
|
|
This is the log file
|
Regarding the above maxbin errors, looks like you are missing some perl library. Try could try and install it manually. Looks like others have reported this error message, maybe read throught this issue here: metagenome-atlas/atlas#328
Regarding the above concoct errors, looks like it has to do with the sklearn version, see these issues for more details on how to solve this BinPro/CONCOCT#323 BinPro/CONCOCT#322 Regarding metabat2, from the logs it looks like it may have succesfully generated some draft bins. After fixing the issues with concoct and maxbin then you should be able to run the refinement and reassembly modules. Remember to try cleaning up the tmp dir in between runs to avoid any existing intermediate file issues that may cause problems. Best, |
|
Hi, I'm using pbs to submit jobs to the server, but as with the command line alone, I have to use the "--local" flag, otherwise I get "This was a dry-run (flag -n). The order of jobs does not reflect the order of execution." and no program will run. projectname="metaGEM_comand" cd $project echo -e "" yes | bash metaGEM.sh -t fastp -j 2 -c 120 -m 500 -h 24 --local |
|
In addition to that, I also found that when using "--local", it seems that no matter how many nodes are applied to the server, the program will only run on one node, and not multiple nodes at the same time in parallel, because I got an out of memory error when I ran GTDB-TK, but in fact, I applied 10 24-core 96GB nodes when I ran GTDB-TK with "--local", and it shouldn't have generated this error: |
|
@franciscozorrilla Do you know how to solve this problem? |
|
@franciscozorrilla I think it might be a problem with the conditional judgment here, but I don't know why I get this message when I don't set the "--local" flag: "This was a dry-run (flag -n). The order of jobs does not reflect the order of execution." In，metagem.sh Parse snakefile for cluster/local jobselif [ $task == "fastp" ]; then |
|
Hey Wupeng,
I do not understand the reasoning here. Why are you trying to use the local flag when submitting jobs to the cluster? In general, you should never run jobs locally on the cluster. The local flag is only for usage with workstations, where there is no scheduler or other users.
This behavior is exactly as expected. To submit jobs to the cluster, instead of running them locally, remove the --local flag. In fact, if you add the local flag, Snakemake will launch the jobs directly in the node you are running and that is why you are running out of memory.
If you are using a pbs cluster then have a look a this discussion and this fork. @fbartusch modified the metaGEM.sh wrapper file and cluster config file to allow submission on qsub, if I were you I would look at those modifications and apply them to your files as well Hope this helps and let me know if you have further questions! p.s. feel free to open up a new issue/discussion :) |
Thanks for your reply, I did find this problem too, when I unchecked "--local", the nohup.out file mentions that the "sbatch" command can't be found, so as you mentioned, it's indeed a problem with qsub! I will take the next step as you suggested, thanks again! |
|
Hello, when I set qsub like this, the following error occurred： RuleException in line 791 of /public/home/wangjj/WP/metaGEM/workflow/Snakefile: |
|
Hi wupeng, it sounds like your issues are no longer related to the original post. Please feel free to open up a new issue and provide more details. Based on the error message, it seems like Snakemake is not properly communicating with your cluster. First make sure that Snakemake is working properly, you should be able to submit/run simple Snakemake jobs on your cluster before trying to use metaGEM. For reference, please see the Snakemake docs and tutorial. |
|
Ok,i know what you means. But the snakemake is work well when i use the --local flag.so i think there maybe not the snakemake false |
|
Please, do not run jobs on the login node of your cluster using the --local flag. This is improper and harmful usage of the cluster, and you will probably get complaints from your HPC admins. Snakemake is supposed to communicate with your HPC job scheduler and submit jobs to compute nodes, as opposed to launching them on the login node which is what you are doing with the --local flag. |
Hi Francisco,
I recently downloaded metagem, I used the manual installation guideline. I'm trying to run fastp but keep getting this error. I am unsure what could be causing this. I was able to run createFolders, downloadToy, organizeData, and check with no problems. I am using sample1 from the toy dataset.
(metagem) dail@swamp:~/metaGEM/workflow$ bash /home/dail/metaGEM/workflow/metaGEM.sh -t fastp -l=================================================================================================================================
Developed by: Francisco Zorrilla, Kiran R. Patil, and Aleksej Zelezniak___________________________________________________________
Publication: doi.org/10.1101/2020.12.31.424982___________________________/\\\/\\\\/\/\
/\///////////\////////////\_/\_
/\/\/\/\//\/\//\
/\_/\_/\\/\\\/\\_/\/\\/\\\/\///\/\//\
/\///\///\/\/////\////\////////////\/\/////\/\////////\///\//\
/\//\/\/\\\/\/\\_/\/\/\/\____////\_
_______/\/\/\//////////\/_/\/////\/\_______/\/\/\_/\
_/\/\/\//\\_//\_//\\/_//\\\//\\\\/\____________/\
//////////////////////////////////___//////////////////////////////___________///
A Snakemake-based pipeline desinged to predict metabolic interactions directly from metagenomics data using high performance computer clusters
Version: 1.0.5
Setting current directory to root in config.yaml file ...
Parsing Snakefile to target rule: fastp ...
Do you wish to continue with these parameters? (y/n)y
Proceeding with fastp job(s) ...
Please verify parameters set in the config.yaml file:
path:
root: /home/dail/metaGEM/workflow
scratch: $TMP
folder:
data: dataset
logs: logs
assemblies: assemblies
scripts: scripts
crossMap: crossMap
concoct: concoct
maxbin: maxbin
metabat: metabat
refined: refined_bins
reassembled: reassembled_bins
classification: GTDBTk
abundance: abundance
GRiD: GRiD
GEMs: GEMs
SMETANA: SMETANA
memote: memote
qfiltered: qfiltered
stats: stats
proteinBins: protein_bins
dnaBins: dna_bins
pangenome: pangenome
kallisto: kallisto
kallistoIndex: kallistoIndex
benchmarks: benchmarks
prodigal: prodigal
blastp: blastp
blastp_db: blastp_db
scripts:
kallisto2concoct: kallisto2concoct.py
prepRoary: prepareRoaryInput.R
binFilter: binFilter.py
qfilterVis: qfilterVis.R
assemblyVis: assemblyVis.R
binningVis: binningVis.R
modelVis: modelVis.R
compositionVis: compositionVis.R
taxonomyVis: taxonomyVis.R
carveme: media_db.tsv
toy: download_toydata.txt
GTDBtkVis:
cores:
fastp: 4
megahit: 48
crossMap: 48
concoct: 48
metabat: 48
maxbin: 48
refine: 48
reassemble: 48
classify: 2
gtdbtk: 48
abundance: 16
carveme: 4
smetana: 12
memote: 4
grid: 24
prokka: 2
roary: 12
diamond: 12
params:
cutfasta: 10000
assemblyPreset: meta-sensitive
assemblyMin: 1000
concoct: 800
metabatMin: 50000
seed: 420
minBin: 1500
refineMem: 1600
refineComp: 50
refineCont: 10
reassembleMem: 1600
reassembleComp: 50
reassembleCont: 10
carveMedia: M8
smetanaMedia: M1,M2,M3,M4,M5,M7,M8,M9,M10,M11,M13,M14,M15A,M15B,M16
smetanaSolver: CPLEX
roaryI: 90
roaryCD: 90
envs:
metagem: envs/metagem
metawrap: envs/metawrap
prokkaroary: envs/prokkaroary
Please pay close attention to make sure that your paths are properly configured!
Do you wish to proceed with this config.yaml file? (y/n)y
Unlocking snakemake ...
Unlocking working directory.
Dry-running snakemake jobs ...
Building DAG of jobs...
Job counts:
count jobs
1 all
1 qfilter
2
[Mon Aug 28 20:08:14 2023]
rule qfilter:
input: /home/dail/metaGEM/workflow/dataset/sample1/sample1_R1.fastq.gz, /home/dail/metaGEM/workflow/dataset/sample1/sample1_R2.fastq.gz
output: /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R1.fastq.gz, /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R2.fastq.gz
jobid: 1
wildcards: IDs=sample1
[Mon Aug 28 20:08:14 2023]
Job 0:
WARNING: Be very careful when adding/removing any lines above this message.
The metaGEM.sh parser is presently hardcoded to edit line 22 of this Snakefile to expand target rules accordingly,
therefore adding/removing any lines before this message will likely result in parser malfunction.
Job counts:
count jobs
1 all
1 qfilter
2
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
Do you wish to submit this batch of jobs on your local machine? (y/n)y
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 20
Rules claiming more threads will be scaled down.
Conda environments: ignored
Job counts:
count jobs
1 all
1 qfilter
2
Select jobs to execute...
[Mon Aug 28 20:08:16 2023]
rule qfilter:
input: /home/dail/metaGEM/workflow/dataset/sample1/sample1_R1.fastq.gz, /home/dail/metaGEM/workflow/dataset/sample1/sample1_R2.fastq.gz
output: /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R1.fastq.gz, /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R2.fastq.gz
jobid: 1
wildcards: IDs=sample1
Activating envs/metagem conda environment ...
/usr/bin/bash: line 2: activate: No such file or directory
[Mon Aug 28 20:08:16 2023]
Error in rule qfilter:
jobid: 1
output: /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R1.fastq.gz, /home/dail/metaGEM/workflow/qfiltered/sample1/sample1_R2.fastq.gz
shell:
Creating temporary directory$TMP/qfiltered/$ {idvar} ... "$TMP/qfiltered/$ {idvar}
mkdir -p
Job failed, going on with independent jobs.
Exiting because a job execution failed. Look above for error message
Complete log: /home/dail/metaGEM/workflow/.snakemake/log/2023-08-28T200816.260324.snakemake.log
The text was updated successfully, but these errors were encountered: