How to run on a cluster without workload manager

[slambrechts]

Dear Francisco,

I realize metaGEM was optimized to run on a HPC cluster system, but the cluster of our lab (40 cores, 500 GB RAM) operates without a workload manager. So as a user you have to take care you don't use all 40 cores, and that you do not submit more jobs then there are cores etc.

I have 43 (soil) samples organized into sample specific subdirectories within the dataset folder

If I run

bash metaGEM.sh --task megahit --local

and I set megahit cores to 24 in the config.yaml file,

Would that mean metaGEM would assemble the samples only one at a time? What I want to avoid is that the metagem parser submits all 43 jobs at once, because on this cluster submitting means starting

Best wishes,
Sam

[franciscozorrilla]

Hey Sam,

Good question. If you look at the metaGEM.sh script, specifically in line 358, you will see that only 1 job will be submitted at a time when running metaGEM with the --local flag:

metaGEM/metaGEM.sh

Lines 338 to 364 in a125bb7

	# Submit local function, similar to submitLogin() but can handle wildcard expanded rules for non-cluster usage
	submitLocal() {
	# Parse Snakefile rule all (line 22 of Snakefile) input to match output of desired target rule stored in "$string". Note: Hardcoded line number.
	echo "Parsing Snakefile to target rule: $task ... "
	sed -i "22s~^.*$~ $string~" Snakefile
	checkParams
	snakeConfig
	echo "Unlocking snakemake ... "
	snakemake --unlock -j 1
	echo -e "\nDry-running snakemake jobs ... "
	snakemake all -n
	while true; do
	read -p "Do you wish to submit this batch of jobs on your local machine? (y/n)" yn
	case $yn in
	[Yy]* ) echo "snakemake all -j 1 -k"|bash; break;;
	[Nn]* ) exit;;
	* ) echo "Please answer yes or no.";;
	esac
	done
	}

Let me know if you run into any issues with this.

Best,
Francisco

[slambrechts]

Hi @franciscozorrilla,

Thank you for your answer. Ok, so it should make sense then to set the number of cores for all jobs to the maximum number of threads I am allowed to use at any time (e.g. 20)? Otherwise, if I use the default values, when fastp is running for example, only 4 cores will be used, and 16 will be left unused?

Best,
Sam

[franciscozorrilla]

Hi Sam,

That is correct. When running metaGEM with the --local flag on a workstation (or a cluster without a scheduler), you should set the max number of threads you are allowed to use for each task in the config.yaml file. The metaGEM.sh parser only configures the number of cores in the cluster_config.json file, so you need to manually modify the number of threads in these lines of the config.yaml file:

metaGEM/config.yaml

Lines 42 to 59 in d81186a

	cores:
	fastp: 4
	megahit: 48
	crossMap: 48
	concoct: 48
	metabat: 48
	maxbin: 48
	refine: 48
	reassemble: 48
	classify: 2
	gtdbtk: 48
	abundance: 16
	carveme: 4
	smetana: 12
	memote: 4
	grid: 24
	prokka: 2
	roary: 12

Hope that is clear and let me know if you run into any issues with this!

Best,
Francisco

[slambrechts]

Hi Francisco,

Yes, very clear, thanks! When running fastp however, it seems 16 threads are used instead of 20. I get a warning that I specified 20 but only 16 will be used. But this seems to be inherent to fastp.

Best,
Sam