smudgeplot.py hetkmers massive memory and walltime required

[tallnuttrbgv]

Hi,

I have run this pipeline to generate the .dump file. However, smudgeplot.py hetkmers will not finish given 1200 GB of memory and 48 cpus:

NCPUs Used: 48      
CPU Time Used: 12:00:52
Memory Used: 371.57GB
Walltime Used: 12:01:22
JobFS used: 16.91GB 

The .dump file is 17GB and the original reads were HiFi, 67 GB. Is this normal? Could it take more than 12 hours or is there a bug. There was no error message and no progress in stderr, except:

Task: hetkmers
Kmers and coverages loaded.

Pipeline:

kmc -k21 -t48 -m256 -ci1 -cs10000 -fa $reads kmcdb tmp

kmc_tools transform kmcdb histogram kmcdb_k21.hist -cx10000

$(smudgeplot.py cutoff kmcdb_k21.hist L)
$(smudgeplot.py cutoff kmcdb_k21.hist U)

echo 'limits' $L $U

kmc_tools -t48 transform kmcdb -ci"$L" -cx"$U" dump -s kmcdb_L"$L"_U"$U".dump

smudgeplot.py hetkmers -o kmcdb_L"$L"_U"$U" < kmcdb_L"$L"_U"$U".dump

smudgeplot.py plot -o damp kmcdb_L"$L"_U"$U"_coverages.tsv`

[tallnuttrbgv]

My mistake, no L=, U=

[tallnuttrbgv]

I had L and U set anyway - so it was no the problem, it seems to be the same as previous issues - taking too long to run. Has this been solved yet? Thanks.

[KamilSJaron]

Hi,

this:
#137 (comment)