error while index building

[AlcaArctica]

I am running the following command:

/lustre/projects/dazzler/uelze/sw/bwa-mem2-lisa/bwa-mem2 index /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta

It seems to start of fine, but then doesnt continue after the RNK step. What could be the problem?

Looking to launch executable "/lustre/projects/dazzler/uelze/sw/bwa-mem2-lisa/bwa-mem2.avx2", simd = .avx2
Launching executable "/lustre/projects/dazzler/uelze/sw/bwa-mem2-lisa/bwa-mem2.avx2"
Absolute directory path: /lustre/projects/dazzler/uelze/sw/bwa-mem2-lisa
[lisa_index] Pack FASTA... * Entering FMI_search
init ticks = 754635906298
ref seq len = 19794865366
binary seq ticks = 516114232150
build suffix-array ticks = 16296061792486
ref_seq_len = 19794865366
count = 0, 6276208608, 9897432683, 13518656758, 19794865366
BWT[3542309723] = 4
CP_SHIFT = 6, CP_MASK = 63
sizeof CP_OCC = 64
pos: -1820609125, ref_seq_len__: 2474358170
max_occ_ind = 309294771
build fm-index ticks = 2517622081012
Read ref file done.
* Entering FMI_search
No existing /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta.rev_comp.8. Building...
ref file name for fmi: /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta size: 9897432683
19794865367: expected 19794865367: size of t
ref file name for fmi: /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta size = 19794865367
ref seq size n = 19794865367, Chunk size K = 20, rmi leaf nodes = 536870912
NUM_IPBWT_BYTES = 10
No existing /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta.rev_comp.ipbwt.8.8.20.36.44. Building...
SA done in 274938501864 cycles.
rnk allocated, n = 19794865367
RNK done in 4545942891960 cycles.
bwa-mem2: ./ext/TAL/src/LISA-FMI/ipbwt_rmi.h:295: IPBWT_RMI<index_t, kenc_t>::IPBWT_RMI(const string&, index_t, std::__cxx11::string, int, int64_t, index_t*, std::__cxx11::string)::<lambda()> [with index_t = long int; kenc_t = long unsigned int]: Assertion `ipbwt_array != __null' failed.
/usr/bin/bash: line 3: 15721 Aborted                 /lustre/projects/dazzler/uelze/sw/bwa-mem2-lisa/bwa-mem2 index /projects/dazzlerAssembly/asm_vpTaxBacc_BK34-6/arima/linked_assemblies/assembly_haplotig1.fasta

rust is installed:

rustc --version
rustc 1.74.0 (79e9716c9 2023-11-13)
cargo --version
cargo 1.74.0 (ecb9851af 2023-10-18)
rustup --version
rustup 1.26.0 (5af9b9484 2023-04-05)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.74.0 (79e9716c9 2023-11-13)

[ifdongs]

hi,
I encountered the issue of insufficient CPU utilization when running multiple instances of BWA2 simultaneously in same host:

info:

• fastq: 20M reads(150*2)
• bwa2: bwa-mem2.avx512bw ( 2.2.1 )
• cmd: bwa-mem2.avx512bw mem -M -t 8 hg19.fa fq1 fq2 >/dev/null 2>/dev/null

host:

• host1: 94G 40cpu (Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz)
• host2: 94G 40cpu (Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz)
• host3: 251G 80cpu (Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz)

system-version:

Linux version 2.6.32-754.35.1.el6.x86_64 (mockbuild@x86-02.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Sat Nov 7 12:42:14 UTC 2020

I am running multi bwa2 of the same command simultaneously on the 3 hosts mentioned above. The table below shows the results of my analysis.

			host1	host2	host3
bwa2_tasks	cpu	mem_usage(G)	cpu_usage(%)	cpu_usage(%)	cpu_usage(%)
1	1	16	100	100	100
1	2	16	200	200	200
3	3	17	300	300	300
4	3	17	250	260	290
4	2	16	200	200	200
4	4	17	340	340	340
3	4	17	400	400	400
3	8	18	800	800	800
3	12	20	1200	1200	1200
3	18	21	-	-	1500

now, there are some tissues:

1. Regardless of the number of CPUs, the minimum memory usage is 16GB, and the maximum is typically around 20GB. Is this different from the previous mentioned of 350MB~4GB/CPU?
2. If more than 2 CPUs are allocated for each BWA2, a maximum of 3 BWA2 tasks can be executed on the same host. Otherwise, the CPU utilization will not be saturated. Does this involve interference in CPU allocation for multiple BWA2 tasks in the same hosts?
3. When running the same cmd on the same host, the bwa2 index loading time sometimes takes 7 seconds, and sometimes 2 minutes. In the case of sufficient RAM, are there any other factors that affect the loading of the index?

Thank you,
chen