Various hints on settings needed to get various MPIs running with SWIFT.
Last update 5th May 2020.
Intel MPI 2018 usually runs without any needs for special settings.
Intel MPI 2019 and 2020 can run for small tests, but without flags will generally deadlock in the MPI exchanges of the engine, or worse. In that case try the following settings.
FI_OFI_RXM_RX_SIZE=4096
FI_OFI_RXM_TX_SIZE=4096
FI_UNIVERSE_SIZE=2048
If you want use the release_mt library, then you also need to use:
source $I_MPI_ROOT/intel64/bin/mpivars.sh release_mt
when initializing the library environment. Some success has also been seen using the asynchronous progression settings:
I_MPI_ASYNC_PROGRESS=1
I_MPI_ASYNC_PROGRESS_THREADS=1
(note these are tested with 2019 update-4 and 2020 update-1 on Mellanox).
Open MPI comes in many flavours with many combinations of underlying transport libraries and running on many different fabrics. A complete description of all combinations is beyond the scope of this guide.
On Mellanox hardware, we have had success running version 4.0 with the UCX layer version 1.6 and using the following settings:
-mca coll_hcoll_enable 0
UCX_TLS=ud_x,shm,self
UCX_RC_MLX5_TM_ENABLE=n
UCX_DC_MLX5_TM_ENABLE=n
These environment variables (set on the Mellanox-base cosma system in Durham) for version 4.0.5 with the UCX backend can also be useful:
OMPI_MCA_btl ^openib
OMPI_MCA_btl_openib_flags 1
OMPI_MCA_plm_rsh_num_concurrent 768
OMPI_MCA_mpool_rdma_rcache_size_limit 209715200