Asynchronous solvers for SubMeshes

[colmenaresj]

Hello, I was wondering if the linear solutions could be done asynchronously on the devices. I don't know if this is currently done already. If not, It could probably be useful to have this. I am thinking in the context of SubMeshes. Let's say you want to solve two subdomains which are maybe coupled at a boundary. From what I understand, you would need to find the solution on one subdomain, wait for the solution to finish, and then solve on the second subdomain. It would be nice if you didn't have to wait for the first solution to finish to launch the second one. If running on NVIDIA, the two could run on different CUDA streams, for example.

I guess the motivation is to do more computations concurrently and to utilize the parallel resources to the maximum. In general it is probably useful to do asynchronous (non-blocking) solves in some cases. I was wondering if this could be done.

[tzanio]

ping: @jandrej, @cjvogl

[v-dobrev]

Currently, MFEM GPU kernels are launched on the default stream 0.

Generalizing this will require some work because running on stream 0 provides implicit synchronizations that will have to be made explicit (at least in some cases) when not using stream 0. Another issue is that executing two solvers asynchronously will require separate CPU threads as well because most solvers are not just one GPU kernel but a sequence of GPU kernels with CPU logic in-between.

For solving different problems on different sub-meshes it may be better to consider mapping the different sub-problems to MPI sub-communicators. This way each set of non-overlapping MPI ranks (and their associated GPUs) will solve a separate problem. This way it is also possible to use the same GPU from different MPI ranks to overlap computations without the need to use separate streams. This approach will require the capability for sub-meshes to be re-defined to on sub-communicators which is currently not implemented, however, I suspect it is not going to be too hard to do.

[colmenaresj]

Thank you @v-dobrev . Your explanation is very clear, I understand the challenges of what I had proposed. I assumed it would be difficult, especially when using external solvers like Hypre, etc.

I like the approach of the mapping the sub-meshes to different MPI ranks. In Nek5000, a similar approach was used for overset grids. Maybe down the line I can help with implementing something like this. I have worked a lot with MPI, groups and inter-communicators. I am new to MFEM so for now I will get familiarized with the code.

[cjvogl]

@colmenaresj , do you have an asynchronous linear solver in mind? I'm involved in some work in developing Krylov-based asynchronous methods that would indeed be interesting to try for something like your two sub-domain coupled at an interface problem... perhaps after the implementation issues that @v-dobrev raised are addressed.

[colmenaresj]

Hi @cjvogl , not really, this was more of an exploratory question. It would be awesome to try something like that. Thanks. -Juan Diego
cc: @bclyons12