State of multi-gpu support

[mfzmullen]

If I had a system with multiple NVIDIA GPUs I would of course just test this by compiling with MATX_MULTI_GPU=ON, but what is the state of multi-gpu support? Do I need to make any library changes (besides the one CMakeLists.txt change) when switching from one-to-many GPUs? Thanks!

[cliffburdick]

Hi @mfzmullen, the state of multi-gpu is complicated and really depends on your use case. Some of our libraries on the backend support multi-gpu to some extent (BLAS, FFT), but not all of them. Can you let us know what you want to use multiple GPUs for? As much detail as you can provide will be appreciated.

[mfzmullen]

Use case (right now) is fairly simple. Doing basic arithmetic on tensors (~10^6 - 10^8 elements) to simulate some (decoupled) physics.

In the future, that will hopefully evolve into some BLAS/GEMM (FFT would also be useful but not required) on tensors/matrices that are too large to fit in current memory for solving inverse problems. I am familiar with methods on solving inverse problems when the entire forward operator can't fit into memory, but was curious on the state of multi-gpu support to see if I could break it up across gpus.

Not quite sure how much detail you are looking for, happy to provide more if needed though. I usually try to be succinct first and expand later.

[mfzmullen]

Oops, forgot to mention cuSPARSE would be useful too.

[cliffburdick]

Hi @mfzmullen, right now multi-gpu support is possibly manually by the user writing device/stream-specific code using cudaSetDevice and others. We currently do not support single operations that launch across multiple GPUs. A lot of times this is best left to the user anyways since there's a tradeoff with transfer times, and that transfer time is dependent on the system. Basic arithmetic on a tensor might fall into that category where trying to automatically handle the data shuffles could perform worse than doing it manually.

cuSPARSE is another that's difficult to support at this time since the tensor types would not be compatible with anything we have now. If we were to support it users would expect that all/most operations in MatX would just work with a sparse type, but in reality the only ones that would work are those supported by cuSPARSE.

[mfzmullen]

Great, really appreciate all the info! Thank you!