Problem with the example TurbPipePeriodic

[Chaggge]

Hi, neks!
I am new with nekrs.

I have compiled nekrs without errors as it is described at: https://github.com/Nek5000/nekRS.
Then I tried to run TurbPipePeriodic using mpirun -np 2 nekrs --setup turbPipe.par. But I faced with the problem:
Message : Device: Creating Device CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE

My configuration:
gcc (Debian 10.2.1-6) 10.2.1 20210110
NVIDIA (R) CUDA Debugger 11.2 release
GPU: GeForce GTX 1050 Ti

[yslan]

Do you have two GPUs on your machine?

[Chaggge]

No, only one

[yslan]

You will need 2 GPU to run with 2 processors. NekRS only support 1-1 CPU-GPU binding.

[Chaggge]

I also have one integrated graphics processor

[vivashchenko]

@yslan Thanks, with -np 1 it works! Does it mean that I can use only 1 CPU core with all GPU's cores?

[yslan]

Yes. The mpi is assigned to a CPU core then it's binding to a GPU. The computation is done in parallel in terms of GPU threads.

The nvidia A100 does support 7 instances that can run different jobs under a GPU. For that case, it is possible (but not recommended) to bind 7 cpu cores with single A100.

[ME-20244]

Hello all!

I am new in using NekRS! I successfully built NekRS alone and within Cardinal on a personal station with one GPU.
In running any case involving nekrs, I get the errors:
Error in device_t: Active occa mode does not match selected backend! OR
Device: Creating Device CUDA Error [ 101 ]: CUDA_ERROR_INVALID_DEVICE

occa info : indicates the presence of GPU and OPENCL.

nvidia-smi gives for me:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.07 Driver Version: 535.161.07 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA T1000 Off | 00000000:01:00.0 Off | N/A |
| 33% 41C P8 N/A / 50W | 396MiB / 4096MiB | 4% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1940 G /usr/lib/xorg/Xorg 166MiB |
| 0 N/A N/A 2219 G /usr/bin/gnome-shell 78MiB |
| 0 N/A N/A 24563 G ...irefox/3728/usr/lib/firefox/firefox 148MiB |
+---------------------------------------------------------------------------------------+

Is it a problem of configuration of GPU or nekrs cannot run on such machine ?
How can I set CUDA_VISIBLE_DEVICES in bashrc ?
Any help, PLEASE !

[yslan]

Need more information.

Have you tried running nekrs standalone? What version are you using?
How did you launch nekrs?
What's the full output from occa info? Does it detect CUDA?

[stgeke]

I think it makes more sense to run the code using the serial backend as the GPU is not really powerful.

[ME-20244]

Hi !
Thank you very much for your reply and guidance.
I am using version 23.0, and occa informs me that I have CUDA installed.
In a standalone version, I changed the backend to CPU and it works for me but with "nrsmpi ethier 2" since nekrs gives error:
An error occurred in MPI_Win_create reported by process [3372023809,0] on communicator MPI COMMUNICATOR 5 DUP FROM 3.

[stgeke]

Could you please provide the complete output when running the code?

[ME-20244]

Hi !
I- For the running case, a piece of execution is:

runtime statistics (step= 17000 totalElapsed= 100151s):
name time abs% rel% calls
solve 1.00137e+05s 100.0
min 4.63299e+00s
max 1.14286e+01s
flops/rank 1.06249e+10
checkpointing 3.71194e-01s 0.0 5
udfExecuteStep 2.83827e-02s 0.0 17000
makef 4.34028e+04s 43.3 17000
udfProperties 1.78460e-02s 0.0 17001
velocitySolve 1.68422e+04s 16.8 17002
rhs 1.88037e+03s 1.9 11.2 17002
preconditioner 1.17190e+03s 1.2 7.0 135246
pressureSolve 3.61099e+04s 36.1 17002
rhs 2.83055e+03s 2.8 7.8 17002
preconditioner 2.35338e+04s 23.5 65.2 88962
pMG smoother 1.84975e+04s 18.5 78.6 177924
pMG smoother 3.43554e+03s 3.4 14.6 177924
coarse grid 4.24077e+01s 0.0 0.2 88962
initial guess 2.57845e+03s 2.6 7.1 17002

projP : resNorm0 2.59e-02 resNorm 2.62e-03 ratio = 9.899e+00 6/8
P : iter 005 resNorm0 2.62e-03 resNorm 7.46e-05
UVW : iter 008 resNorm0 2.99e-03 resNorm 9.30e-07 divErrNorms 1.49e-10 1.52e-01
flowRate : uBulk0 1.00e+00 uBulk 1.00e+00 err 7.77e-16 scale 5.39812e-02
step= 17001 t= 1.01988000e+02 dt=6.0e-03 C= 2.93 elapsedStep= 5.92e+00s elapsedStepSum= 1.00143e+05s
projP : resNorm0 2.59e-02 resNorm 2.20e-03 ratio = 1.174e+01 7/8
P : iter 005 resNorm0 2.20e-03 resNorm 9.12e-05
UVW : iter 008 resNorm0 2.99e-03 resNorm 9.29e-07 divErrNorms 8.06e-11 1.52e-01
flowRate : uBulk0 1.00e+00 uBulk 1.00e+00 err 4.44e-16 scale 5.39671e-02

II - For the failed case, I re-Run again and gives:
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
hwloc/linux: Ignoring PCI device with non-16bit domain.
Pass --enable-32bits-pci-domain to configure to support such devices
(warning: it would break the library ABI, don't enable unless really needed).
UTC time: Mon Apr 29 14:26:23 2024

             __    ____  _____

____ ___ / /__ / __ / /
/ __ \ / _ \ / //// // /_ \
/ / / // // ,< / , // /
// // __///||// ||/____/ v23.0.0 (no sha)

COPYRIGHT (c) 2019-2023 UCHICAGO ARGONNE, LLC

MPI tasks: 1

reading par file ...
Initializing device
active occa mode: Serial

using NEKRS_HOME: /home/me/.local/nekrs
using NEKRS_CACHE_DIR: /home/me/nekRS/examples/turbPipePeriodic/.cache
using OCCA_CACHE_DIR: /home/me/nekRS/examples/turbPipePeriodic/.cache/occa/

skip building nekInterface (SIZE requires no update)
loading nek ... done

<<< [GENERAL]
<<< #verbose = true
<<< polynomialOrder = 7
<<< #startFrom = "restart.fld"
<<< stopAt = endTime
<<< endTime = 200
<<<
<<< dt = 6.0e-3
<<< timeStepper = tombo2
<<< subCyclingSteps = 2
<<<
<<< writeControl = simulationTime
<<< writeInterval = 20
<<<
<<< regularization = hpfrt + nModes=1 + scalingCoeff=10
<<< constFlowRate = meanVelocity=1.0 + direction=Z
<<<
<<< [PRESSURE]
<<< residualTol = 1e-04
<<<
<<< [VELOCITY]
<<< boundaryTypeMap = wall
<<< viscosity = -19000
<<< density = 1.0
<<< residualTol = 1e-06
<<<
<<< [CASEDATA]
<<< Re_tau = 550
<<< zLength = 6
<<<

<<< //
<<< // nekRS User Defined File
<<< //
<<< #include "udf.hpp"
<<<
<<< static dfloat Re_tau;
<<< static dfloat zLength;
<<< static dfloat dt0;
<<<
<<< #include "ci.inc"
<<<
<<<
<<< void userf(nrs_t *nrs, dfloat time, occa::memory o_U, occa::memory o_FU)
<<< {
<<< mesh_t *mesh = nrs->meshV;
<<< dfloat mue, rho;
<<< platform->options.getArgs("VISCOSITY", mue);
<<< platform->options.getArgs("DENSITY", rho);
<<< const dfloat RE_B = rho / mue;
<<< const dfloat DPDZ = 4 * (2 * Re_tau / RE_B) * (2 * Re_tau / RE_B);
<<<
<<< occa::memory o_FUz = o_FU + 2 * nrs->fieldOffset * sizeof(dfloat);
<<< platform->linAlg->fill(mesh->Nlocal, DPDZ, o_FUz);
<<< }
<<<
<<< void UDF_Setup0(MPI_Comm comm, setupAide &options)
<<< {
<<< platform->par->extract("casedata", "re_tau", Re_tau);
<<< platform->par->extract("casedata", "zlength", zLength);
<<<
<<< options.getArgs("CI-MODE", ciMode);
<<< if (ciMode) {
<<< ciSetup(comm, options);
<<< }
<<<
<<< if (platform->options.compareArgs("BUILD ONLY", "FALSE")) {
<<< double *const nek_cb_scnrs = (double *)nek::ptr("cb_scnrs");
<<< nek_cb_scnrs[0] = zLength;
<<< }
<<<
<<< platform->options.getArgs("DT", dt0);
<<< }
<<<
<<< void UDF_Setup(nrs_t *nrs)
<<< {
<<< if (platform->options.compareArgs("CONSTANT FLOW RATE", "FALSE")) {
<<< // assign function pointer to drive flow by constant mean pressure gradient
<<< udf.uEqnSource = &userf;
<<< }
<<< }
<<<
<<< void UDF_ExecuteStep(nrs_t *nrs, dfloat time, int tstep)
<<< {
<<< if (tstep < 5) {
<<< nrs->dt[0] = 0.25 * dt0;
<<< } else {
<<< nrs->dt[0] = dt0;
<<< }
<<<
<<< if (nrs->lastStep) {
<<< if (ciMode) {
<<< ciTestErrors(nrs, time, tstep);
<<< }
<<< }
<<< }
<<<

<<< #ifdef okl
<<<
<<< #include "turbPipe.oudf"
<<<
<<< #endif
<<< #ifdef okl
<<< void velocityDirichletConditions(bcData *bc){}
<<< void meshVelocityDirichletConditions(bcData *bc){}
<<< void velocityNeumannConditions(bcData *bc){}
<<< void pressureDirichletConditions(bcData *bc){}
<<< void scalarNeumannConditions(bcData *bc){}
<<< void scalarDirichletConditions(bcData *bc){}
<<< #endif
<<<

loading udf kernels ...
done (0.000407018s)
benchmarking hot kernels ...
JIT compiling kernels (this may take awhile if they are not in cache) ...
done (0.0805764s)

initializing linAlg ...
wdotp: 4.273e-03s

ENABLE GS COMM OVERLAP disabled

reading turbPipe.re2
hdr:#v002 7840 3 7840 hdr
[me-HP-Workstation-Desktop-PC] *** An error occurred in MPI_Win_create
[me-HP-Workstation-Desktop-PC] *** reported by process [3631546369,0]
[me-HP-Workstation-Desktop-PC] *** on communicator MPI COMMUNICATOR 5 DUP FROM 3
[me-HP-Workstation-Desktop-PC] *** MPI_ERR_WIN: invalid window
[me-HP-Workstation-Desktop-PC] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[me-HP-Workstation-Desktop-PC] *** and potentially your MPI job)

[yslan]

Modify your script to add --mca osc ucx should fix this issue.

mpirun --mca osc ucx -np $2 $bin 

My guess is, you are using ubuntu 20.04 or 22.04 and the MPI shipped from it's default apt repository.
open-mpi/ompi#10726

[ME-20244]

Many Thanks! that helped me a lot.