Non deterministic convergence history using NEWTON-KRYLOV with OMP

[tbellosta]

Convergence history is non deterministic when running the RANS solver with SA model using OMP-only parallelism. When fixing the min RMS residual, the computation stops at a different iteration each time I run the solver.

That does not happen with the EULER or NAVIER_STOKES solvers.

Convergence history is non deterministic when using the NEWTON_KRYLOV solver using OMP-only parallelism. When fixing the min RMS residual, the computation stops at a different iteration each time I run the solver.
That happens with the EULER or NAVIER_STOKES and RANS solvers.

I am using a common setup for each case i.e. ROE+MUSCL convective fluxes, WLSQ for viscous gradients, LSQ for MUSCL gradients. Tried also switching ON and OFF the CFL adaptation and limiters, different linear solvers, and precision (single/double) of the linear solver but that doesn't change the outcome of the test (RANS conv history remains non deterministic while EULER and NS have always the same convergence history).

I compiled in release (-O3), with OMP support and exported the mtune/march variables to enable SIMD types.

I'll be looking in the SA/Scalar solver for some uninitialized vars. Any other idea?

I'll be looking in the CNewtonIntegration files for some uninitialized vars/thread safety issue. Any other idea?

Bug report checklist

• Consulting the build instructions (https://su2code.github.io/docs_v7/Build-SU2-Linux-MacOS/).
• Looking for similar problems on GitHub or CFD-Online (https://www.cfd-online.com/Forums/su2/).
• Updating to the newest version of SU2 (either develop, master, or the pre-built executables https://su2code.github.io/download.html).

Desktop (please complete the following information):

• OS: Linux (Ubuntu 18.4 LTS)
• C++ compiler and version: g++ (GCC) 7.5.0
• MPI implementation and version: OpenMPI 2.1.1
• SU2 Version: 8.0.1

[tbellosta]

SCRATCH THAT

convergence history is non-deterministic when using the NEWTON-KRYLOV solver, regardless of the equation set.
Using the standard linear solver, the RANS-EULER-NS solvers always converge in the same number of iterations. When switching on the NEWTON-KRYLOV option, the solver converges in a different number of iterations each time I run the solver.

[pcarruscag]

Certain parts of the code use atomic reductions, and dynamic work sharing for load balancing. That's why there is slight (hopefully) non determinism.
There are more operations per iteration when using the NK solver, which makes it more noticeable.