Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trouble Compiling with Visual Studio and Intel Fortran #67

Open
Bill-gitt opened this issue Aug 3, 2021 · 11 comments
Open

Trouble Compiling with Visual Studio and Intel Fortran #67

Bill-gitt opened this issue Aug 3, 2021 · 11 comments

Comments

@Bill-gitt
Copy link

Hello dear developers,

Summary:
I am working for a University project where I have access to Windows 10, Visual studio 2013 and Intel Parallel studio 2017 or Visual Studio 2019 Community along Intel OneApi (includes Fortran Classic 2021).

For about 1 month now I am trying to compile the CVode with Fortran Interface, from Sundials 5.7 package. I hope it has thread-safely even when the Fortran interface is used. Anyways after lots of effort I managed to compile and build CVode. Though when I run the Fortran examples in debug mode the program crashes.

The steps I followed for compilation:

  1. I am using Cmake-gui version 3.21.1 (have tried this with version 3.20 too). I properly start Cmake and point it to the source, build and install directories. Everything seems normal. Cmake locates my compiler C properly. Here is some output from Cmake:
Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.19042.
The C compiler identification is MSVC 19.29.30040.0
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.29.30037/bin/Hostx64/x64/cl.exe - skipped
Detecting C compile features
Detecting C compile features - done
Looking for sys/types.h
Looking for sys/types.h - found
Looking for stdint.h
Looking for stdint.h - found
Looking for stddef.h
Looking for stddef.h - found
Check size of int64_t
Check size of int64_t - done
Using int64_t for indices
Looking for _POSIX_TIMERS
Looking for _POSIX_TIMERS - not found
Performing Test COMPILER_HAS_DEPRECATED_ATTR
Performing Test COMPILER_HAS_DEPRECATED_ATTR - Failed
Performing Test COMPILER_HAS_DEPRECATED
Performing Test COMPILER_HAS_DEPRECATED - Success
Added NVECTOR_SERIAL module
Added NVECTOR_MANYVECTOR module
Added SUNMATRIX_BAND module
.
.
.

  1. The Cmake finds all the Sundial 5.7 source (Arcode, CVode....). I select only the CVode and Build_Fortran_Module_Interface and I deselect everything that has to do with testing and examples. Cmake now locates the Intel compiler properly:
Using int64_t for indices
The Fortran compiler identification is Intel 2021.2.0.20210228
Detecting Fortran compiler ABI info
Detecting Fortran compiler ABI info - done
Determine Intel Fortran Compiler Implicit Link Path
Determine Intel Fortran Compiler Implicit Link Path - done
Check for working Fortran compiler: C:/Program Files (x86)/Intel/oneAPI/compiler/2021.2.0/windows/bin/intel64/ifort.exe - skipped
Checking whether C:/Program Files (x86)/Intel/oneAPI/compiler/2021.2.0/windows/bin/intel64/ifort.exe supports Fortran 90
Checking whether C:/Program Files (x86)/Intel/oneAPI/compiler/2021.2.0/windows/bin/intel64/ifort.exe supports Fortran 90 - yes
Checking whether C:/Program Files (x86)/Intel/oneAPI/compiler/2021.2.0/windows/bin/intel64/ifort.exe supports F2003
Checking whether C:/Program Files (x86)/Intel/oneAPI/compiler/2021.2.0/windows/bin/intel64/ifort.exe supports F2003 -- yes
Added NVECTOR_SERIAL module
Added NVECTOR_SERIAL F2003 interface
Added NVECTOR_MANYVECTOR module
Added NVECTOR_MANYVECTOR F2003 Interface
Added SUNMATRIX_BAND module
Added SUNMATRIX_BAND F2003 interface
Added SUNMATRIX_DENSE module
.
.
.
  1. After pressing generate, the Visual studio 2019 project is being build. So far so good. The problems start after I open the project and try to build the "ALL_BUILD" target. The compiling stops with errors of the type:
Error	LNK1181	cannot open input file 'C:\Users\....\build_dir\src\sunmatrix\band\fmod\sundials_fsunmatrixband_mod_obj_static.dir\Debug\fsunmatrix_band_mod.f90.obj'	sundials_fsunmatrixband_mod_static	C:\Users\....\build_dir\src\sunmatrix\band\fmod\LINK	1	
  1. The reason for this kind of errors after lots of search, is caused because the visual studio 2013 & 2019 cant handle a mix C and Fortran files in same project. As very well stated here in order for the compilation to succeed a Fortran project containing all .f90 files must be created and then added to the current project created from Cmake in Build_dir. Furthermore all "....._mod_static" targets must have all their "..._mod.f90.obj" objects change their location and point to the output directory of the Fortran project created above. For example the "sundials_fcvode_mod_static" contains many objects like "fcvode_mod.f90.obj", fnvector_serial_mod.f90.obj and so on. The location of these objects is changed to the output of the custom Fortran project.

  2. A trick needed above is that all .f90 files must be renamed to duplicate their extension. For example the the fcvode_mod.f90 must be renamed to fcvode_mod.f90.f90 . This is needed because the produced fortran obj files retain inside them ".f90" and be in form "fcvode_mod.f90.obj". This is how the Cmake writes the ALL_BUILD.vcxproj file.

  3. After doing all the above the code All_Build target runs well and compiles. Though as expected the Cmake_Fortan_Flags: "/W1 /nologo /fpp /libs:dll /threads" found in the Cmake are wrongly passed to cl compiler and produce warnings of type
    cl : command line warning D9002: ignoring unknown option '/fpp' . I assume these warnings can be ignored. A bigger output of All_Build target output can be seen:

Rebuild started...
1>------ Rebuild All started: Project: ZERO_CHECK, Configuration: Release x64 ------
1>Checking Build System
2>------ Rebuild All started: Project: sundials_generic_obj_static, Configuration: Release x64 ------
2>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sundials/CMakeLists.txt
2>sundials_band.c
2>sundials_dense.c
2>sundials_direct.c
2>sundials_futils.c
2>sundials_iterative.c
2>sundials_linearsolver.c
2>sundials_math.c
2>sundials_matrix.c
2>sundials_memory.c
2>sundials_nonlinearsolver.c
2>sundials_nvector.c
2>sundials_nvector_senswrapper.c
2>sundials_version.c
2>Generating Code...
2>sundials_generic_obj_static.vcxproj -> C:\Users\...\build_dir\src\sundials\sundials_generic_obj_static.dir\Release\sundials_generic_obj_static.lib
3>------ Rebuild All started: Project: sundials_generic_static, Configuration: Release x64 ------
4>------ Rebuild All started: Project: sundials_sunmatrixdense_obj_static, Configuration: Release x64 ------
5>------ Rebuild All started: Project: sundials_sunmatrixband_obj_static, Configuration: Release x64 ------
6>------ Rebuild All started: Project: sundials_sunmatrixsparse_obj_static, Configuration: Release x64 ------
7>------ Rebuild All started: Project: sundials_sunnonlinsolfixedpoint_obj_static, Configuration: Release x64 ------
8>------ Rebuild All started: Project: sundials_sunlinsolspfgmr_obj_static, Configuration: Release x64 ------
9>------ Rebuild All started: Project: sundials_sunnonlinsolnewton_obj_static, Configuration: Release x64 ------
10>------ Rebuild All started: Project: sundials_sunlinsolspgmr_obj_static, Configuration: Release x64 ------
11>------ Rebuild All started: Project: sundials_nvecserial_obj_static, Configuration: Release x64 ------
12>------ Rebuild All started: Project: sundials_sunlinsolpcg_obj_static, Configuration: Release x64 ------
13>------ Rebuild All started: Project: sundials_sunlinsolspbcgs_obj_static, Configuration: Release x64 ------
14>------ Rebuild All started: Project: sundials_sunlinsolsptfqmr_obj_static, Configuration: Release x64 ------
3>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sundials/CMakeLists.txt
3>sundials_generic_static.vcxproj -> C:\Users\...\build_dir\src\sundials\Release\sundials_generic.lib
15>------ Rebuild All started: Project: sundials_fgeneric_mod_obj_static, Configuration: Release x64 ------
4>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunmatrix/dense/CMakeLists.txt
4>sunmatrix_dense.c
5>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunmatrix/band/CMakeLists.txt
5>sunmatrix_band.c
6>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunmatrix/sparse/CMakeLists.txt
12>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunlinsol/pcg/CMakeLists.txt
5>C:\Users\...\cvode-5.7.0\src\sunmatrix\band\sunmatrix_band.c(401,66): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
5>C:\Users\...\cvode-5.7.0\src\sunmatrix\band\sunmatrix_band.c(402,31): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
4>C:\Users\...\cvode-5.7.0\src\sunmatrix\dense\sunmatrix_dense.c(312,25): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
4>C:\Users\...\cvode-5.7.0\src\sunmatrix\dense\sunmatrix_dense.c(313,31): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
7>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunnonlinsol/fixedpoint/CMakeLists.txt
10>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunlinsol/spgmr/CMakeLists.txt
9>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunnonlinsol/newton/CMakeLists.txt
8>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunlinsol/spfgmr/CMakeLists.txt
14>Building Custom Rule C:/Users/.../cvode-5.7.0/src/sunlinsol/sptfqmr/CMakeLists.txt
12>sunlinsol_pcg.c
6>sunmatrix_sparse.c
12>C:\Users\...\cvode-5.7.0\src\sunlinsol\pcg\sunlinsol_pcg.c(523,24): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
12>C:\Users\...\cvode-5.7.0\src\sunlinsol\pcg\sunlinsol_pcg.c(524,24): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
11>Building Custom Rule C:/Users/.../cvode-5.7.0/src/nvector/serial/CMakeLists.txt
6>C:\Users\...\cvode-5.7.0\src\sunmatrix\sparse\sunmatrix_sparse.c(1022,23): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
6>C:\Users\...\cvode-5.7.0\src\sunmatrix\sparse\sunmatrix_sparse.c(1023,41): warning C4244: '=': conversion from 'sunindextype' to 'long', possible loss of data
9>sunnonlinsol_newton.c
10>sunlinsol_spgmr.c
14>sunlinsol_sptfqmr.c
.
.
.

7>sundials_sunnonlinsolfixedpoint_obj_static.vcxproj -> C:\Users\...\build_dir\src\sunnonlinsol\fixedpoint\sundials_sunnonlinsolfixedpoint_obj_static.dir\Release\sundials_sunnonlinsolfixedpoint_obj_static.lib
10>Done building project "sundials_sunlinsolspgmr_obj_static.vcxproj".
22>------ Rebuild All started: Project: sundials_sunlinsolspfgmr_static, Configuration: Release x64 ------
23>------ Rebuild All started: Project: sundials_sunnonlinsolfixedpoint_static, Configuration: Release x64 ------
24>------ Rebuild All started: Project: sundials_sunlinsolspgmr_static, Configuration: Release x64 ------
15>cl : command line warning D9002: ignoring unknown option '/fpp'
15>cl : command line warning D9002: ignoring unknown option '/libs:dll'
15>cl : command line warning D9002: ignoring unknown option '/threads'
.
.
.
1>Building Custom Rule C:/Users/.../cvode-5.7.0/src/cvode/CMakeLists.txt
61>sundials_cvode_static.vcxproj -> C:\Users\...\build_dir\src\cvode\Release\sundials_cvode.lib
60>sundials_fsunlinsolband_mod_static.vcxproj -> C:\Users\...\build_dir\src\sunlinsol\band\fmod\Release\sundials_fsunlinsolband_mod.lib
62>sundials_fsunlinsoldense_mod_static.vcxproj -> C:\Users\...\build_dir\src\sunlinsol\dense\fmod\Release\sundials_fsunlinsoldense_mod.lib
63>------ Rebuild All started: Project: sundials_fcvode_mod_obj_static, Configuration: Release x64 ------
63>Building Custom Rule C:/Users/.../cvode-5.7.0/src/cvode/fmod/CMakeLists.txt
63>cl : command line warning D9002: ignoring unknown option '/fpp'
63>cl : command line warning D9002: ignoring unknown option '/libs:dll'
63>cl : command line warning D9002: ignoring unknown option '/threads'
63>fcvode_mod.c
63>sundials_fcvode_mod_obj_static.vcxproj -> C:\Users\...\build_dir\src\cvode\fmod\sundials_fcvode_mod_obj_static.dir\Release\sundials_fcvode_mod_obj_static.lib
63>Done building project "sundials_fcvode_mod_obj_static.vcxproj".
64>------ Rebuild All started: Project: sundials_fcvode_mod_static, Configuration: Release x64 ------
64>Building Custom Rule C:/Users/.../cvode-5.7.0/src/cvode/fmod/CMakeLists.txt
64>sundials_fcvode_mod_static.vcxproj -> C:\Users\...\build_dir\src\cvode\fmod\Release\sundials_fcvode_mod.lib
65>------ Rebuild All started: Project: ALL_BUILD, Configuration: Release x64 ------
65>Building Custom Rule C:/Users/.../cvode-5.7.0/CMakeLists.txt
========== Rebuild All: 65 succeeded, 0 failed, 0 skipped ==========
  1. Time to build the Install target. A necessary step that is needed is that a folder inside the "build_dir" must be created with the name "Fortran_Static". Here all mod files found in the custom Fortran project must be copied. This is a required step since at the last step of the INSTALL target the data of "Fortran_Static" are copied to the "Install_dir/fortran".

  2. A word of caution its good to have the "Install_dir" in a directory for which the user has access to write. The "c:\program files (x86)" is not one of them. In which case the user might have to call a cmd.exe with admin rights and the call "msbuild INSTALL.vcxproj".

  3. After all the above steps I got the Install_dir with the folder containing "Fortran", "Include" and "lib" folders. The real problem starts now. After including the install_dir properly to one of the included examples (cv_analytic_sys_dns_f2003.f90) I got the following erroneous output:

 Finished initialization, starting time steps

        t           y1           y2           y3
 ------------------------------------------------------
   0.00000E+00  1.00000E+00  1.00000E+00  1.00000E+00
forrtl: severe (408): fort: (10): Subscript #1 of the array YVEC has value 2 which is greater than the upper bound of 1

Image              PC                Routine            Line        Source
libifcoremdd.dll   00007FFCBC2C736C  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A37214CE  RhsFn                     109  Example1.f90
testing_CVode.exe  00007FF7A3735700  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A3724DDA  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A372332B  MAIN__                    237  Example1.f90
testing_CVode.exe  00007FF7A3729E4E  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A372DB39  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A372DA5E  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A372D91E  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF7A372DBAE  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFD27C37034  Unknown               Unknown  Unknown
ntdll.dll          00007FFD27F42651  Unknown               Unknown  Unknown

  1. I do note that that when I configured the Cmake i removed all tests and examples so i could reduce the amount of errors and warning so I could finish the compilation. So I manually created a Fortran project for "examples/cvode/F2003_serial/cv_analytic_sys_dns_f2003.f90" with the name "testing_CVode". The crash as shown above indicates the the "yvec" is an array that contains one element only (with value "yvec(1)=1") but there is a line of code in which more elements of it are requested see "yvec(1:neq)" in fvec = matmul(Amat, yvec(1:neq)). Furthermore "yvec" gets its size from a call to the "FN_VGetArrayPointer".

yvec => FN_VGetArrayPointer(sunvec_y).

  1. A part of the code is shown below:
 integer(c_int) function RhsFn(tn, sunvec_y, sunvec_f, user_data) &
       result(ierr) bind(C,name='RhsFn')

    !======= Inclusions ===========
    use, intrinsic :: iso_c_binding
    use fsundials_nvector_mod

    !======= Declarations =========
    implicit none

    ! calling variables
    real(c_double), value :: tn        ! current time
    type(N_Vector)        :: sunvec_y  ! solution N_Vector
    type(N_Vector)        :: sunvec_f  ! rhs N_Vector
    type(c_ptr),    value :: user_data ! user-defined data

    ! pointers to data in SUNDIALS vectors
    real(c_double), pointer :: yvec(:)
    real(c_double), pointer :: fvec(:)

    ! ODE system matrix
    real(c_double) :: Amat(neq,neq)

    !======= Internals ============

    ! get data arrays from SUNDIALS vectors
    yvec => FN_VGetArrayPointer(sunvec_y)
    fvec => FN_VGetArrayPointer(sunvec_f)

    ! fill A matrix (column major ordering)
    Amat = reshape([&
         lamda/4.d0 - 23.d0/40.d0, lamda/4.d0 + 21.d0/40, lamda/2.d0 + 1.d0/20.d0,   &
         lamda/4.d0 - 3.d0/40.d0,  lamda/4.d0 + 1.d0/40,  lamda/2.d0 + 1.d0/20.d0,   &
         lamda/4.d0 + 13.d0/40.d0, lamda/4.d0 - 11.d0/40, lamda/2.d0 - 1.d0/20.d0 ], &
         [3,3])

    ! fill RHS vector f(t,y) = A*y
    fvec = matmul(Amat, yvec(1:neq))

    ! return success
    ierr = 0
    return

  end function RhsFn

  1. It seems that the "FN_VGetArrayPointer()" function returns properly only the first element of the array and fails to do the same for the rest. If the same example is compiled in release mode and not debug then the error is ignored and the output is:
 Finished initialization, starting time steps

        t           y1           y2           y3
 ------------------------------------------------------
   0.00000E+00  1.00000E+00  1.00000E+00  1.00000E+00
   5.00000E-03  7.03275E-01  7.06270E-01  4.10045E-01
   1.00000E-02  5.22668E-01  5.28650E-01  5.23171E-02
   1.50000E-02  4.12493E-01  4.21452E-01 -1.64556E-01
   2.00000E-02  3.45038E-01  3.56966E-01 -2.95998E-01
   2.50000E-02  3.03496E-01  3.18384E-01 -3.75623E-01
   3.00000E-02  2.77673E-01  2.95512E-01 -4.23820E-01
   3.50000E-02  2.61385E-01  2.82166E-01 -4.52956E-01
   4.00000E-02  2.50882E-01  2.74596E-01 -4.70529E-01
   4.50000E-02  2.43891E-01  2.70529E-01 -4.81091E-01
   5.00000E-02  2.39030E-01  2.68584E-01 -4.87399E-01

  General Solver Stats:
    Total internal steps taken =       73
    Total rhs function calls   =      105
    Num lin solver setup calls =       21
    Num error test failures    =        6
    Last method order          =        5
    Next method order          =        5
    First internal step size   = 6.86623E-06
    Last internal step size    = 1.52517E-03
    Next internal step size    = 1.52517E-03
    Current internal time      = 5.10351E-02
    Num nonlinear solver iters =      102
    Num nonlinear solver fails =        0
    Num Jacobian evaluations   =        2

  1. Which is exactly the expected output shown in the out file . Thusly the Cvode ran properly under release mode but not under debug. Moreover it seems that the "FN_VGetArrayPointer()" function shows properly the head of the array, but it ignores the rest of the body.

  2. If I were to run the cv_analytic_fp_f2003.f90 example that has only one ode equation to solve and not a system of equations( as above) then no error will occur in the debug mode. The "yvec" properly is array with only one value. Obviously all the examples that have more than one ode to solve crash as also the test "test_fnvector_serial_mod" crashes. The test contains the files "test_fnvector_serial_mod.f90", "test_nvector.f90", "test_utilities.f90" and is part of the ALL_Build (If Fortran tests are included in Cmake). Some small output of the test_fnvector_serial is:

 Serial N_Vector Fortran 2003 interface test
 SUCCESS: smoke tests passed
 PASSED test -- FN_VMake
forrtl: severe (408): fort: (10): Subscript #1 of the array XDATA has value 2 which is greater than the upper bound of 1

Image              PC                Routine            Line        Source
testing_CVode.exe  00007FF6E1FA59AC  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF6E1F931AF  CHECK_ANS                 163  test_fnvector_serial_mod.f90
testing_CVode.exe  00007FF6E1F9168B  TEST_FNVECTOR_mp_          79  test_nvector.f90
testing_CVode.exe  00007FF6E1F92F07  TEST_NVECTOR_SERI         137  test_fnvector_serial_mod.f90
testing_CVode.exe  00007FF6E1F93550  MAIN__                    204  test_fnvector_serial_mod.f90
testing_CVode.exe  00007FF6E1FAEC8E  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF6E2043389  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF6E20432AE  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF6E204316E  Unknown               Unknown  Unknown
testing_CVode.exe  00007FF6E20433FE  Unknown               Unknown  Unknown
KERNEL32.DLL       00007FFD27C37034  Unknown               Unknown  Unknown
ntdll.dll          00007FFD27F42651  Unknown               Unknown  Unknown
  1. To catch some questions before asked, I pay attention to build and release modes in Fortran and All_build projects to be compatible (despite showing output from a release mode and running example in debug mode). Also the Fortran project is in 64bit mode well.

  2. So please, I have spent too much time trying to compile Sundials/CVode under Visual studio with Intel Fortran environment, any useful input on how to fix the "Yvec" size would be greatly appreciated!!!!

@balos1 balos1 self-assigned this Aug 3, 2021
@Bill-gitt
Copy link
Author

At least, does anyone tried to compile CVode with Visual studio (2013 version or later) and encountered similar problems where Fortran objects are not compiled because the Fortran and C objects must be in different projects?

@balos1
Copy link
Member

balos1 commented Aug 11, 2021

There was a previous user report on the sundials email list about the issue with Visual Studio and Fortran projects. It seems that you have found a resolution for that though. I don't quite follow what you have done to allow it to compile, but If you had to change code in SUNDIALS, posting the diff or a pull request would be appreciated.

As far as the run-time issue in Debug mode... I believe this is because the FN_VGetArrayPointer function calls c_f_pointer with a shape of [1] here. In Debug mode there is likely a bounds check that does not like this.

You can try to update the FN_VGetArrayPointer function to:

function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
type(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function

If this works, then we can try to incorporate it into the next SUNDIALS release.

@Bill-gitt
Copy link
Author

Bill-gitt commented Aug 12, 2021

Dear Balos,

Yep we have a winner!
After 1.5 month of efforts CVode on Fortran runs!

The code you proposed fixed the problem with out of boundaries exception.
After updating the "FN_VGetArrayPointer" Function the example ("examples/cvode/F2003_serial/cv_analytic_sys_dns_f2003.f90") ran properly and after doing some debug the "yvec" is an array with size of 3 elements as it was supposed to be.

A couple of things to note:

  1. The fix you proposed has a typo:
function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
type(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function

The line "type(C_INT64_T) :: length " is supposed to be "integer(C_INT64_T) :: length " else it won't compile, so the correct update would be:


function FN_VGetArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 
integer(C_INT64_T) :: length 

farg1 = c_loc(v)
fresult = swigc_FN_VGetArrayPointer(farg1)
length = FN_VGetLength(v)
call c_f_pointer(fresult, swig_result, [length])
end function
  1. The subroutine "c_f_pointer" is used in a similar way elsewhere is the code and may need an update/fix too.
    Please have a look:
function FN_VGetDeviceArrayPointer(v) &
result(swig_result)
use, intrinsic :: ISO_C_BINDING
real(C_DOUBLE), dimension(:), pointer :: swig_result
type(N_Vector), target, intent(inout) :: v
type(C_PTR) :: fresult 
type(C_PTR) :: farg1 

farg1 = c_loc(v)
fresult = swigc_FN_VGetDeviceArrayPointer(farg1)
call c_f_pointer(fresult, swig_result, [1])
end function

Best regards,
Bill

@Bill-gitt
Copy link
Author

Dear Balos,

Regarding the steps I followed to compile under Visual Studio 2013 and 2019 unfortunately I didn't update any code. I just did some manual changes on the visual studio project that the Cmake-gui had generated. This means that every time the generate button is pressed on Cmake all the changes I did, must be redone. For the a simple version of Cvode (no examples or tests) this requires about 50 mins of copy paste. To fix properly Cvode, extensive changes on Cmake files are needed, so that the C and Fortran files are not anymore in the same project and instead are separated in different projects. The compilation order should be, first build the Fortran project and then the C targets.

Since the Sundials Cmake files seemed to me very complicated (so many targets with different options)

  1. I just created a simple Fortran project and I added all .f90 files I could find in Cvode. I duplicated the extension of Fortran file from .f90 to .f90.f90.
  2. I imported the Fortran project in the existing solution the Cmake-gui produced.
  3. I found all targets that pointed to a .f90.obj file and I changed the location of where the linker should search to find them. So for example in the solution explorer (right top hand side of visual studio) there is target (project) named "sundials_fcvode_mod_static" and if you expand it (expand the object folder too) this contains many objects like "fcvode_mod.f90.obj", fnvector_serial_mod.f90.obj and so on. If you look at the properties of these objects (right hand side of screen, under solution explorer) you will see a field called "Relative Path". This path contains the path where the linker should look to locate these objects. If you change this path to the "debug" or "release" folder of the Fortran project created above, then all will be ok. The linker will find the fcvode_mod.f90.obj objects and the "All_Build" will compile properly.
  4. To build the "Install" target I just created a new folder named "Fortran_Static" in the "build_dir" directory and inserted inside it all the .mod files created from the Fortran project above. Then hit build the "Install" target and everything worked.

I hope this helped a bit on the process I followed to compile Cvode with Visual Studio. If there is a need for more clarification let me know. It took me some time compile Cvode and I hope this is useful somehow.

Cheers,
Bill

@balos1
Copy link
Member

balos1 commented Aug 31, 2021

@Bill-gitt Thanks for the detailed writeup that others can refer to for a workaround. We will try to fix both issues (the Visual Studio build, and FN_VGetArrayPointer problem) in a future SUNDIALS release.

@Bill-gitt
Copy link
Author

Dear Balos,

It seems I have encountered another issue with the FCvode. It is related to its thread-safety, it seems independent threads access shared memory when executing memory release functions (FCVodeFree or FSUNMatDestroy etc...) and the program crashes. So, should I create a new issue in github or continue here in the current thread (it may be compile related)?

Regards,
Bill

@drreynolds
Copy link
Collaborator

drreynolds commented Sep 7, 2021 via email

@balos1
Copy link
Member

balos1 commented Sep 7, 2021

@drreynolds is correct that FCVODE (the FORTRAN 77 interface to CVODE) is not theadsafe, but I think you are referring to the FCVode function from the sundials_fcvode_mod module (i.e. the F2003 interfaces), correct @Bill-gitt?

If you are indeed referring to the FCVode function from the sundials_fcvode_mod module, please start a new issue for it.

@Bill-gitt
Copy link
Author

Thank you for your replies @drreynolds and balos1.

-The reason I turned to Cvode from Intel ode solver was that FCvode (version 2003) was thread safe. It means a lot that Pr. Reynolds confirmed that Fcvode (version 2003) is thread safe. Unfortunately the past month I didn't succeed to parallelize my code through openmp. I am using threadprivate allocatable arrays to feed data to Fcvode inside openmp regions and I get inconsistent results with lots of crashes due to either Floating point exceptions or Memory access violations. My code works perfectly in serial compilation (absence of any openmp directives). So Its either the latest Intel OneApi Fortran compiler has made small changes in the latest openmp implementation that I am not aware of or using Fcvode forced my compiler to use F2003 Fortran that may be incompatible with existing code.

-At the moment I am trying to produce a simple example that clearly exhibits inconsistent results between serial and openmp version of my code. If I have something concrete I 'll create a new issue in hope for resolution.
-I replied in the current thread cause I am not quite sure the compilation process for producing Fcvode is absolutely correct. I worry that internal optimizations of fcvode or Cvode may spawn extra threads inside openmp regions and create unexpected results. Though this shouldn't be the case.

Best Regards

@Bill-gitt
Copy link
Author

Bill-gitt commented Oct 20, 2021

Dear balos1,

After lots of effort, I finally managed to make Fcvode work properly inside openmp regions. This is a success I believe (unless something else comes up later).
To give a bit of backstory so others may benefit from it, this is what happened. After trying to run simple Fcvode examples in parallel yielded random results. Most of the time everything worked out well but one out ten runs 1/10 the program could give erroneous results. Thusly I decided to use Intel inspector on my code. What I found out was eye opening, more than 25 warnings where revealed inside Fcvode libraries (irrelevant of the code I have written) that indicated failure of thread safely. I knew though that Fcvode should be thread safe and thusly the reason for failing should be in compilation. Searching a bit further and testing lots of different compilation flags I found out what was missing. The directive "qopenmp" was missing from the compilation of Fcvode interface. Searching online in Intel I found this

(1) “Static local data” is shared by each thread, i.e. each thread may access the same data location! This is potentially unsafe.
(2) “Automatic data” is independent between each thread; it has its own, independent copy on its own stack.
With Intel Fortran when compiled and running serially, the defaults are local scalar variables are automatic and local arrays are static. But when compiling with -qopenmp, local arrays are automatic by default. It is the same as compiling with -auto. This may require an increase in the maximum stack size. If the stack size is too small, there will be a segmentation fault at runtime.

Which means that when not using qopenmp local arrays are not thread safe...

After using qopenmp and running intel inspector again the 25 warnings were now only 2 which seemed nothing serious.
So for anyone that wants to run Fcvode inside openmp regions make sure that you compile code with qopenmp.

That's it for now. I have managed to compile Fcvode with Intel Visual studio and use its thread safety features, successfully.

Best regards

@balos1
Copy link
Member

balos1 commented Oct 27, 2021

@Bill-gitt Thanks for posting your findings.

@balos1 balos1 closed this as completed Oct 27, 2021
@balos1 balos1 reopened this Oct 27, 2021
@balos1 balos1 removed the bug label Nov 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants