Using HIP-backend on Windows

[blinkfrog]

Now that HIP is officially supported on Windows, can AdaptiveCpp be built to use it as a backend?

[blinkfrog]

I've just tried to built AdaptiveCpp with HIP support. First, I rebuilt LLVM with HIP support by adding AMDGPU to LLVM_TARGETS_TO_BUILD. AdaptiveCpp, still, refused to build until I added also -DLLVM_ENABLE_RUNTIMES="compiler-rt"`` to compiler line (CUDA didn't require this). After this, and after adding C:\Program Files\AMD\ROCm\5.5\bintoPATH` environment variable, AdaptiveCpp found HIP and was successfully built.

However, when trying to compile any SYCL-program, I get this error:
acpp fatal error: Required command line argument --acpp-rocm-path or environment variable ACPP_ROCM_PATH not specified.

If I add --acpp-rocm-path="C:/Program Files/AMD/ROCm/5.5", it tries to compile, but I get these errors during the linking stage:

LINK : warning LNK4044: unrecognized option '/rpath=C:/Program Files/AMD/ROCm/5.5/lib'; ignored
LINK : warning LNK4044: unrecognized option '/rpath=C:/Program Files/AMD/ROCm/5.5/hip/lib'; ignored
LINK : fatal error LNK1181: cannot open input file 'amdhip64.lib'

C:/Program Files/AMD/ROCm/5.5/lib is correct path (and C:/Program Files/AMD/ROCm/5.5/hip/lib is incorrect), but these options are ignored anyway.

So, it can be some error (if needed, I can open an issue).

I workarounded it by copying this lib to current folder (alternatively it is possible to set it in target_link_directories when using cmake), and it compiled, but I currently can't test if it works as I don't have AMD GPU (will try to test on another computer).

Also, a bit strange, is that when I tried to compile this as cmake project, ACPP_ROCM_PATH variable was ignored, and only --acpp-rocm-path in target_compile_options worked.

[blinkfrog]

I think ROCm runtime is supported now on Windows. According to this page: https://rocm.docs.amd.com/en/latest/release/windows_support.html
Edit: now it is this page: https://rocm.docs.amd.com/projects/install-on-windows/en/latest/reference/system-requirements.html

Support on Windows is provided with two levels on enablement.
Runtime: Runtime enables the use of the HIP/OpenCL runtimes only.
HIP SDK: Runtime plus additional components refer to libraries found under [Math Libraries] ...

I'll try to find computers with AMD GPUs to test the executable I compiled.

I also want to ask if it is possible to compile universal binary that would support CUDA and HIP backends and work with GPU which is available. Currently, exe with HIP-backend which I compiled is crashed with System Error saying that amdhip64.dll was not found (it is shipped with AMD drivers), and this error occurs before I can catch it with try/catch. So, an universal binary with CUDA and HIP support probably won't work on a system which has nVidia card but hasn't AMD card.

[fodinabor]

Cool, thanks for the link :)

hmm... in theory, the executable should not link directly to the amdhip64.lib nor the amdhip64.dll, if I'm not mistaken.. that's the whole reason why we have a plugin architecture with different backend DLLs / .sos -> trying to load the library should fail gracefully if a required library isn't found... so 2 things:

• @illuhad I am confused, why are we linking the user application with cudart and amdhip64 (in (explicit) multipass)?
• I'm not sure the APIs that we use on Windows for the dynamic loading are indeed fault-tolerant ... can you tell (e.g. by setting the env variable HIPSYCL_DEBUG_LEVEL=3) if it's immediately at exe start or if it prints some diagnostics of trying to load backend plugins first?

[illuhad]

@illuhad I am confused, why are we linking the user application with cudart and amdhip64 (in (explicit) multipass)?

Probably mostly due to historic reasons when we didn't have the plugin architecture yet. It might be necessary for some interop stuff, so perhaps removing them changes interop guarantees. I'm also not sure what would happen to our CUDA/HIP kernel invocations (which ultimately rely on hip/cudaLaunchKernel) inside the kernel launchers, which get compiled as part of application code. So, it might be worth investigating if they can be removed, but I'm not sure whether this is indeed the case.

[blinkfrog]

@fodinabor, unfortunately, compiled app is crashed with that System error ("The code execution cannot proceed because amdhip64.dll was not found. Reinstalling the program may fix this problem") before any debug message at any debug level is printed.

I think, this problem could be managed if LoadLibrary and GetProcAddress would be used to load amdhip64.dll at runtime instead of linking against amdhip64.lib at build time, but I don't know if AdaptiveCpp architecture allows this.

[blinkfrog]

Also, this is interesting, now that I have rt-backend-hip.dll plugin, and launch a binary which was compiled without any hip targets (for example, it targets cuda and/or omp), I get this warning:
[AdaptiveCpp Warning] backend_loader: Could not load backend plugin: C:/AdaptiveCpp\bin\hipSYCL\rt-backend-hip.dll with: 126
Probably, this is because of the same problem: this plugin requires amdhip64.dll. Still, this isn't clear why AdaptiveCpp can't just output the same warning instead of crashing when hip is one of targets.

[blinkfrog]

So, now I've had an opportunity to test my binary on a computer with AMD videocard. And it works! So, we can now say that AdaptiveCpp supports HIP-backends on Windows. Great!

There are some caveats though:

1. That uncatchable and not-processible The code execution cannot proceed because amdhip64.dll was not found. Reinstalling the program may fix this problem error still remains on a computers without AMD-card (and drivers) installed, meaning that this isn't possible currently to compile a single AdaptiveCpp binary with support of cuda and hip-backends - current workaround I use is to compile them into a separate dlls and load required dll dynamically. The opposite thing (trying to run this binary on a system with AMD-card but without nVidia-card installed) works, although still requires the presence of cudart64_12.dll file, causing uncatchable error otherwise.
   Edit: probably, if I would put amdhip64.dll into the directory with acpp binary, this would solve this problem (same way as having cudart64_12.dll in the directory of acpp binary solves the opposite problem), but I can't test this as I have no amdhip64.dll on my computer.

2. --acpp-rocm-path parameter, as well as ACPP_ROCM_PATH environment variable don't work, causing Warning ignoring unknown argument '-rpath=<HIP_PATH> warning and following linking error. This currently can be solved by copying amdhip64.lib into build directory.

[blinkfrog]

Wow, this helped! Thank you very much! I now can compile a single binary with omp, cuda and amd backends on Windows, and there is no runtime errors! Amazing!

Could you make changes to AdaptiveCpp's cmakelists.txt, similar to I made:

set(DEFAULT_WIN32_HIP_LINK_LINE "-L$HIPSYCL_ROCM_PATH/lib -lamdhip64")
if(WIN32)
  if(NOT ROCM_LINK_LINE)
    set(ROCM_LINK_LINE ${DEFAULT_WIN32_HIP_LINK_LINE} CACHE STRING "Arguments passed to compiler to link ROCm libraries to SYCL applications")
  endif()
else()
  if(NOT ROCM_LINK_LINE)
    set(ROCM_LINK_LINE "-Wl,-rpath=$HIPSYCL_ROCM_PATH/lib -Wl,-rpath=$HIPSYCL_ROCM_PATH/hip/lib ${ROCM_LIBS}" CACHE STRING "Arguments passed to compiler to link ROCm libraries to SYCL applications")
  endif()
endif()

or, alternatively, would you consider my PR with these changes if I make it? The same question is regarding adding host attribute to fma declaration in cmath file?

[illuhad]

Very cool! I totally forgot that the multipass modes also have different kernel invocation logic, so the dependencies on hipLaunchKernel are gone too :) (Seems like we were clever when designing this mode ;) )

I'm not sure we can globally enable the change to CMakeLists, as the default integrated multipass modes will still need the additional linking steps. We would have to distinguish between an integrated and an explicit multipass link line. Users today already can enforce the behavior that you want by building AdaptiveCpp using cmake -DROCM_LINK_LINE=" ... ".

The same question is regarding adding host attribute to fma declaration in cmath file?

This change seems more uncritical. Submit a PR and if everything in CI is fine, we can merge it :)

[blinkfrog]

I'm not sure we can globally enable the change to CMakeLists, as the default integrated multipass modes will still need the additional linking steps.

But that proposed change is conditional, on Linux and Mac all remains the same, it's just -rpath removed on Windows, where it doesn't work anyway, causing link-time errors which can be avoided only by placing amdhip64.lib into to build directory.

Well, with using explicit-multipass, it probably will compile without this change, not sure. But this change also make possible to compile binaries with integrated multipass, let they are hard-dependent on amdhip64.dll.

[illuhad]

Sorry, I missed that you are just removing rpath on Windows. In that case, it might be fine. Can you open a PR so that we can have CI check it?

[blinkfrog]

Thanks, I've opened two PRs. :)

[blinkfrog]

Closing the discussion, as all problems were solved. HIP-backend is proved to work on Windows, and, when using explicit-multipass, all issues with dependencies are resolved: it is now possible to compile a single binary which would target OMP, HIP and CUDA and could work on a computer which has no GPUs, supporting CUDA or/and HIP without any runtime errors.

[blinkfrog]

I am reopening the discussion.

HIP backend (when using HIP SDK 5.5) works on Windows when using integrated multipass compile flow.
However, when I compile binaries using explicit multipass, it doesn't. It correctly finds the device, creates queue, but can't perform kernels.

This is the output I get:

[AdaptiveCpp Warning] backend_loader: Could not load library: g:\Temp\amdtest\hipSYCL\rt-backend-cuda.dll with: 126
Running on device: AMD Radeon RX 7600
[AdaptiveCpp Error] from D:/source/AdaptiveCpp/src/runtime/hip/hip_code_object.cpp:67 @ build_hip_module(): hip_executable_object: could not create module (error code = HIP:218)
[AdaptiveCpp Error] from D:/source/AdaptiveCpp/src/runtime/hip/hip_queue.cpp:574 @ submit_multipass_kernel_from_code_object(): hip_queue: Code object construction failed
[AdaptiveCpp Error] from D:/source/AdaptiveCpp/src/runtime/hip/hip_code_object.cpp:67 @ build_hip_module(): hip_executable_object: could not create module (error code = HIP:218)
[AdaptiveCpp Error] from D:/source/AdaptiveCpp/src/runtime/hip/hip_queue.cpp:574 @ submit_multipass_kernel_from_code_object(): hip_queue: Code object construction failed
============== hipSYCL error report ==============
hipSYCL has caught the following undhandled asynchronous errors:
   0. from D:/source/AdaptiveCpp/src/runtime/hip/hip_code_object.cpp:67 @ build_hip_module(): hip_executable_object: could not create module (error code = HIP:218)
   1. from D:/source/AdaptiveCpp/src/runtime/hip/hip_queue.cpp:574 @ submit_multipass_kernel_from_code_object(): hip_queue: Code object construction failed
   2. from D:/source/AdaptiveCpp/src/runtime/hip/hip_code_object.cpp:67 @ build_hip_module(): hip_executable_object: could not create module (error code = HIP:218)
   3. from D:/source/AdaptiveCpp/src/runtime/hip/hip_queue.cpp:574 @ submit_multipass_kernel_from_code_object(): hip_queue: Code object construction failed
The application will now be terminated.

The problem is only with hip.explicit-multipass; cuda.explicit-multipass works well, and so is hip.integrated-multipass.

[blinkfrog]

I am closing this discussion as I decided to move the last mentioned problem into a separate discussion, so it would be easier to find it using search: #1447