🙏Please Add AMD Support🙏

[Uninportant]

Hi, I've been using stable diffusion for a while now and have always enjoyed making artwork and images, A while back I got into training AI models when dreambooth first came out as an extension to Auto 1111. But my gpu never had enough VRAM to do many of the things I wanted. So I decided to Upgrade my computer with a radeon 7900. It had a massive 20GB of VRAM and was an amazing switch for all my gaming and LLM activities I do. But it couldn't have been a worse mistake when it came to AI image generation. Many features are disabled on Auto 1111 such as training. and ComfyUI fully lacks support for it. I understand that many of people in the AI image generation world have a NVIDIA gpu or use a cloud service such as clipdrop. But if there is any way to add support for AMD to your todo list it would be greatly appreciated.

[comfyanonymous]

AMD is supported but to have the best experience you have to use it on Linux: https://github.com/comfyanonymous/ComfyUI#amd-gpus-linux-only

[fractal-fumbler]

amd recently released ROCm for windows https://rocm.docs.amd.com/en/latest/release/windows_support.html

haven't tried it myself, but there are plenty of supported gpus.

[aa-parky]

My rig,

RX 7900 XTX
Ryzen 7 7700X
Windows 11
Python --version 3.10.6
pip 23.2.1
win-pyenv 3.1.1

Working great for my use case. Much more efficient and effective than Automatic1111, for me, which I also still use.
I am not training models or anything like that so I'm not missing that feature.

Some samplers also throw some erorrs, but again not an issue for me.

[Hank-Rearden-11]

Seconded. Glad AMD now made ROCM available for windows. Maybe some more support via an install guide in the readme for amd gpu's with rocm for windows? The high cost of nvidia gpu's with sufficent vram makes it a barrier to entry, especially for training.

[lord-lethris]

so it seams that there is a windows install for ROCM 5.6.1

However, either DirectML and/or ROCM need to be included in PyTorch - and it just feels like there is no motivation for it, because PyTorch is depreciated.

[NeedsMoar]

Pytorch + ROCM isn't deprecated at all, ROCM on Windows is just too new for them to have support for yet and AMD doesn't have their shit together, so they're waiting for AMD to port MIOpen to windows... AMD ported ROCM / HIP to support Redshift and the other renderers they're working with to get HIP support in which is the profitable area they're getting killed by NVidia in because none of those renderers worked with AMD at all before and they can get lots of sales just being the cheaper option... Inference and machine learning applications are an afterthought because people tend to use whatever card they have for that, or already bought an NVidia card if they really needed to do it...

Anyway the weird thing about the MIOpen port is that it's already ported. If you download current Radeon ProRender plugins for Blender, Houdini, or Unreal Engine (mainly because those are the zipped plugins) which were built back in May, they each have a MIOpen.dll and RadeonML_MIOpen.dll included to run the AI denoising models and possibly for other functionality... so AMD has had some kind of working internal build for a while (ProRender releases are considered functioning products) but for whatever reason the MIOpen git project isn't able to release a Windows version. I don't know if the prorender team just spun up a windows build without telling the MIOpen people or what, but the main project not having one and the renderer having a functional version makes very little sense otherwise. In any case AMD releases beta drivers all the time (whether they're called that or not) which have far more impact than a library and MIOpen has a fixed interface between operating systems so I don't see why they couldn't hand off a static library build to torch so they can get going on that now rather than in october / november when MIOpen seems to be talking about releasing it.

Someone with more motivation than me could probably get the MIOpen.dll from the renderers working by creating an export .lib file for it and creating a custom torch build with that linked in on windows and the rocm flags set in CMake to redirect all the CUDA stuff. It's not horribly outdated and I don't think it depends on a specific rocm build or all the render plugins would be broken. It seems to have most of the functionality torch is probably using like softmax and such.

The torch_directml backend is incredibly slow but it's not documented well / at all either which isn't helping anyone program for it. I'm relatively certain from reading how directml works that it needs to be converted into a graph and optimized via torch.onnx.trace and export with dynamic inputs (which will cache functions and modules to disk) or torch.jit to perform a trace that can be optimized into a more compatible graph before running. There are a bunch of restrictions on the torchscript torch.jit can generate like not modifying tensors in ways that change size in-place (tensor += blah needs to be tensor = tensor + blah), not using numpy I think, and setting dynamic inputs up so they'll stay that way in the jitted code... onnx lowering has the same rules, and there are others about conditionals on non-local data or at all depending. Even though it can be trivially done with decorators it ends up requiring conversions of everything meant to run on the GPU. Without that DirectML just executes one instruction at a time very stupidly; it's a low level interface and the torch backend is incredibly alpha stage so maybe that'll get fixed some day.

Once ROCM can actually be used it has the same giant library of pre-compiled op versions that CUDA has so the jit / graphing becomes less important although it'll still improve speed to a degree.

AMD really should have avoided advertising the 7900xtx with "ML acceleration" when it came out... first thing they've done in over 20 something years of owning ATI/AMD cards that left a bad taste in my mouth because it implied a level of support that doesn't exist and isn't their usual ironing out of driver bugs that mysteriously tend not to hit me as much as everyone else. I'm tempted to just buy an ampere A6000 (since they ditched nvlink on non-datacenter ada and who knows what stupid program will exceed 48GB of vram in 2 years) and suck it up on the cost. It's still a bit cheaper than a W7900 and doesn't take up 3 slots like the AMD version (which is just incredibly stupid for a workstation GPU).

[Andyholm]

My rig,

RX 7900 XTX
Ryzen 7 7700X
Windows 11
Python --version 3.10.6
pip 23.2.1
win-pyenv 3.1.1

Working great for my use case. Much more efficient and effective than Automatic1111, for me, which I also still use. I am not training models or anything like that so I'm not missing that feature.

Some samplers also throw some erorrs, but again not an issue for me.

I'm only getting 2s/it on my 7900 XTX...

[NeedsMoar]

I picked up a 4090 and have happily been doing outputs at higher resolutions that would OOM on the directml backend. It's currently about 30x faster than the 7900XTX on directml was at the same resolution.

One other thing that's really not helping it is that about a week ago, so over 9 months after the card was released, support for the wavefront matrix acceleration instructions on RDNA3 was finally added to the DirectX agility SDK. I'm reasonably sure that DML requires that the base directx12 installation (not agility SDK) supports something in order to use it, so who knows how long it will be before directml can make any kind of use of this acceleration which is fairly necessary to keeping speed from tanking on large matrix sizes. It's a race of the tortoises: Microsoft who ported directml to pytorch as a sort of afterthought since it's meant to be a native API, or the combination of AMD getting MIOpen done + pytorch integrating windows ROCm into builds. MIOpen's windows support merge thread has disappeared so I'm hoping that's a good sign and they're in testing right now; I'd like to be able to keep the 7900XTX around and use both cards for this sort of othing and supposedly ROCm can pass things off to CUDA when needed.

Someone at AMD responded on the Windows port for MIOpen thread and said the issue with the old MIOpen.dll for their renderer is that it uses the OpenCL backend for ROCm which is much slower than HIP in current versions. Makes sense since OpenCL C code can't properly represent complex types like tensors and the compiler needs that information from the frontend so it can generate optimal GPU machine code without doing painfully expensive analysis that can still fail. They were probably just outputting it as a series of vector operations which weren't necessarily padded optimally. For the renderer that doesn't matter as much, it's not going to be calling the library tensor stuff as much as torch. If I were them I would have released it anyway to enable the pytorch team to do a port since it still contains the functionality (which then only requires a DLL swap to gain the speed advantages) in tandem with their working on it, but whatever.

[dfghsderftgerdf]

I have a system running amd and rtx gpu's comfy is already installed how and running fine for my rtx card. I'm wondering what steps i need to take to get my amd running. After following the direction in comfy i get this:
D:\ComfyUI_windows_portable>.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --directml --highvram
Failed to execute startup-script: D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager\prestartup_script.py / [WinError 32] The process cannot access the file because it is being used by another process: 'comfyui.log' -> 'comfyui.prev.log'

Prestartup times for custom nodes:
0.0 seconds (PRESTARTUP FAILED): D:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Manager

Traceback (most recent call last):
File "D:\ComfyUI_windows_portable\ComfyUI\main.py", line 72, in
import execution
File "D:\ComfyUI_windows_portable\ComfyUI\execution.py", line 11, in
import nodes
File "D:\ComfyUI_windows_portable\ComfyUI\nodes.py", line 20, in
import comfy.diffusers_load
File "D:\ComfyUI_windows_portable\ComfyUI\comfy\diffusers_load.py", line 4, in
import comfy.sd
File "D:\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 5, in
from comfy import model_management
File "D:\ComfyUI_windows_portable\ComfyUI\comfy\model_management.py", line 33, in
import torch_directml
ModuleNotFoundError: No module named 'torch_directml'

D:\ComfyUI_windows_portable>pause
Press any key to continue . . .

Step by step please

[lord-lethris]

I don't understand why PyTorch doesn't support ROCm for windows - They only support ROCm on linux. AMD's ROCm for windows is upto 5.7 now... I don't get it, because I even have it installed.

I'm just lost for words really - I'm not a programmer or a Dev so I don't understand, but people say AMD needs to pull their finger out, but from what I see, they have - its Pytorch that isn't.

[Andyholm]

I don't understand how there's seemingly only one guy doing something.
They did go from 11 open pull requests down to 3 within like 2 weeks though so I'm hoping we'll see the rest of them finished sooner than later.
Either way, shout out to that one guy that's doing something... Literally carrying the MIOpen port lol.

[Kademo15]

We are almost there. One last push ROCm/MIOpen#2570

[Andyholm]

I don't wanna get hyped yet, I'll just end up disappointed lol. No hate to the devs ofc.

[Kademo15]

Its going to be finished pretty soon i would say but then the question remains how long will pytorch need. Are there other dependencies than miopen that are missing.

[Andyholm]

I'm unsure, but it'd be dope to have it within 2023

[bdergin]

I have rocm installed on linux with Krita ai diffusion and Comfyui but I only have a drop down option for run on CPU or run on nvidia with an AMD 7900XTX, AMD support would be nice, thanks.

Or please add detailed instructions on how to add files from the rocm sdk to the local server or to comfyui..

Linux link for rocm: https://github.com/RadeonOpenCompute/ROCm

Windows Link for rocm https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html

[davenmth]

I must admit, it was the most painless install experience in this context so far. Worked like a charm. Thank you very much for your great work!

[davenmth]

Your solution should be part of the "official" README for the AMD section.

[Kademo15]

Thank you much appreciated. If i finish my exams I will update some stuff and make it for rdna2 users as well.

[lord-lethris]

I thought the point of this thread was for MS window users.?

We all know AMD support works in Linux. But there is next to none on windows.

[Kademo15]

The person who created this discussion never specified windows. If you want windows then you have to go the Zluda route works decently well.

[lord-lethris]

OMG OMG this is actually happening!!!

https://ryzenai.docs.amd.com/en/latest/index.html

Aaaand it requires Python 3.9...

[NeedsMoar]

No it isn't, python versioning wouldn't matter too much.

Vitis / XDNA is just a ton of FPGA elements (or in the case of laptop CPUs probably not very many) and costs a similar kind of fortune as everything else high-end and capable of running things right now. Their graphics cards don't have FPGAs or similar elements on them and probably won't ever because despite the speed and versatility they have programming delays and would be massively outperformed by AMD just moving CDNA3 matrix cores over to the consumer line. There's no real reason to do this though... the damage is kind of done by a whole bunch of people having GPUs that should technically be able to run things quickly and AMD trying their best to ignore them.

A fairly slow one of these Versal accelerators in terms of inference (that's not really their strong point, they were made to potentially handle 4TB/s of traffic to/from the chip at full speed without storing it, not run large models. Models that need to be run to detect events in 10,000 fps images maybe) costs $14000. They can only do INT8 inference so that's all I can compare; the 4090 runs INT8 at 660 TOPS non-sparse, at base clock. These things do 180 TOPS peak and draw half the power. I know shark was capable of compiling the SD 2.0 512x512 base model to quantized int8, but it wasn't really any faster. I'm assuming these things are so expensive because they have AI in the description and fast onboard optical transceiver slots, because 900,000 LUT elements isn't all that impressive for an FPGA. The better ones were at 7 million LUTs years ago.

You can buy small general purpose FPGAs that don't have all the dedicated parts that plug into M.2 slots, and another form factor that plugs into DDR memory slots of some sort that don't work with actual memory on specialized PCIe cards, or track down Xilinx stuff on ebay (but it's pretty much a requirement that you buy a new one if you don't have the programming software already because they don't like people having it thanks to the industry of "plug 'n play" hardware stand in "cores" written for FPGAs that companies who don't have the time to wait or the money to hire someone to re-implement the entire ddr5 spec in a "core" and the 100Gb Infiniband spec in another module ).

The fastest of the laptop processors that works with that dev kit you linked is "Up to 10 TOPS". I kinda hope everybody ignores it so they do some kind of real work on getting things running quickly on their GPU cores which are probably faster even on those integrated mobile versions if they'd stop messing around and do it.

[lord-lethris]

Well, I had to re-install windows and I re-installed Python 3.11 (which is 100% what I had before, because the install directory was still on my d:)...
... looks like "torch-directml" is no longer a thing for 3.11. Which is annoying. looks like its avalable for 3.10 still.

HOWEVER - I did see this appear -> pytorch-triton-rocm... and its installable on windows - What is this....? are my hopes to high?

[NeedsMoar]

Triton itself doesn't build on windows (and that package says Python 3.6) and the MIOpen libs still aren't out AFAIK (anybody want to buy a 7900XTX cheap?) so I'm guessing you're hoping for too much for now.

[NeedsMoar]

There are nightlies of torch-directml... MS made them nearly impossible to find and I think you have to sign up for temporary access to pull builds but there might be a py 311 build in there... I was running some bizarro version number when I last used my AMD card for anything non-vulkan.

[lord-lethris]

Stable Diffusion has hipBLAS using ROCm - Is this an option?

https://github.com/leejet/stable-diffusion.cpp/blob/master/docs%2FhipBLAS_on_Windows.md

[Jupinel]

It seems like EXPERT programmers really CAN'T make a .exe with 4-5 installation options. NO, After 10k hours of coding, they can't make an installer, it's 2000s technology, but somehow, they can't... ALWAYS the same thing. If you have AMD pray GOD for everything to work, because there is no explanation. Simply ONE LINE of instructions "run this"... Okay, where? How? After and before what?

If you all made cars, it would be necessary to have an engineer degree to be able to dive them. Doing this you change you potential target form 30% to 0,01% of the planet.

[Kademo15]

It seems like EXPERT programmers really CAN'T make a .exe with 4-5 installation options. NO, After 10k hours of coding, they can't make an installer, it's 2000s technology, but somehow, they can't... ALWAYS the same thing. If you have AMD pray GOD for everything to work, because there is no explanation. Simply ONE LINE of instructions "run this"... Okay, where? How? After and before what?

If you all made cars, it would be necessary to have an engineer degree to be able to dive them. Doing this you change you potential target form 30% to 0,01% of the planet.

What gpu and what issue are you facing I may be able to help.

[bigcat88]

https://blogs.windows.com/windowsdeveloper/2024/05/24/build-the-next-wave-of-ai-on-windows-with-directml-support-for-pytorch-2-2/

I'll test them when the release and not the "Pre-Release" version will be available..